Various enterprises and personal interests, such as Man-Machine Interaction (MMI), gesture studies, signs, language, social robotics, healthcare, innovation, music, publications, etc.

Category: Gesture Semiotics Page 1 of 2

Is a Yawn a Gesture?

In an old article BBC News reported about research showing that Pet dogs can ‘catch’ human yawns. The article is available online in Biology Letters here. (Article ‘Dogs catch human yawns’ by Ramiro M Joly-Mascheroni, Atsushi Senju* and Alex J Shepherd, 2008).

The copying activity suggests that canines are capable of empathising with people, say the researchers who recorded dogs’ behaviour in lab tests.
Until now, only humans and their close primate relatives were thought to find yawning contagious.
The team – from Birkbeck College, University of London – reports its findings in Biology Letters.
Yawning, although sometimes a response to extreme stress, is more often a sign of tiredness; but the reason for why yawning is catching is not fully understood.

Human cues. There is evidence that autistic individuals are less inclined to yawn into response to another human yawning, suggesting that contagious yawning betrays an ability to empathise, explained Birbeck’s Dr Atsushi Senju. Dr Senju and his team wondered whether dogs – that are very skilled at reading human social cues – could read the human yawn signal

There are several very interesting things in these statements. Firstly, I am interested in yawning itself. It is called a social cue. What is a ‘social cue’ as opposed to an ‘intentional act of communication’, which is how I define ‘gestures’?

The article itself has this to say about dogs’ abilities:

Dogs are unusually skilled at reading human social and communicative cues. They can follow human gaze and pointing (Hare et al. 2002; Miklo´ si et al. 2003; Miklo´ si & Soproni 2006), they can show sensitivity to others’ knowledge states (e.g. indicating the location of a hidden toy more frequently to someone not involved in hiding it than to someone who did the hiding, Vira´nyi et al. 2006) and they are even able to match their own actions to observed human actions (Topa´l et al. 2006).

Goffman and Kendon both make a distinction between ‘giving information’ and ‘giving off information’. In most cases, a yawn gives off information to possible observers, but a yawner does not mean to give information, I would think (although in many cases yawners may want to indicate their tiredness or boredom). The distinction is important because giving information is typically attended to and reacted upon, whereas giving off information is not. Expectations and social etiquette are likewise.

So, how about contagious yawning? It seems to be caused by empathy or to require empathy, at least in humans and dogs. As such a co-yawn also gives off the information that this other persons is observing you and empathizes with you, for what it’s worth.

And I think that that could well be the best explanation. Contagious yawning is behaviour that serves to provide information to those present that they are aware of each other and ’empathizing’ in a very economic way. It is economic because none of those present has to overtly attend to the behaviour and react upon it with speech or gestures. A bonding mechanism mostly below the surface of our consciousness.

And possibly, contagious yawning is much like all sorts of other behaviour, such as mirroring. It is a kind of mirroring I suppose. But there are many other sorts of mirroring.

Here is an alternative interpretation and explanation of contagious yawning

Note that there is a considerable and growing literature on yawning, contagious yawning and how this relates to our psychology and biology. In humans, dogs, chimpansees, other apes and monkeys, birds, cats, etc.

A very interesting research case. Take any animal and see if it catches your yawn.

I’m off yawning at the chickens, bye…

Why the Robonaut Handshake with ISS Captain Burbank is not a Greeting

Last Februari 15, 2012, NASA made a little fuss about Robonaut (or R2) performing ‘the first human-humanoid handshake in space’. You can watch it in the video below.

‘Robonaut Shakes Hands’, uploaded by ReelNASA

I would like to use this occasion to enlighten both humans and humanoids about greetings. Although it is of course nice that cpt. Burbank has some kind words regarding the robot’s firm handshake, a nice bit of programming in itself, the normal phases of a greeting, which is a fairly well documented social interaction pattern, are remarkably absent. A handshake is what is called the ‘close salutation’. These ‘close salutations’ are the final phase of the entire greeting episode, which also includes sighting and announcement, distance salutations, and an approach phase (Kendon, 1990). None of the first three steps were followed by Robonaut and cpt. Burbank, the latter merely waited for Robonaut to try and grab his hand. I think this is important, because greetings are among the best documented social interaction patterns of humans. If we want to create social robots, then programming a proper greeting may be one of the most important, and most manageable things we can strive for.

Some references:
Arendsen, J. (2008). Greeting by gesture, a review. Gesture, 8(3), pp. 386-390, link, pdf
Kendon, A. (1990). A description of some human greetings. Conducting Interaction, pp. 153-207. Cambridge University Press.

Update, here is a nice example of a handshake that includes all of the phases of a greeting. They make eye contact, nod slightly or incline their head, approach each other and shake hands.

Brilliantly funny ‘body language’ instructional tape

Here is a must-see video for anyone who is interested in gestures and body language and has a sense of humour. Be warned, it may force you to rethink some of your ideas about the conventionality of body language and the extent to which interpreting it can be taught (should you be a communications trainer).

In any case, it’s good for a laugh 🙂


Here is a collection of the sort of body language instruction that the above video is a parody of (with the exception of the fifth which again is a parody):

Thesis of Mats Andrén

For those among you who are interested in good gesture research it may be of interest to know that Mats Andrén, from Lund University, has published (online) his thesis called Children’s Gestures from 18 to 30 months. So far, I am enjoying reading it very much. Good job, Mats, and good luck with the defense 🙂

Golden Oldie: Garfield & the Waving Snowman

Boy, I have been looking for this cartoon for two and a half years (off and on, that is, see here). So, a big thanks to diamond-blade for pointing out the link.

Garfield and the Waving Snowman
It was the Garfield daily comic from February 27, 2005 (source)

The reason why I like this comic so much (besides my general fascination for the gestures in Garfield, see these posts) is that it is a wonderful illustration of an idea that I would love to test experimentally. The idea is that certain movements (i.e. having certain kinematic characteristics) might automatically trigger the perception of a gesture (in the sense of a movement that is intended to communicate). This idea is not new and was, for example, described by Adam Kendon in his 2004 book ‘Gesture. Visible action as utterance’. But this idea is also present, to some extent, in the work of Gunnar Johansson (1973) and Albert Michotte.

Prelimary ideas about experiments:

  • Generate a randomized display of motion (within some likely parameter space) and let people all watch that same fragment, then check if they see a gesture at the same time.
  • Condition: Manipulate it in such a way that it is easier or harder to imagine seeing a hand.
  • Condition: Create conditions where subjects feel it is likely or unlikely they will be insulted (perception of intention to communicate appears linked to sensitivity to insults, see my thesis or Bucci et al. 2008).

Gesture Definitions

I am going to try and coin some definitions regarding gestures.

A gesture is any act, except speech, by which we intend to communicate something beyond the act itself

This definition includes all normal gestures, that have meaning (the message that was communicated intentionally) because of cultural conventions (including languages) or iconicity, and all such acts (like giving flowers) that serve, mostly through context, as bearers of an additional meaning (like an apology). Speech is viewed as basically the same type of behavior (it fits the definition), but speech is given an exceptional status because of its importance and because it has certain characteristics that set it apart from other gestures. Excluded are all acts which either do not communicate anything (for example because the actor is not aware of an observer) or which only communicate themselves to an observer (”[look at me,] I am fishing/reading/sleeping/walking”).

I further wish to emphase the difference between normal, straightforward gestures and those acts that serve some other purpose in the first instance and only serve as gestures (or rather ‘gestures of something‘) in the second instance like the example of giving flowers to apologize. If I wish to distinguish between these different gestures I will add the term ’simple’ to the first category and ‘complex’ to the second category.

A gesture-simple is a gesture where the (sole) purpose of the act is to communicate

A gesture-complex is first some action but communicates an additional message in the second instance

I tried to find better words to express what I mean, but it’s the best I could come up with so far (it has been brewing for about a year, see one of my first posts).

Note that any gesture-simple may also be a gesture-complex (even a speech act can be a gesture-complex). I tried to explain this with this example of Pee Wee Reese standing by Jackie Robinson in the face of racist fan-abuse. The shoulder embrace was both a gesture-simple and a gesture-complex.

Pee Wee Rise
The gesture that touched a nation (source)

There are important reasons for making these distinctions (but I forgot them :-) ). Well, at least it will allow me (and maybe you) to better analyze observations of gesture. And perhaps it is necessary to be precise if you want to make statements about gestures. For example, I think that people can typically see that a movement is a gesture-simple from just its appearance but this is not true for a gesture-complex. I think that people easily miss that an action was intended as a gesture-complex or, vice versa, see/read too much in what was just some action, for example in this cartoon from Garfield:

Jon misreads Garfield's intentions
Jon mistakes Garfield’s intentions (source)

Let us see how far these definitions can take us.
Or does anyone have better suggestions?

And a final wild speculation: Women are not able to see a man’s action as just that action but are always convinced it is some gesture-complex (if we do not bring flowers we do not care, if we do bring flowers we have something to apologize for [but we just thought they would be nice on the table]). Men conversely tend to miss most of the complex gestures made by women (when they wear something nice and new to show their appreciation of some event for example). Any takers?

Iconicity in Gestures and Sign Language

Let us assume for a moment that Mueller was right: Iconicity in Gestures is achieved by hands that act, embody, model and draw. We further agree with her that achieving iconicity presupposes certain knowledge in the observer (and in the producer for that matter). But it is general knowledge about techniques such as modeling. It does not require spcific conventional knowledge of symbols. What we have is then a handful of strategies that people can apply to try to produce (or understand) gestures.

We can then ask the question: To what extent can people rely on the strategies of gestural iconicity to understand and produce gestures or sign language? Two examples came to my mind that I would like to share. The first is about applying the rules of gestural iconicity to reading sign language (jokingly) if you do not know it. The second example is of a applying the rules of gestural iconicity to create a fantasy signed interpretation of the lyrics of Torn (also a well known joke).

My first impression upon revisiting these movies was that the rules of gestural iconicity are at work here. It seems to be a good way of looking at what these people are doing. The joke interpretation of signing is almost entirely built from guesses at meaning through applying iconicity rules (needless to say, not very succesfully). But at the same time it does not seem to be enough. One thing especially appears to be missing. Much of the pantomime in Torn is of displaying emotions with the face and the body. Although you could see that as enacting, in the sense of imitating, it seems to be a slightly different matter.

MPI Lecure on Metonymy in Metaphoric Gestures

The Nijmegen Gesture Centre Lecture Series 2007 hosted a talk by Alan Cienki (currently at the VU Amsterdam as lecturer MA in English Language and Culture) and Cornelia Müller (Berlin Gesture Center) last week. I attended the lecture ‘How metonymic are metaphoric gestures?’ together with about 20 people. The talk and discussion afterwards were perhaps a little incoherent, partly because of most people’s unfamiliarity with the concepts of Metonymy and Metaphor (or was that just me) and particularly why they would be useful for gesture studies. It should also be noted that McNeill (1992) introduced a specific use of the words metaphoric gestures:

Hand and Mind (1992), p145: Metaphoric gestures create images of abstractions [as opposed to (iconic) gestures that exhibit images of events and objects in the concrete world]. In such gestures, abstract content is given form in the imagery of objects, space, movement and the like.

It is crystal clear that there is a whole world of thought (and fun) behind the word Metaphor. Big names Lakoff and Turner wrote brilliant books about it that are the stuff cognitive linguistics is made from. And to cap it all off, Mueller and Cienki lifted a tip of the veil about a hot upcoming book on metaphor in gestures too. It will be all I can do to stop from rushing to my online bookstore (as soon as I find time). We once had a serious young undergraduate student called Michelle Hilscher for a couple of months in our faculty. Then she was doing experiments with metaphor in images and I remember having the same incoherent discussions (from my part at least) about metaphor and iconicity with her as I did at the MPI now. Curiously, she is winning awards, and I am here blogging my PhD away. Time going to waste, I better get back to grooming my paper.

ps. If you can guess how many metaphors this post counts you receive an honorary mentioning.

Is there a gesture in every click?

I have been catching stories in my newsreader where the word ‘gesture’ is used in a way I found hard to understand. A group of people are talking about the gestures that can be seen in people’s internet behaviour or computer usage. Who started this use of the word gesture? I don’t know exactly, but here are some people involved: Steve Gillmor (GestureLab) Doc Searls (weblog) Danny Ayers (Raw) Robert Scoble (Scobleizer) Ayers summarized it nicely:

So here’s my reading of what he’s on about. The gesture as the unit of attention. They are (heavily context-dependent) events. Gestures are just our intentional, directed interactions with the software. These communication acts contain in themselves valuable information. That information could be used to assist the person in their activities (e.g. with predictive search) or it could be used by marketeers, in a way it’s like a very wide broadening of AdSense.

And then there was the GestureBank, which popped up in December 2nd 2005 on ZDNet (by Gillmor). That turned into the AttentionTrust, of which Gillmor is the president (see governance). Their Mission:

1. Empower people to exert greater control over their “attention data,” i.e. any records reflecting what they have paid attention to and what they have ignored. We accomplish this by promoting the principles of user control, by distributing our Attention Recorder, and by supporting the development of other appropriate tools, standards and practices.

2. Educate people about the value of their attention and the importance of attention data. 3. Build a community of individuals and organizations that will guarantee users’ rights to own, move, and exchange their attention data, in a transparent environment that gives users the freedom to decide how their data will be used.

So the term gesture is used in these contexts to refer to any significant action such as sending someone a link, searching for something, reading something, writing something, etc, etc. These things can be recorded and then algorithms should be able to scan this data to enable all sorts of things. Not everyone would agree though, and I think the following comment by Christopher Coulter to this post on Scobleizer captures the criticism neatly:

My my, from ‘attention’ to ‘gesture algorithms’ ¦ the never-ending supply of meaningless buzzwords, keeps on trucking. In the real world, this is called having friends, knowing and caring about them, keeping in touch, and gasp, unconditionally at that, even when they are of no use to you, or can’t help boost your traffic or can’t play your marketing pitch up. Life is not a computer program spitting out info and links in some sort of raw attempt to mimic human emotions. And the ‘gestures’ we send out, might be pure fiction, what we dream them to be, not what they actually are.

All in all, I thought it would be good to keep an eye on these developments. At the very least it will help you filter out the nonsense from the good stuff in your newsreader. And with a single click you can then wave goodbye to it.

Is Cube-Flopping Gesturing?

Fellow PhD student at the TU Delft, Miguel Bruns-Alonso created a nice video of his Music Cube (his graduation project, see paper). And then Jasper van Kuijk (another colleague) blogged it for usability. And here I come wandering wondering: whether moving this Cube in certain ways to control music playing can or should be considered gesturing.


Perhaps this is a highly irrelevant question. I am pretty sure Miguel could barely care less. But that’s me, always worrying about silly gesture stuff.

In a way the question is similar to a previously unanswered question Is Sketching Gesturing?

Like with sketching it is not the movement itself that matters. Rather, it is the effect that the movement causes that is important. Although the case of “shuffling” may be an exception because the “shaking” movement is fairly directly registered. Other commands are given by changing the side of the Cube that is up (playlists), or by pressing buttons (next, turn off), or turning the speaker-that-is-not-a-speaker (volume). These are fairly traditional ‘controlling’ movements, comparable to adjusting the volume or radiofrequency with a turn-knob (as in old radios).

I will leave aside the question whether such tangibility constitutes a more valuable or enjoyable interaction with our machines. Some believe that it does and who am I to disagree. Like it or not, take it or leave it, you choose for yourself.

What concerns me is whether such developments and other gesture recognition developments share certain characteristics. If so, then exchanging ideas between the areas may be a good idea. One of my bits of work is on discriminating fidgeting and gestures.

The question rises whether the Music Cube will allow people to pick it up and fidget with it without immediately launching commands. Can I just handle it without ‘touching the controls’? Like with other gesture recognition applications I want this Cube to allow my fidgeting. In that case rules for human behaviour regarding the difference between behaviour that is intended to communicate (or control) and behaviour that is just fidgeting would be useful. And why don’t we carry the thought experiment of the Music Cube further? If it has motion sensing, it should be able to do the sort of things that the Nintendo Wii can too. Why not trace gestures in the air to conjure up commands of all sorts? How about bowling with the Cube? Or better yet, playing a game of dice?

Page 1 of 2

Powered by WordPress & Theme by Anders Norén