Various enterprises and personal interests, such as Man-Machine Interaction (MMI), gesture studies, signs, language, social robotics, healthcare, innovation, music, publications, etc.

Category: Semiotics

Under the Spell of Tinkerbell

Come and travel with me
Fly with me to Neverland
Peter Pan will come for us

Just think your happy thoughts
Oh and of course a bit of pixie dust
From the one and only
Tinkerbell

My kids just got the old Disney version of Peter Pan and are loving it. And so do I. What a lovely story about fantasy and growing up. But the tiny Tinkerbell caught my professional eye as well. She is interesting to watch for her gestures and body language. She doesn’t talk, she just tingles and gestures. Check the movie for examples. Still Peter Pan, Captain Hook and the others who believe in fairies understand her.

I started to pay attention to how she gestures and it struck me that she barely mimes, nor does she create language-like gestures. Nearly all of her gesturing is spent on expression of her emotions or attitudes towards Peter and Wendy.

Now I find myself at a crossroads. Behind me is the road I traveled through a land of gestures that were all about form and meaning, about semiotics and perception. I felt the expressive aspect of gesturing to be mostly besides the point. And the point was that practically all gestures signify something. They all carry meaning from maker to taker. And my main interest has been how humans achieve this so effortlessly.

Oh no, not feelings! (source)

I had expression of emotions nicely stacked in a box of (mostly) facial expressions, subtitled ‘unvoluntary universal displays’. One does not so much intend to communicate with them, but onlookers can glean information from them.

But I am afraid it is just not true. Humans (and fairies) use displays of emotion as a way to communicate them to others. We can fully intend to influence others to our advantage this way. An expressive gesture can in fact be a complex social act, even if it has its roots in a simple reflex.

And so my choice is made
I will not grow up
But endulge myself in fantasy
And let my feelings come alive

Let us gesture
Let us fly

Do your gestures look like they should?

Are both sign language and gestures best defined in terms of what they should look like?

Reading in Crasborn‘s (2001) thesis always gives me plenty to think about. One thing struck me yesterday. The emphasis placed on perception in the modeling of sign language. I read the same before in a review by Sara Fortuna of ‘Ideeen zu Einer Mimik’ and other works of Johann Jakob Engel. Crasborn describes sign language (NGT) and Engel describes gestures mostly in a theatrical context. One is about signers and their conversational partners, the other about actors and their audience. Yet they both state that a gesture (or sign) is defined in terms of what it should look like.

Is a perfect sign in the eye of the beholder? (source Heather Brooks)
What a gesture should look like is the code shared between producer and observer. It is the language, or the semiotic system, employed by them. We articulate a gesture in such a way that it will hopefully carry the right message. We try to produce something that looks like the target. When we do this we take into account both how we think the observer sees it, but we may also reflect on our production ourselves.

Wait a minute. I might be writing trivial stuff here. Is not the same statement about production for perception true for almost anything? Do we all try to be the perfect wife or husband, or do we try to look like it? And when my daughter of 4 is told to clean up, I think she is trying very hard to look busy, while minimizing actual effort.

Do we act to look like we should? (source)

My mentor Ans said we all carry part of the world inside us just as much as we are part of the world. We adjust our behaviour for others. Gestures are no exception. And so, the importance of perception is nothing special for gestures.

In comparison with studies of production it is an empty statement to say perception is more important. True, without an underlying perceptual specification there can be no efficient communication. But we may get to a shared code only after many tried productions. And without production there is nothing to perceive. I may think I know exactly what a tree falling in the woods should sound like, but without an actual tree falling, can there be a debate?

Is An Octopus’s Signal in his Shade?

Behold the Octopus. The James Bond of the sea. Ringo Starr once wrote:

I’d like to be
Under the sea
In an Octopus’s garden
In the shade

They pinned a romantic story on him about Octopi being aesthetic collectors. But an Octopus’s garden is no more than the leftovers, the bones, spines and shells, outside a den. Of course, a true romantic could counter that an Octopus picks aesthetic food…

The question I would ask is: does an octopus gesture? First, let us say that the term gesture originated in the distinction between voice and gesture, or oralité et gestualité as the French would say. The pair together refer to our total of communication means. It functions as a rough division of all the ways in which we create meaning for eachother. Since an octopus does not talk we could say that all his signaling behaviour falls under gesture. But that would be a fairly useless statement, unless we add that gesture is reserved to behaviour that is intended to communicate.


Make my day, punk.

At fUSION Anomaly, some gladly accept the linguistic status of the Octopus’s signaling. They even feel it is superior to our own language, which as we all know is strictly limited to our tiny mouths.

What do they do exactly? They change colour and texture (see explanation of chromatophores), they may use polarization of light. But why? Often the colour and texture changes are camouflage, but chey can also indicate arousal and/or threat.

I find it all rather amazing. But then I saw what cuttlefish can do…

Twenty Thousand Signs Under the Sea

If you’re something of a gesture fan you probably heard about diving signals aka underwater hand signals. The Active Divers Association (ADA) keeps pictures of the essentials. Other than that there’s not a lot to check out on the web.

(img source: ADA)

Now here’s the story. Some Dutch divers put up their own creative contributions to diving signals. Their playful sign inventions nicely highlight the way we go about gesturing in our daily lives. I recently invented a joke gesture with my friend A. It was a challenge best translated as ‘do you need dick extension?’. It was done by running the index finger down the nose. We’ll keep the story behind it secret, so we’ll be able to use it as a side joke.

Look Ray, a Fish!

Are diving signals exceptional? Are those Dutch divers, or A. and me, doing something extraordinary? I doubt it. I believe any bunch of people put in the right circumstances will invent gestures, a gesture system, or if so required a sign language. Monkey see, monkey do. Man can say, man can sign.

It’s the Gestures that Get to You

Ever found yourself deeply moved as you watched other people? What was it that got to you? Was it what they said? Or what they did? Or was it the fact that behind the simple acts (speech or otherwise) you saw the depth of meaning being conveyed?

I will bet you a euro for a euro-dime that it’s the ‘higher meaning’ of a simple act that had the capacity to move you. It’s what we call a nice gesture, a gesture of good faith, a gesture of support‘, or a gesture of love, loyalty, forgiveness, etc.

Here’s a nice story of how a simple act of waving a raised finger to a song became a grand gesture of allegiance to a school, and how it will be sorely missed. It’s a lot like Hook ’em Horns for the University of Texas.

Russian Icons

Peirce, a semiotician, wrote about signs (of any sort) as being either symbol, index or icon. How does this relate to Russian icon paintings, such as this one:

I saw many such paintings when I was in Russia (Moscow region), where I also had a look at Russian gestures. It appears like the gestures of the figures are very much prescribed for each icon. The blessing with two fingers is most common. But there are many, which I hope in due time to have a better look at. Unfortunately, that’s it for now, but feel free to comment…

A Gesture that Touched a Nation

Here’s a nice story (pdf) about two baseball players, Pee Wee Reese standing by Jackie Robinson in the face of racist fan-abuse.

Reese’s action unites the two levels of gesture: first in the narrow sense there is the actual action of ‘standing by someone’, a simple form-meaning pair of standing beside someone and embracing his shoulders that means to communicate ‘I am with you/this person’, this is the gesture-simple. Second, there is the high-level message. The act in it’s context: a white player showing his support for a black player who faces racism in a sport that was then (1947) considered ‘white’, in a racist society. The meaning: “I (PW Reese) admire Robinson as a man and a ballplayer and do not tolerate this racism” or “The Dodgers will stand together” or whatever the reader makes of it (there is a larger beholder’s share at a higher level, I think). This second act-meaning pair is the gesture-complex.

Pen Gestures

Those who study them like to call them Pen Gestures. The best known example is perhaps the Graffiti on Palm, which a helpful chap managed to implement in a Flash version using his own pattern recognition for Flash. Sharon Oviatt is the great champion of multimodal HCI with pen and speech interfaces. A tell-tale sign of the trouble this technique has to tackle is given in the Palm UI. Graffiti is used to ‘type’, not to ‘command’. For commands, the UI provides tap-buttons.


Some Examples (source: InkGesture by Jumping Minds)

Pen gesturing may well be a form of semiotically challenged HCI. Graffiti is actually writing in camouflage. Tapping buttons equals mouse clicks equals practical actions. Symbolical pen strokes that convey commands or modulate them come closest to being a ‘movement intended to communicate’. But then to whom? If its to the computer, then what is the message? The message is that a certain command is to be executed. Nothing new. I see no a priori advantage for ‘pen gestures’ over clicking buttons or even a DOS-prompt. All that matters are the same old pro’s and con’s of UI means; of usability in a given task with given users.

With some leniency any and all of the above actions can be seen as a communicative act towards the computer. In this sense Human-Computer Interaction is always a communicative dialogue. But I don’t like it like that. I like to think there is a difference between talking and gesturing to addressees (be they computer or not) and using them as tools. And I’ll make this promise: At the first sign that my computer is actually interest in what I have to say, I will tell it all my dreams and ambitions for his future.

Page 2 of 2

Powered by WordPress & Theme by Anders Norén