Various enterprises and personal interests, such as Man-Machine Interaction (MMI), gesture studies, signs, language, social robotics, healthcare, innovation, music, publications, etc.

Category: Expressive Gestures

Avatar Kinect


Introduction to Avatar Kinect by Microsoft.

Avatar Kinect is a new social entertainment experience on Xbox LIVE bringing your avatar to life! Control your avatar’s movements and expressions with the power of Avatar Kinect. When you smile, frown, nod, and speak, your avatar will do the same.

Ah, new developments on the Kinect front, the premier platform for Vision based human action recognition if we were to judge by frequency of geeky news stories. For a while we have been seeing various gesture recognition ‘hacks’ (such as here). In a way, you could call all interaction people have with their Xbox games using a Kinect gesture recognition. After all, they communicate their intentions to the machine through their actions.

What is new about Avatar Kinect? Well, the technology appears to pay specific attention to facial movements, and possibly to specific facial gestures such as raising your eye brows, smiling, etc. The subsequent display of your facial movements on the face of your avatar is also a new kind of application for Kinect.
 

The Tech Behind Avatar Kinect

So, to what extent can smiles, frowns, nods and such expressions be recognized by a system like Kinect? Well, judging from the demo movies, the movements appear to have to be quite big, even exaggerated, to be handled correctly. The speakers all use exaggerated expressions, in my opinion. This limitation of the technology would certainly not be surprising because typical facial expressions consist of small (combinations of) movements. With the current state of the art in tracking and learning to recognize gestures making the right distinctions while ignoring unimportant variation is still a big challenge in any kind of gesture recognition. For facial gestures this is probably especially true given the subtlety of the movements.


A playlist with Avatar Kinect videos.

So, what is to be expected of Avatar Kinect. Well, first of all, a lot of exaggerating demonstrators, who make a point of gesturing big and smiling big. Second, the introduction of Second Life styled gesture routines for the avatar, just to spice up your avatars behaviour (compare here and here). That would be logical. I think there is already a few in the demo movies, like the guy waving the giant hand in a cheer and doing a little dance.

Will this be a winning new feature of the Kinect? I am inclined to think it will not be, but perhaps this stuff can be combined with social media features into some new hype. Who knows nowadays?

In any case it is nice to see the Kinect giving a new impulse to gesture and face recognition, simply by showcasing what can already be done and by doing it in a good way.

Eila Goldhahn

Eila Goldhahn:
What can be learnt from the MoverWitness Exchange for the development of gesture-based human-computer interfaces?

Goldhahn holds a very cloudy talk about people being movers and witnesses, holds up Durer’s famous woodcut of perception drawing. I am totally missing the point. We the engineers should all be ‘movers’ as well? So we can share a more embodied knowledge with each other or with our ‘subjects’. Really, no idea what she is trying to get at. But it must be my limited engineer’s point of view or something.

Fortunately she is going to show us some videos. Perhaps it will become clearer now.
– A man is licking a wall, apprently enjoying a very deep sensory haptic embodied experience…
– A woman is looking like she needs to go to the bathroom…
– Ah, a nice one with people falling/flying. She mentions how associations and imagination can play a role in our perceptions (really?) and how these can mediate between the mover and the witness. Good point.

Asked about a more concrete example of what is missing in ‘our methods’ she points out how, in the talk by Stoessel on the elderly, how they could have engaged the movements of the elderly in a more open way. One could let the elderly talk about how they had experienced the movement and then see if this coincides with the ‘witness’s observation of the movement. Hmm, interesting.

Gesture and Emotion

Let us broaden our horizon.
Let me turn your gesture perspective to new topics (well, revisited actually, see here).
Let us ponder emotion.

PrEmo
Pieter Desmet created the PrEmo method and interface (source)

Several colleagues here are studying ‘design and emotion’. One of the methods developed to evaluate people’s emotional response to product is PrEmo. It shows a little guy in different pictures that represent different emotions. The trick is, however, that they are not pictures but animations with sounds. So, the little guy makes facial expressions, gestures, and some exclamations. Of course, the question immediately comes up how reliably these gestures represent the intended emotions. Does everyone see it the same way? Apparently, Gael showed me some results, people do see it the same way for most of the pictures. Yet, some of them, such as surprise, are not see reliably perceived.

Sadly, the animations, which are in flash, are not available publicly. I understood there is licensing involved, and you have to see them to be able to really evaluate the gestures.

If you want to read more about emotion and how it can be measured: The Design & Emotion Society is quite a useful resource. You can register as a member for free and then they provide a good knowledge base. Another site is the HUMAINE Portal. With them you have to pay a small amount.

Air Guitar Toy by Mannak

Ronald Mannak, a former colleague, is now developing toys at his own 1uptoys. His toys at hand are the SilverLit V-Beat AirDrums, AirGuitar and BoomBox. This week our university’s ‘newspaper’ has an interview with him: Luchtgitaar met Geluid. And here he is in a video demonstrating his AirGuitar:

Still a long way to go before he can try for world champion airguitar, I think. But the product is interesting to consider. At first I thought it looked quite nice and cool. But then I wondered: why would anyone want to actually have an AirGuitar? Isn’t the point of playing air guitar that you don’t have to have the damn thing? If I am going to buy something to play guitar I might as well, or even better, buy a real (toy) guitar, right?

Is this going to be cheaper than a real guitar? I would guess that the additional electronics will not be cheaper than the bits of extra wood, metal or plastic needed for a physical guitar. But then again, microelectronics can be cheap if they are sold in large quantities.

So, is this going to provide a better experience? I think that by definition that is impossible. The point of playing air guitar is to imitate the actual playing, to go thorugh the motions and almost ‘feel like’ you are really playing. In other words, it can never be better than the real thing, or can it?

Maybe it can. Maybe it can help people who can not play guitar ‘feel more like’ they are playing guitar. Maybe the AirGuitar can take care of the difficult stuff like putting your fingers in the right position on the strings and remembering the chords and licks, and leave the exciting stuff to you, like strumming wildly, creating vibrato or smashing it.

That would be neat, Ronald if you read this, can you make it so it can be smashed?

Art of Gesture on Stage

Here is nice article on the art of gesture in theatre: Music students help revive the art of Baroque gesture.

Paris Judgment
Reviving an ancient art: students from the University’s Faculty of Music worked with theatre director Helga Hill to present a fully-staged and gestured season of Eccles’ The Judgment of Paris: Above, Paul Bentley as Paris and Janelle Hopman as Venus. [Photo: Mark Wilson] (source)

Johann Jakob Engel (DE) wrote in a very interesting way about gestures, especially in Ideen zu einer Mimik. From the perspective of actors on stage, he analyzed how gestures function.

I read only the paper by Sara Fortuna (2003) in Gesture: Gestural expression, perception and language. A discussion of the ideas of Johan Jakob Engel. It is intriguing reading material. A bit difficult to summarize in a few sentences here, so I will not try. An open mind, keen on philosophical musings is a good companion while chewing on Engel’s thoughts.

If we go further back in time, the work of Quintillian (and Cicero) is related. They wrote for orators, which were actors as much as they were politicians and lawyers. Wittgenstein is also referenced a lot.

Powered by WordPress & Theme by Anders Norén