Category: Gesture Recognition Page 1 of 3

Avatar Kinect

On May 20, 2011

In Expressive Gestures, Facial Expressions, Gaming, Gesture Recognition, Motion Sensing

Introduction to Avatar Kinect by Microsoft.

Avatar Kinect is a new social entertainment experience on Xbox LIVE bringing your avatar to life! Control your avatar’s movements and expressions with the power of Avatar Kinect. When you smile, frown, nod, and speak, your avatar will do the same.

Ah, new developments on the Kinect front, the premier platform for Vision based human action recognition if we were to judge by frequency of geeky news stories. For a while we have been seeing various gesture recognition ‘hacks’ (such as here). In a way, you could call all interaction people have with their Xbox games using a Kinect gesture recognition. After all, they communicate their intentions to the machine through their actions.

What is new about Avatar Kinect? Well, the technology appears to pay specific attention to facial movements, and possibly to specific facial gestures such as raising your eye brows, smiling, etc. The subsequent display of your facial movements on the face of your avatar is also a new kind of application for Kinect.

The Tech Behind Avatar Kinect

So, to what extent can smiles, frowns, nods and such expressions be recognized by a system like Kinect? Well, judging from the demo movies, the movements appear to have to be quite big, even exaggerated, to be handled correctly. The speakers all use exaggerated expressions, in my opinion. This limitation of the technology would certainly not be surprising because typical facial expressions consist of small (combinations of) movements. With the current state of the art in tracking and learning to recognize gestures making the right distinctions while ignoring unimportant variation is still a big challenge in any kind of gesture recognition. For facial gestures this is probably especially true given the subtlety of the movements.

A playlist with Avatar Kinect videos.

So, what is to be expected of Avatar Kinect. Well, first of all, a lot of exaggerating demonstrators, who make a point of gesturing big and smiling big. Second, the introduction of Second Life styled gesture routines for the avatar, just to spice up your avatars behaviour (compare here and here). That would be logical. I think there is already a few in the demo movies, like the guy waving the giant hand in a cheer and doing a little dance.

Will this be a winning new feature of the Kinect? I am inclined to think it will not be, but perhaps this stuff can be combined with social media features into some new hype. Who knows nowadays?

In any case it is nice to see the Kinect giving a new impulse to gesture and face recognition, simply by showcasing what can already be done and by doing it in a good way.

Brilliantly funny ‘body language’ instructional tape

By Jeroen

On April 1, 2011

In Body Language, Gesture Recognition, Gesture Semiotics, Playing with Gestures

Here is a must-see video for anyone who is interested in gestures and body language and has a sense of humour. Be warned, it may force you to rethink some of your ideas about the conventionality of body language and the extent to which interpreting it can be taught (should you be a communications trainer).

In any case, it’s good for a laugh 🙂

Update:

Here is a collection of the sort of body language instruction that the above video is a parody of (with the exception of the fifth which again is a parody):

Gesture Recognition in the Operating Room

By Jeroen

On February 7, 2011

In Gesture Recognition, Healthcare, Speech Recognition

In the news recently: Future Surgeons May Use Robotic Nurse, ‘Gesture Recognition’.

ScienceDaily (Feb. 3, 2011) — Surgeons of the future might use a system that recognizes hand gestures as commands to control a robotic scrub nurse or tell a computer to display medical images of the patient during an operation.

Purdue industrial engineering graduate student Mithun Jacob uses a prototype robotic scrub nurse with graduate student Yu-Ting Li. Researchers are developing a system that recognizes hand gestures to control the robot or tell a computer to display medical images of the patient during an operation. (Credit: Purdue University photo/Mark Simons)

I have noticed similar projects earlier, where surgeons in the OR were target users of gesture recognition. The basic idea behind this niche application area for gesture recognition is fairly simple: A surgeon wants to control an increasing battery of technological systems and he does not want to touch them, because that would increase the chance of infections. So, he can either gesture or talk to the machines (or let other people control them).

In this case the surgeon is supposed to control a robotic nurse with gestures (see more about the robotic nurse here). You can also view a nice video about this story here; it is a main story of the latest Communications of the ACM.

Well, I have to say I am in doubt if this is a viable niche for gesture recognition. So far, speech recognition has been used with some succes to dictate operating reports during the procedure. I don’t know if it has been used to control computers in the OR. Frankly, it sounds a bit scary and also a bit slow. Gesture and speech recognition are known for their lack of reliability and speed. Compared to pressing a button, for example, they give more errors and time delays. Anything that is mission-critical during the operation should therefore not depend on gesture or speech control would be my opinion.

However, the real question is what the alternatives for gesture or speech control are and how reliable and fast those alternatives are. For example, if the surgeon has to tell another human what to do with the computer, for example displaying a certain image, then this can also be unreliable (because of misinterpretations) and slow.

The article mentions several challenges: “… providing computers with the ability to understand the context in which gestures are made and to discriminate between intended gestures versus unintended gestures”. That sounds like they also run into problems with fidgeting or something similar that surgeons do.

In sum, it will be interesting to see if surgeons will be using gesture recognition in the future, but I wouldn’t bet on it.

Thesis of Mats Andrén

By Jeroen

On October 22, 2010

In Academia, Gesture Perception, Gesture Recognition, Gesture Semiotics

For those among you who are interested in good gesture research it may be of interest to know that Mats Andrén, from Lund University, has published (online) his thesis called Children’s Gestures from 18 to 30 months. So far, I am enjoying reading it very much. Good job, Mats, and good luck with the defense 🙂

Golden Oldie: Garfield & the Waving Snowman

By Jeroen

On June 12, 2009

In Gesture Recognition, Gesture Semiotics, Seeing Intentions

Boy, I have been looking for this cartoon for two and a half years (off and on, that is, see here). So, a big thanks to diamond-blade for pointing out the link.

Garfield and the Waving Snowman
It was the Garfield daily comic from February 27, 2005 (source)

The reason why I like this comic so much (besides my general fascination for the gestures in Garfield, see these posts) is that it is a wonderful illustration of an idea that I would love to test experimentally. The idea is that certain movements (i.e. having certain kinematic characteristics) might automatically trigger the perception of a gesture (in the sense of a movement that is intended to communicate). This idea is not new and was, for example, described by Adam Kendon in his 2004 book ‘Gesture. Visible action as utterance’. But this idea is also present, to some extent, in the work of Gunnar Johansson (1973) and Albert Michotte.

Prelimary ideas about experiments:

Generate a randomized display of motion (within some likely parameter space) and let people all watch that same fragment, then check if they see a gesture at the same time.
Condition: Manipulate it in such a way that it is easier or harder to imagine seeing a hand.
Condition: Create conditions where subjects feel it is likely or unlikely they will be insulted (perception of intention to communicate appears linked to sensitivity to insults, see my thesis or Bucci et al. 2008).

Opera Fool’s Day Face Gestures

By Jeroen

On April 7, 2009

In Facial Expressions, Gesture Recognition, Playing with Gestures

Check out the hilarious joke from the folks at Opera Labs. I will let it speak for itself.

Finally, a soldier who doesn’t think

By Jeroen

On March 12, 2009

In Defense & Military, Gesture Recognition, Robots

“Should I stay or should I go?” That seems to be the only concern of the robot that features in a demo movie made by researcher at Brown University. It takes its gestured orders in a way that is easily associated with the way soldiers on a patrol would gesture to ‘stop’ or to ‘go’ to the next soldier in the line. Or at least it looks similar to a man of my limited experience. And having DARPA as a main sponsor also helps the association.

A first impression

It looks like they have done quite a good job with the robot, given the current state of gesture recognition. I especially like it that people don’t have to wear sensors. This is achieved in part by using a depth camera (the CSEM Swiss Ranger) Besides that, the recognition of individuals does still seem to be a bit shaky, since you appear to have to show you face quite clearly before it sees who you are (but then again, given current face recognition technology that is no surprise either, and they have actually done a nice job of getting it up and running in the first place).

The contributors:
Chad Jenkins, Brown University
Matthew Loper, Brown University
Nathan Koenig, USC
Sonia Chernova, former Brown graduate student
Chris Jones, iRobot

Technovelgy posted a good description (here) of the system and some first impressions and associations (lol @ the HHGTTG quote).

A movie of the robot in action, following people and taking orders by gesture (here).

Christian Stoessel Helps the Elderly

By Jeroen

On February 27, 2009

In Gesture Recognition, Touch Gestures

Christian Stoessel, Hartmut Wandke & Lucienne Blessing:
Gestural interfaces for elderly users: Help or hindrance?
Publications here

Christian starts out by pointing towards the changing demographics (Dutch ‘vergrijzing’). There will be many elderly people. Or perhaps better, many people above 65 years, because these people may be more healthy in body and mind than previous generations of elderly (is that true?).

There is a somewhat optimistic view of the potential of gesture technology, in the sense that he thinks it is possible to identify sets of ‘intuitive’ gestures in a gestural interface. In the end he is measuring accuracy with which people performed gestures, rather than if they were intuitive or not.

Regarding the reality of creating a set of ‘intuitive’ gestures. He expands nicely on ‘intuitiveness’ as being something that is fuzzy. They don’t mean a gesture is intuitive from the start, but perhaps it will be more easy to remember.

Hitachi shows off Canesta’s 3-D depth camera for a gesture-controlled TV

By Jeroen

On January 12, 2009

In Gesture Recognition, YouTube Faves

Canesta showed off a demo at the International Consumer Electronics show in Hitachi’s booth. Basically, the 3-D depth camera can detect your movements. Hitachi used this system to create a gesture-controlled TV. You don’t need a remote control.

Demo of Orange Vallee’s gesture-controlled TV, with Softkinetics technology

By Jeroen

On January 12, 2009

In Gesture Recognition, YouTube Faves

This video from the 2009 International Consumer Electronics Show shows how you can control your TV using hand gestures that are detected by the 3-D camera atop the TV. Softkinetics does the software, a Swiss company does the depth camera, and Orange Vallee will deploy it in its interactive TV network