Various enterprises and personal interests, such as Man-Machine Interaction (MMI), gesture studies, signs, language, social robotics, healthcare, innovation, music, publications, etc.
Control a Beamed Powerpoint Presentation with Gestures
These students appear to have created a gesture based application that we also considered about four years ago. I know IBM and Philips were interested in this sort of application. So, well done guys! And excellent presentation too. I think they managed to make the best of it, given a difficult application.
Why is a presentation system a difficult application? Well if someone is presenting he will usually gesture during talking. These gestures are directed at the audience and not at the presentation software. So, the first task of such a system is to discriminate between those gestures: what is for me and what is for the audience. Furthermore, a presenter may also be fidgeting during his talk which shouldn’t be interpreted as a gesture. Unfortunately, it is unclear whether these students treated these issues.
The things they did do seem to be designed well enough. I think I like the calibration they designed: It creates a connection between the user’s physical environment and the camera he must address. It grounds the interaction. The subsequent examples of the functionality they have built in is less impressive. The forward-back commands are okay, but the drawing and highlighting are not very valuable in my opinion. People in the audience can see that you are pointing at something so there is perhaps little need to do more. But maybe these are first steps which need a bit more maturity in their interaction design to become useful.
A wonderfull new video on YouTube of two guys (programmers, it says) talking and ‘co-speech-gesturing’ (is that a verb?).
“Real programmers use sign language” (by ekabanov)
I think it is safe to assume that it is for real. Their whole behaviour looks too natural and wacky to be scripted.
I also think this is a great case study to spend some time on while discussing some of the ideas of David McNeill. Because what we have here is what his theories and ideas are concerned with. There is (of course) no sign language nor did I spot any other ’emblematic gesture’ (those vulgar things you get fined or jailed for or the goofy ones that seem to be must-haves for ad campaigns). I also do not see any pantomime. No, this is the stuff they like in Chicago: Co-speech gestures. An episode full of deictics, beats, iconic and metaphoric gestures, right?
From the McNeill lab: A misconception has arisen about the nature of the gesture categories described in Hand and Mind, to wit, that they are mutually exclusive bins into which gestures should be dumped. In fact, pretty much any gesture is going to involve more than one category. Take a classic upward path gesture of the sort that many speakers produce when they describe the event of the cat climbing up the pipe in our cartoon stimulus. This gesture involves an iconic path-for-path mapping, but is also deictic, in that the gesture is made with respect to an origo –that is, it is situated within a deictic field. Even “simple” beats are often made in a particular location which the speaker has given further structure (e.g. by setting up an entity there and repeatedly referring to it in that spatial location). Metaphoric gestures are de facto iconic gestures, given that metaphor entails iconicity. The notion of a type, therefore, should be considered as a continuum –with a given gesture having more or less iconicity, metaphoricity, etc.
Wrong! Apparently the main problems of McNeill’s typology of gestures, that has sent many an engineer on a wild goose hunt for iconic gestures, are now even recognized at the source (McNeill, 1992). It is not mutually exclusive but rather an index of the functioning of a gesture (‘as a beat’ – ‘through spatial reference (deictic)’ – ‘referring thorugh iconicity to something concrete’ – ‘referring via iconicity first to something concrete and second through metaphor to something abstract’). Good. I never liked ‘beats’ for example. I don’t think I ever saw one. But to say that it was a misconception… I vaguely recall an annotation procedure called the ‘beat filter’ that begs to differ.
Anyway, at least this clears up the discussions regarding ‘metaphoric gestures’ considerably [they are de facto also iconic, the metaphor functions on another level]. And it also clears the way for an annotation of this video. Any volunteers? Well, you would have to get a decent file of the movie instead of the YouTube flash stuff anyway, so let’s forget about it.
McNeill wrote a new book recently (2005) which is mostly about growth points. But before you read the summary by McNeill you might want to check Kendon’s brilliant poem called ‘The Growth Point‘, which he delivered at McNeill’s festen. I find it neatly captures my feelings towards growth points (and more that is beyond my grasp). I am at once awed, baffled, and stupefied when I read about growth points and catchments.
And so it goes. Again I tried to get it. Again I failed to learn anything from reading about growth points. One thing only. If David McNeill (or Susan Duncan) is right, then annotating gestures in episodes like this will be eternal hell 🙂 And without the speech it will not work. Thank God. I can go to bed with a clear conscience.
Books:
McNeill, D. (Fall 2005) Gesture and Thought. Chicago: University of Chicago Press.
McNeill, D. (2000) (Ed.). Language and Gesture. Cambridge: Cambridge University Press.
McNeill, D. (1992). Hand and Mind. Chicago: University of Chicago Press.
Sunday the French will cast their final votes and get a new president: either Sarkozy or Royal. On the big final TV debate there was one ‘gesture’ that received considerable attention. Segolene pointed her finger repeatedly at Sarkozy while accusing him of something.
Watch out for the scene that starts around 4:00 in this video and lasts for a couple of minutes (or at 6:50 in case of counting back in time). In her main accusation Mme Royal first points her finger about 14 times, then makes a wave-away gesture three times and continues with another 5 or 6 angry points.
MSN: The highlight came as Ms Royal said it was scandalous that Mr Sarkozy could talk with a tear in his eye of giving handicapped children an enforceable right to schooling, when his government had scrapped a similar measure she had introduced as schools minister. The centre-right favourite replied: “Calm down. Don’t point your finger at me like that. I don’t know why Ms Royal, usually so calm, has lost her nerve…You have shown how easily you get angry. But to be president of the republic carries heavy responsibilities.” Ms Royal hit back, saying: “Not when there is injustice. There is some anger that is perfectly healthy.”
A more literal translation for good measure:
CNN: Highlights from the showdown: SARKOZY: “Calm down, don’t point at me with your finger like that.” … ROYAL: “No, I won’t calm down.” SARKOZY: “To be president you have to be calm.” ROYAL: “Not when there is injustice. There is anger that is perfectly healthy… I won’t allow the immorality of political speeches to gain the upper hand.” SARKOZY: “I don’t know why Madame Royal, who is usually calm, has lost her cool.” ROYAL: “I have not lost my cool, I’m angry. It’s not the same, don’t be contemptuous, Mr Sarkozy.” … SARKOZY: “I am not calling into question your sincerity, Madame Royal, don’t call into question my morality. And with that, Madame, the dignity of the presidential debate will be preserved. “But at least it’s served one purpose, which is to show that you get angry very quickly, you go off the rails very easily, Madame. A president is someone who has important responsibilities.”
I saw an old debate between Francois Mitterand and Giscard d’Estaing on TV in 1981. A similar situation arose. The socialist claimed the moral high ground and the conservative said that Mitterand did not have the monopoly on compassion: “Vous n’avez pas le monopole du coeur”. It was an important moment. Maybe this small scene will be remembered as well?
Perhaps gesture and speech are somehow wired together in the brain. What is certain however is that in US brains everything is wired to politics come election time.
