Tag: GW2009 Page 1 of 2

Eila Goldhahn:
What can be learnt from the MoverWitness Exchange for the development of gesture-based human-computer interfaces?

Goldhahn holds a very cloudy talk about people being movers and witnesses, holds up Durer’s famous woodcut of perception drawing. I am totally missing the point. We the engineers should all be ‘movers’ as well? So we can share a more embodied knowledge with each other or with our ‘subjects’. Really, no idea what she is trying to get at. But it must be my limited engineer’s point of view or something.

Fortunately she is going to show us some videos. Perhaps it will become clearer now.
– A man is licking a wall, apprently enjoying a very deep sensory haptic embodied experience…
– A woman is looking like she needs to go to the bathroom…
– Ah, a nice one with people falling/flying. She mentions how associations and imagination can play a role in our perceptions (really?) and how these can mediate between the mover and the witness. Good point.

Asked about a more concrete example of what is missing in ‘our methods’ she points out how, in the talk by Stoessel on the elderly, how they could have engaged the movements of the elderly in a more open way. One could let the elderly talk about how they had experienced the movement and then see if this coincides with the ‘witness’s observation of the movement. Hmm, interesting.

Christian Stoessel Helps the Elderly

By Jeroen

In Gesture Recognition, Touch Gestures

Christian Stoessel, Hartmut Wandke & Lucienne Blessing:
Gestural interfaces for elderly users: Help or hindrance?
Publications here

Christian starts out by pointing towards the changing demographics (Dutch ‘vergrijzing’). There will be many elderly people. Or perhaps better, many people above 65 years, because these people may be more healthy in body and mind than previous generations of elderly (is that true?).

There is a somewhat optimistic view of the potential of gesture technology, in the sense that he thinks it is possible to identify sets of ‘intuitive’ gestures in a gestural interface. In the end he is measuring accuracy with which people performed gestures, rather than if they were intuitive or not.

Regarding the reality of creating a set of ‘intuitive’ gestures. He expands nicely on ‘intuitiveness’ as being something that is fuzzy. They don’t mean a gesture is intuitive from the start, but perhaps it will be more easy to remember.

Frédéric Landragin Puts-That-There Again

By Jeroen

In Uncategorized

Frédéric Landragin:
Effective and spurious ambiguities due to some co-verbal gestures in multimodal dialogue
Publications here and here

Landragin talks about ‘put-that-there‘, the classic multimodal interface developed by MIT. He also developed a similar application.

He did his PhD in Nancy, but a Dutch Professor of Computational Linguistics, called Henk Zeevat, was one of his promotors. Zeevat is at the UvA… “The Institute for Logic, Language and Computation (ILLC) is a research institute of the University of Amsterdam, in which researchers from the Faculty of Science and the Faculty of Humanities collaborate”.

The content of his presentation revolves arounds a single idea that I find puzzling. He treats the transitionary movement between the that-deictic and the there-deictic as a gesture that says something about the manner in which ‘that’ is supposed to be ‘put’ ‘there’. I would contend that normally speaking no meaning resides in the transitional movement.

It starts getting interesting though, as he introduces ‘move that there’ as an indication that a path is intended with the transitional movement. I can imagine the difference between ‘put’ and ‘move’. Moreover, he says that the nature of ‘there’ depends on the nature of ‘that’. If ‘that’ is a carpet, then ‘there’ may be broad. If ‘that’ is a nail, then ‘there’ is probably quite precise. Good point, if you’ll excuse the expression.

GW2009 Keynote: Antonio Camurri

By Jeroen

In Academia, Gesture with Emotion and Feeling, Musical Gestures

Keynote: Antonio Camurri (also here)
Toward computational models of empathy and emotional entrainment

Casa Paganini – InfoMus – EyesWeb

Camurri has already done a lot of interesting work on movement and gesture, all of it in the ‘expressive corner’, working with dance and with music.

He just talked about a really nice application: He created a system to paint with you body movements, But it does so only if you move without hesitation. So, patients with hesitant movements (Parkinson?) get a stimulus to move better.

Next, about part of Humaine: something about the visibility of emotion in musical movements (not the sounds). There were previous talks in this area:

Florian Grond, Thomas Hermann, Vincent Verfaille & Marcelo Wanderley:
Methods for effective ancillary gesture sonification of clarinetists

Rolf Inge Godøy, Alexander Refsum Jensnius & Kristian Nymoen:
Chunking by coarticulation in music-related gestures

Next work with Gina Castellana (?): influence the way you listen to music through movement and gesture. Nice video.

There is also work on robotic interfaces. A ‘concert from trombone and robot’. Stockhausen, Milano. Robot had a radio, drove around, so spatially and in playing the robot had to be in tune with the trombone player. Collaboration with S. Hashimoto and K. Suzuki (Waseda University), See here for a publication.

He also worked together with Klaus Scherer from Geneva. Gael talked about Scherer’s work on the emotions as being quite good.

Camurri seems to be involved in many European networks and projects.

He is now explaining a project on synchronization. Quite interesting stuff about violin players (as cases of oscillators) try to get synchronized with a manipulated signal or with each other. It is going too fast to write much about it, but it all looks really nice. Violinists synchronizing their movements. And he is making much of a concept called ’emotional entrainment’. There is decent explanantion of the term here, but I’ll quote it:

A Quote by Daniel Goleman on emotional entrainment, influence, charisma, and power
Setting the emotional tone of an interaction is, in a sense, a sign of dominance at a deep and intimate level: it means driving the emotional state of the other person. This power to determine emotion is akin to what is called in biology a zeitgeher (literally, “time grabber”), process (such as the day-night cycle of the monthly phases of the moon) that entrains biological rhythms. For a couple dancing, the music is a bodily zeitgeber. When it comes to personal encounters, the person who has the more forceful expressivity – or the most power – is typically the one whose emotions entrain the other. Dominant partners talk more, while the subordinate partner watches the others face more – a setup for the transmissions effect. By the same token, the forcefulness of a good speaker – a politician or an evangelist, say – works to entrain the emotions of the audience. That is what we mean by, “He had them in the palm of his hand.” Emotional entrainment is the heart of influence.
Daniel Goleman : Harvard PhD, author, behavioral science journalist for The New York Times
Daniel Goleman
Source: Emotional Intelligence: Why It Can Matter More Than IQ, Page: 117

Interesting remark about the violinists who synchronize with an adjusted signal: They did not hear their own sound but rather a manipulation of the pitch of the movement. So what they did did not match what they heard. At some point these players got motion sickness…

Now there is a weird video from the opera, where a man and a woman use a chair to communicate (?). He lost me there for a moment.

Announcement: eNTERFACE 2009, European Workshop on Multimodal Interfaces, 13 July – 7 Aug, Casa Paganini (here)

Questions:
– About publications: you can download them from ftp.infomus.org/pub/camurri

Isabel Galhano Rodrigues

By Jeroen

Isabel Galhano Rodrigues:
Gesture choreography and gesture space in european vs. african portuguese interactions
her page.

She talked about cultural differences, regarding proxemics, regarding ‘touch gestures’.
Hmmm, she just showed two really nice videos, one with Portuguese students and one with Angolese students, who also talk Portuguese. They differ quite a lot in their gesturing, in the sense of their use of space.
Rodrigues points out that the Africans have little problem with gesturing close to the other people, whereas the Europeans appear to respect a large ‘body buffer’.

Questions: Were they representative of their ‘group’? Answer: this was just a first recording that I analysed.

Stuart Battersby on Interaction Studies Experiments

By Jeroen

In Academia, Interaction Studies

Patrick G. T. Healey, Chris Frauenberger, Marco Gillies & Stuart Battersby:
Experimenting with non-verbal interaction

For me personally, one of the more interesting posters was explained to me by Stuart Battersby (his page), of Queen Mary, University of London. He was working on theory from Interaction Studies, whcih I would also like to apply in robotics. He did not work on robotics but he wanted to create an experimental environment to test observations from interaction studies. This would be done with virtual environments where people are not communicating directly but through avatars. Then, he could tinker with their behavior without the participants knowing. Very clever.

Some literature he was using:
– Most of Kendon’s stuff (conducting interaction, stuff from the 90’s papers): F-frames
– Furuyama (here)
– Early work by Asli Özyürek (regarding spatial relationships)
– Older stuff from Goffman, Scheflen…

Do I Share Common Ground with Holler?

By Jeroen

Judith Holler & Katie Wilkin:
Gesture use on common ground
Her page.

Thank god, she starts off by stating that the grand theories about why we gesture are not necessarily mutually exclusive. Hear, hear!

Nice explanation about common ground. What is it and how do people build up common ground between them. And how people adjust their speech as they share more or less common ground. E.g. experts versus novices amongst each other. Nice reference to the 2004 Gerwing and Bavelas study.

Experiment:
Common ground versus non-common ground conditions. Surspirsingly, participants did not show a reduced gesture rate when they shared common ground, which was what Holler expected. Also, gesture did not become smaller or less informative. It stayed rather large and informative. This was perhaps because they were grounding the conversation, they would keep much information in gestures so the speech could become more ‘elliptical’ (?)

Yoshioka on Gesture in Different Languages

By Jeroen

Keiko Yoshioka:
Gestural reference to space by speakers of typologically different languages
some publication here and here

This talk was introduced by De Ruiter as a nice follow-up on Asli’s talk. Keiko is introducing Talmy’s theory about ‘sattelite-framed languages’ (English, Dutch, Chinese) versus ‘verb-framed languages’ (Spanish, etc). She explains how, in a verb-framed languages it is less easy to ‘compact Ground elements’. These are new concepts to me, and I have a hard time staying with her. She quotes Slobin a lot (1996) who showed that speakers of a verb-framed language allocate more attention to Ground, which shows in their preference for a certain rhetorical style.

She compared Dutch and Japanese speakers. The explained a story called ‘Frog, where are you’. She is also comparing head-front and head-final languages. She compares speech with and without gestures, and whether the ground was referenced in the verb phrase (VP) or the non-verb phrase. Damn, if only I had studied comparative linguistics better…

Take home message, because our languages differ, we differ in the placement and the content of our gestures.

Keynote by Asli Özyürek

By Jeroen

Keynote: Asli Özyürek
The role of gesture in production and comprehension of language: Insights from behavior and brain

Asli is giving a good keynote, presenting a good overview of current gesture theory. I do have some trouble to follow her presentation of two different ‘grand hypotheses’ about gestures and speech. She seems to polarize the work on gestures into two views, which may not be necessary in most cases. Poor old Krauss is still being held up as a straw man who thought gestures were not intended to communicate. In my own presentation I skipped this part, assuming everyone would already agree on gestures being movements intended to communicate, but it turns out that is not the case yet.

For example, Hedda is talking about fidgeting as ‘self-touch gestures’, disregarding the differences between movements that communicate and movements that are intended to communicate. Other people I talk too, are also sometimes questioning whether gestures are intended to communicate. Quite a surprise I must say. The overall level of knowledge of the current state of the work on gestures is surprisingly low. Kendon’s book of 2004, is not something you can rely on as a shared source of knowledge, for example. McNeill’s 1992 book is more or less common knowledge, but that includes some of the misconceptions that have arisen from that book. For example, people are not properly aware of the limited scope of McNeill’s 1992 book: It was solely about certain types of co-speech gestures. It was not about all gestures. And the difference is very important if you are talking, as I did, about emblems. Or if you consider, like Kendon (1995), the differences between emblems and other gestures, to be graded.

Anyway, Asli is using the opportunity to recount all the ‘Sylvester and Tweety Bird’ work that followed McNeill’s work. Bit by bit she is demonstrating the extent to which gestures and speech are intertwined. This is really a classic collection of work, performed by her and by her colleagues and other co-researchers.

Some quotes:
“gestural differences (in study 1) are not due to deep culture and language-specific representation”
“gesturers from all language backgrounds used an SOV order when asked to pantomime something without speech…. We sometimes refer to it as the cognitive deault language…”
“what can be packaged semantically in a clause determines the gesture… the effect is not absolute!”

She also presents the work on brain research, done at the FC Donders centre with Hagoort and Willems.
Quite a few questions from different directions.

References for Jérémie Segouat

By Jeroen