Category: Musical Gestures

Imaginary Gestures for Music Therapy?

On September 20, 2010

In Mirror Neurons, Musical Gestures, Psychobabble

A while ago I received an inspiring email with an interesting question, see below (and btw, I would encourage anyone reading my blog to start corresponding, which I greatly enjoy). I will try to respond here, sharing my thoughts not only with Paulo Suzuki, but with anyone interested.

Hi Jeroen,

I’m an IT professional and Music Therapist.

I’m working on a research using the Theremin as a music therapeutic tool and
I’d like that you recommend a book (basic or introdutory) regarding
“gestures” so that I can use to justify my issues.

My research is oriented to music therapy patients with little or without
movements – and so I’d like to know if you are familiar with any research on
“imaginary gestures”?

Thanks for the time.

Sincerely,

Paulo Suzuki

PS: my 1st resource regarding “gesture” is your blog “Nice Gesture”; thanks
and congratulations.

Before I give you some answers, I would like to point to a prior post about Musical Gesturing with a Theremin and other posts in the Musical Gestures category. Looking back on that post, I would like to add that I believe that the feelings that music arouse in us are as real as we allow them to be, which is ‘very real’ for many people. Undoubtedly, this can be put to many positive uses, such as ‘Musical Therapy’. However, if you are in this business then I think it is important to remember that you are working for a large part with created conventions that need to be learned and accepted by your ‘patients’. Granted, there may be some feelings or ‘gestures’ that certain music has, in and of itself. For example, ‘beats per minute’ may be related to a slower or faster beating of the heart, and therefore to our arousal or activation. However, trusting too much in the emotional values of certain music quickly becomes speculative, highly subjective, and therefore not advisable (unless you either can rely on or have the time to build up a library of shared conventions with your audience, or patients).

So, to answer the first question, I would like to recommend two books:

Gesture, Visible Action as Utterance, by Adam Kendon (the basics about gesture and how humans use gestures).
Music and Gesture, by Gritten & King (Eds.) (I haven’t read this one myself, but from the contents it looks like it covers the topics you might be interested in).

To answer your second question, regarding research on ‘imaginary gestures’:

There is the work on ‘mirror neurons‘ that shows that there is a very close connection in our brains to ‘seeing a gesture’ and ‘producing a gesture’. That might be useful to you? A wild thought: if one wants patients to be able to play music with imaginary gestures perhaps it is possible to record their brains while watching a musician play a certain bit of music, identify the ‘brain activation’, and then watch for that when they ‘play in their minds’ themselves, translating their ‘imaginary gestures’ back to music. I’m taking a couple of shortcuts with this line of reasoning, but well.
Then there is the work on helping people who lost a hand or an arm. See for example this story: Robot arm ‘controlled with thoughts’. To some extent the nervous system may still process stimuli that pertain to certain hand actions, or gestures.
Another interesting bit of work was done a while ago by Sidney Fels: GloveTalk, a system that translates hand gestures to speech through an adaptive interface. Perhaps the micro gestures Fels uses, can also be translated to music instead of speech. Incidentally, his first ‘user’ was reported to be an experienced piano player…

Hope it helps 🙂

GW2009 Keynote: Antonio Camurri

By Jeroen

On February 27, 2009

In Academia, Gesture with Emotion and Feeling, Musical Gestures

Keynote: Antonio Camurri (also here)
Toward computational models of empathy and emotional entrainment

Casa Paganini – InfoMus – EyesWeb

Camurri has already done a lot of interesting work on movement and gesture, all of it in the ‘expressive corner’, working with dance and with music.

He just talked about a really nice application: He created a system to paint with you body movements, But it does so only if you move without hesitation. So, patients with hesitant movements (Parkinson?) get a stimulus to move better.

Next, about part of Humaine: something about the visibility of emotion in musical movements (not the sounds). There were previous talks in this area:

Florian Grond, Thomas Hermann, Vincent Verfaille & Marcelo Wanderley:
Methods for effective ancillary gesture sonification of clarinetists

Rolf Inge Godøy, Alexander Refsum Jensnius & Kristian Nymoen:
Chunking by coarticulation in music-related gestures

Next work with Gina Castellana (?): influence the way you listen to music through movement and gesture. Nice video.

There is also work on robotic interfaces. A ‘concert from trombone and robot’. Stockhausen, Milano. Robot had a radio, drove around, so spatially and in playing the robot had to be in tune with the trombone player. Collaboration with S. Hashimoto and K. Suzuki (Waseda University), See here for a publication.

He also worked together with Klaus Scherer from Geneva. Gael talked about Scherer’s work on the emotions as being quite good.

Camurri seems to be involved in many European networks and projects.

He is now explaining a project on synchronization. Quite interesting stuff about violin players (as cases of oscillators) try to get synchronized with a manipulated signal or with each other. It is going too fast to write much about it, but it all looks really nice. Violinists synchronizing their movements. And he is making much of a concept called ’emotional entrainment’. There is decent explanantion of the term here, but I’ll quote it:

A Quote by Daniel Goleman on emotional entrainment, influence, charisma, and power
Setting the emotional tone of an interaction is, in a sense, a sign of dominance at a deep and intimate level: it means driving the emotional state of the other person. This power to determine emotion is akin to what is called in biology a zeitgeher (literally, “time grabber”), process (such as the day-night cycle of the monthly phases of the moon) that entrains biological rhythms. For a couple dancing, the music is a bodily zeitgeber. When it comes to personal encounters, the person who has the more forceful expressivity – or the most power – is typically the one whose emotions entrain the other. Dominant partners talk more, while the subordinate partner watches the others face more – a setup for the transmissions effect. By the same token, the forcefulness of a good speaker – a politician or an evangelist, say – works to entrain the emotions of the audience. That is what we mean by, “He had them in the palm of his hand.” Emotional entrainment is the heart of influence.
Daniel Goleman : Harvard PhD, author, behavioral science journalist for The New York Times
Daniel Goleman
Source: Emotional Intelligence: Why It Can Matter More Than IQ, Page: 117

Interesting remark about the violinists who synchronize with an adjusted signal: They did not hear their own sound but rather a manipulation of the pitch of the movement. So what they did did not match what they heard. At some point these players got motion sickness…

Now there is a weird video from the opera, where a man and a woman use a chair to communicate (?). He lost me there for a moment.

Announcement: eNTERFACE 2009, European Workshop on Multimodal Interfaces, 13 July – 7 Aug, Casa Paganini (here)

Questions:
– About publications: you can download them from ftp.infomus.org/pub/camurri

Air Guitar Toy by Mannak

By Jeroen

On September 24, 2007

In Expressive Gestures, Motion Sensing, Musical Gestures

Ronald Mannak, a former colleague, is now developing toys at his own 1uptoys. His toys at hand are the SilverLit V-Beat AirDrums, AirGuitar and BoomBox. This week our university’s ‘newspaper’ has an interview with him: Luchtgitaar met Geluid. And here he is in a video demonstrating his AirGuitar:

Still a long way to go before he can try for world champion airguitar, I think. But the product is interesting to consider. At first I thought it looked quite nice and cool. But then I wondered: why would anyone want to actually have an AirGuitar? Isn’t the point of playing air guitar that you don’t have to have the damn thing? If I am going to buy something to play guitar I might as well, or even better, buy a real (toy) guitar, right?

Is this going to be cheaper than a real guitar? I would guess that the additional electronics will not be cheaper than the bits of extra wood, metal or plastic needed for a physical guitar. But then again, microelectronics can be cheap if they are sold in large quantities.

So, is this going to provide a better experience? I think that by definition that is impossible. The point of playing air guitar is to imitate the actual playing, to go thorugh the motions and almost ‘feel like’ you are really playing. In other words, it can never be better than the real thing, or can it?

Maybe it can. Maybe it can help people who can not play guitar ‘feel more like’ they are playing guitar. Maybe the AirGuitar can take care of the difficult stuff like putting your fingers in the right position on the strings and remembering the chords and licks, and leave the exciting stuff to you, like strumming wildly, creating vibrato or smashing it.

That would be neat, Ronald if you read this, can you make it so it can be smashed?

Lead Guitar Body Language

By Jeroen

On September 9, 2007

In Musical Gestures, Pantomime, Theater & Stage Performance

Here is the Air Guitar World Champion 2007, Ochi “Dainoji” Yosuke (Japan) performing at Air Guitar World Championships 2007, Oulu, Finland:

What a nice gesture performance: the pantomime, the gestures, the emotional expressions, the mimicry of the actual guitar play, and of course the dramatic gestures of a lead guitar player on stage. It makes me realize that a language may be found around in every hidden corner of human activity. In this case Dainoji shows a hilarious command of the body language of lead guitars.

It also makes me wonder what exactly would remain of ‘musical gestures‘, when all of a musicians ‘body language’ were hidden to the audience? I guess something would remain, and that would then be the real musical gesture.

Musical Gesturing with a Theremin

By Jeroen

On June 4, 2007

In Musical Gestures

Here is a man called Eliot Fintushel, who neatly exemplifies how people come to talk about the expression of feelings in music. Furthermore, because he is using ‘gestures’ to operate his Theremin he even talks about his feelings being present in his movement and therefore (magically) transferred to the music of the Theremin. So, if you make a ‘nervous’ movement, you get a ‘nervous’ sound.

It all sounds like a big circular argument: If you tell people that a certain movement is ‘nervous’ and then that the sound they hear is also ‘nervous’ then what will they see and hear? That’s right, ‘nervousness’.

But what if you had not told the audience what to hear or see? They would simply rely on the facial expressions (which are overacted and conventional rather than universally understandable in this case) to know the feelings they should be feeling. Projection will do the rest.

But like I said before, this is not necessarily a problem. If you wish to share a fantasy of going through all sorts of feelings, guided by a performer, then let that fantasy be as real as possible. Some of that feeling may actually be genuine, the rest may serve our emo-thirsty souls just as well.

Musical Gesturing with a Theremin

By Jeroen

On June 4, 2007

In Musical Gestures

To my ears, it all sounds like a big circular argument: If you tell people that a certain movement is ‘nervous’ and then that the sound they hear is also ‘nervous’ then what will they see and hear? That’s right, ‘nervousness’. But what if you had not told the audience what to hear or see? They would simply rely on the facial expressions (which are overacted and conventional rather than universally understandable in this case) to know the feelings they should be feeling. Projection will do the rest.

But like I said before, relying on conventions (even if they are shared with only a select few) to achieve some effect is not necessarily a problem. If a concert audience wishes to share a fantasy of going through all sorts of feelings, guided by a performer, then let that fantasy be as real as possible. Some of that feeling may actually be genuine, the rest may serve our emo-thirsty souls just as well.

Gesture as a Metaphor for Musical Expression

By Jeroen

On May 29, 2007

In Musical Gestures

Talking about gesture and metaphor often gets me into a critical state. I am not sure I agree with McNeill’s idea of opposing iconic and metaphoric gestures (just because they refer to something concrete or abstract respectively). I was also somewhat puzzled by the MPI Lecure on Metonymy in Metaphoric Gestures by Cienki and Mueller. And that was the sort of talking over lunch just now. One of the professors here, Paul Hekkert, is also interested in metaphors. He gave a masterclass at a workshop at the university of Tilburg, as part of their Advanced Studies Initiative on MultiModal Metaphor. We talked a bit about metaphor and gesture, and then… It struck me that in the experience of music people often use the term Musical Gestures, which I think is actually a metaphoric use of the word gesture. As far as I can tell there is no actual gesturing involved, there are just musicians playing their instruments. But somehow, when they play the music in a certain and when the audience picks that up, people start to talk about musical gestures. That would mean that the music played is understood in terms of a gesture. What is a gesture then? Well, let us say now that gesture is the display, through any action, of an intention to communicate. Musical score
A dramatic gesture of “Maternal Love” ? (source)

Going back to the music, we can then say that with a musical ‘gesture’, the musician displays an intention to communicate (his feelings?) through his playing. I guess we should add: ‘beyond that which is displayed through the usual playing of music’. And maybe we should also add: ‘or at least insofar as such an intention is perceived by an audience’. Perhaps it would even be more accurate to add: ‘insofar as an audience chooses to project such intentions to communicate on the performance’. For as far as I can tell nothing is stopping people in the audience from projecting the grandest displays of feelings on the tiniest blowing of a flute. It actually does not matter whether the musical gestures are real (produced and perceived) or imaginary (projected but not actually produced). If I have paid 50 euro to attend a concert I have every right to experience the music in whatever way suits me best.

Alexander Jensenius’ Musical Gestures

By Jeroen

On February 23, 2007

In Musical Gestures

I found a fellow blogger called Alexander Refsum Jensenius who has a nice collection of posts about gestures. His main interest is in Musical Gestures and the technology to capture them. There are also some nice thoughts and pictures regarding the nature of gesture (that I could not resist commenting on). He made a nice illustration of the movement phases of a gesture unit according to Kendon.

Illustration of Kendon's Gesture Phases
Source: ARJ: Action/Gesture Units

And he is also trying to include fidgeting into his ‘movement-flux diagram’. I wonder how far he can get in capturing the movement possibilities in such pictures? Anyway, it is nice to find another PhD-student thinking about similar topics.

Update 24 feb: Jensenius’ site seems to be down, together with the picture, alas.