Michael Schutz - 
Michael Kubovy - 
Department of Psychology
University of Virginia
102 Gilmer Hall
Charlottesville, VA 22902
Popular version of paper 3pPP4
Presented Wednesday afternoon, June 6, 2007
153rd ASA Meeting, Salt Lake City, UT
Click here for a PDF version of this paper
Introduction
Are gestures--body movements--really part of a musical performance? Where does music exist? If a tree falls in the woods, does it make a sound?
Musicians, philosophers, and psychologists have been debating these questions for centuries without reaching a clear consensus on most of these matters. While purists may that argue music is an auditory experience and therefore visual information is irrelevant, the enormous investment in clothing, lighting, smoke machines and other visual effects in live concerts demonstrates audiences clearly respond to and appreciate visual information as part of the musical experience.
Our research focuses on one narrow element of this debate – do the gestures used by percussionists have any musical value irrespective of their acoustic consequences? Even trained musicians routinely disagree as to the role of gesture (e.g., the “up-down” motion used to strike a percussion instrument) in live performance. Eden Bailey, longtime percussionist with the New York Philharmonic, observed that, “[when] sharp wrist motions are used the only possible results can be sounds of a staccato nature. . . [When] smoother, relaxed wrist motions are used, the player will then be able to feel and project a smoother, more legato-like style.” Others, such as recent Percussive Arts Society Hall of Fame inductee Leigh Howard Stevens, are adamant that gesture length in and of itself is irrelevant, arguing they have “no more to do with [the] duration of bar ring than the sound of a car crashing is dependent on how long a road trip was taken before the accident.” How can musicians disagree so strongly on what appears to be such a simple question – does the gesture used to strike an instrument affect the length of the resulting note?
Unraveling the confusion depends in large part on understanding the distinction between sound (acoustics) and the way that sound is experienced by a listener (perception). Perception is not a perfect representation of the physical world. It is not uncommon to observe interactions between auditory and visual information such as the “ventriloquist effect” in which speech appears to originate from the moving lips of an inarticulate puppet. Therefore, in an effort resolve the debate over the role of gesture, we designed 2 studies to explore the contributions of auditory and visual information to music perception.
Experiment 1
We recorded a world-renowned percussionist performing notes using long and short gestures on a professional quality marimba (a percussion instrument similar to a xylophone), a sample of which is shown in Figure 1. Participants then rated the duration of each note twice: once with the gesture (audio-visual), and once without (audio-alone). In the audio-visual condition they were instructed to ignore visual information and base their ratings on the sound alone.
Figure 1: A world-renowned percussionist performed a series of notes using long and short gestures.
You can also view video of the performance: Long gesture and Short gesture.
As shown in Figure 2, although notes produced by long and short gestures were indistinguishable when presented as audio alone, these same notes were judged to be significantly different when presented with the corresponding gesture. This effect occurred despite instructing participants to ignore visual information when making their ratings. Our results indicate that while gesture fails to alter the sound of the note, it (serendipitously) alters the way the note sounds.
Experiment 2
The results of the first experiment suggest gestures play an important role in the musical experience. However, can we be sure the gesture is really changing the sound of the note – is it possible that this effect is due to confusion over the difficulty of making such judgments? After all, previous research strongly suggests visual information does not influence ratings of note length (Walker & Scott, 1981). How is it that videos of gestures affect auditory perception whereas other types of visual information do not?
We believe our unusual results are due to the clear relationship between the auditory and visual information presented. A visual impact clearly causes an auditory impact sound, therefore we predict our unusual results are limited to impact (a.k.a. percussive) sounds. In order to test this idea, we ran a second experiment using a variety of instrument timbres.
As the piano is technically a percussive instrument in that sound is produced from a hammer striking a string, we predicted impact gestures should influence the duration of piano notes. However, as sounds from non-percussive instruments could not be produced by impact gestures, we anticipated gestures would not influence perception.
This prediction was tested by pairing the original videos with notes produced by the marimba (original sound) as well as notes produced by a piano, french horn, clarinet, and voice. A second set of participants viewed these videos and followed a procedure identical to that of the first experiment - notes were presented under two conditions (audio-visual and audio-alone) with instructions to ignore the gestures in the audio-visual condition.
Here you can see videos of the piano, marimba, clarinet, and voice.
The results show the influence of the gesture is unique to percussive sounds. While the piano timbre showed some visual influence, non-percussive sounds were unaffected. We interpret this as indicating the brain binds together auditory and visual information sharing a meaningful relationship. While an impact gesture could produce a percussive sound, it surely could not produce a clarinet or french horn timbre. Indeed, the videos showing impact gestures paired with a sung vocal note generally cause great laughter! Because it is obvious the two cannot go together, the pairing reaches the point of absurdity, in which case the gesture is not perceptually bound with the sound and there is no visual influence.
Conclusion
From an acoustic perspective, the gesture was a miserable failure - notes produced by long and short gestures are no different in the absence of visual information. However, quite accidentally it proves an enormous perceptual success. As participants were instructed to ignore visual information when judging duration, these observed effects suggest the auditory-visual binding is obligatory. It is no more possible to ignore relevant visual information when judging auditory duration than it is to see the letters C-A-T without reading them as the word “CAT.”
Therefore gestures can and do play a meaningful role in audio-visual musical performances. By using visual information, performers can (inadvertently) overcome a profound acoustical limitation of percussion instruments. Gestures are what permit expert percussionists to convey duration information, allowing them to align audience perception with performer intent.
In light of these findings, it is important to remember that music is only music in the mind of the listener. Therefore gestures that change the sound within the mind have done more than “alter perception” – they have effectively changed the music.
References
Schutz, M. and Lipscomb, S. (in press). Hearing Gestures, Seeing Music: vision influences perceived tone duration. Perception
Walker, J. T., & Scott, K. J. (1981). Auditory-visual conflicts in the perceived duration of lights, tones, and gaps. Journal of Experimental Psychology: Human Perception and Performance, 7(6), 1327–1339.