Ugoku: Using Physical Computing and Interactivity to

Explore Gesture as Language and Dance

Presented to the Department of Theater Studies toward

fulfillment of the requirements for a Bachelor of Arts degree at
Yale College in Computing and the Arts on April 25, 2014

Sho Matsuzaki
Advised by Elise Morrison

TABLE OF CONTENTS
I.
II.
III.
IV.

The Program
The Concept
The Performance
The Future

I. THE PROGRAM
What if a dancers body were able to alter music instead of only respond to it? I
first entertained this question after watching an art installation by David Rokeby entitled
Dark Matter (2010). Rokeby, who specializes in interactive electronic art, has pioneered
the movement towards incorporating physical computing into installation art. His Very
Nervous System, a network of cameras and computers that captures body movement and
translates it into sound related to the qualities of the movement, was the first motionsensing system of its kind when it was created in 1986.1 Since then, physical computing
has evolved by leaps and bounds, and Rokeby has continued to integrate new innovations
of the field into his art. In Dark Matter, Rokeby uses technology similar to his Very
Nervous System to create empty spaces within a room that trigger various sounds
whenever someone reaches his or her hands into one of them.2 He refers to these spaces
as sound sculptures because one is capable of sensing the precise boundaries of the
sculptures by gently feeling around them; unlike tangible sculptures, whose form must be
admired from afar, Rokebys sound sculptures can only be taken in by getting up close.3
Rokebys work introduced me to the concept of sound produced exclusively by a
traversal through space. Never before had I seen such a graceful approach to

1 David Rokeby, Very Nervous System (1986-1990), 24 November 2010, David Rokeby, 12 April 2014
<http://www.davidrokeby.com/vns.html>.
2 THEMUSEUMtv, David Rokeby Explains Dark Matter, 16 November 2011, YouTube, 13 April 2014
<https://www.youtube.com/watch?v=QE9NE9n3HTI>.
3 Ibid.

interactivity, in which the participants could impact the work without the aid of any
external device impeding their movements. I felt that this absolute freedom of motion
lent itself not only to simple reaching movements, but also to complex dance as a means
of interactivity. In my mind, the causality between dance and sound was entirely
unidirectional; the tempo, timbre, and style of the sound governed every aspect of a
dancers movement. However, using a method similar to Rokebys, one could design a
space in which dance impacted or even created the music playing. In such a space, how
is interactivity redefined? Is dance also redefined, or is such movement that influences
sound instead of responds to it perhaps not dance at all? I decided to follow in Rokebys
footsteps in order to explore these questions.
David Rokebys work not only prompted me challenge my conception of
interactivity and dance, but also reinforced the notion that interactive art could alter ones
way of living outside the gallery or performing space. Rokeby strongly believes that his
installations can make one more aware of ones body movement and connection to the
surrounding environment. He often describes his installations as virtual realities in which
the interface through which one interacts with the world is different than that of the real
world.4 He uses the phrase virtual spillage to refer to the sensation of feeling within a
virtual reality even after one has left it.5 For example, people who interact with Very
Nervous System for an extended period of time become acclimated to an interface in
which their body movements produce sound, so after leaving the work, they still feel
directly involved in the sounds around them.6 This level of immersion is only possible
4 David Rokeby, The Construction of Experience: Interface as Content, Digital Illusion: Entertaining
the Future with High Technology, Ed. Clark Dodsworth, (New York: ACM, 1998) 2.
5 Ibid.
6 Rokeby, Very Nervous System (1986-1990).

because the participants in the virtual reality are not hindered in any way and move just
as they would in the real world. I had never before heard of a work capable of leaving
such an imprint on ones perceptual system, and the opportunity to have such a concrete
impact, however brief, on a persons way of living urged me to create my own virtual
reality.
I was quite lucky in that my introduction to physical computing came at a turning
point in the field, when technology such as the Microsoft Kinect had first opened the
doors to user-friendly motion detection. When David Rokeby started work on Very
Nervous System, the only way to capture accurate data from the three-dimensional
world was with many precisely located cameras and an efficient algorithm capable of
translating large quantities of pixel data into a coherent scene that a computer could
interpret in almost real-time. More than twenty years later, I had a variety of motionsensing hardware to choose from, each with its own comprehensive software
development kit.
One such hardware called the Leap Motion hit the market in 2013 as the first
mainstream sensor designed to recognize hand and finger movements. The sensor
differentiated itself from products such as the Kinect with its high sensitivity and compact
size. I was immediately attracted to the Leap Motion because no other sensor sacrificed
full-body detection in favor of highly detailed gesture recognition, and the sensitivity
would allow me to incorporate highly subtle changes in movement. On the other hand,
however, I would not be able to detect traditional dance as I had originally intended. I
was so enticed by the novelty and uniqueness of the machine that I originally scrapped
the idea of exploring dance and sound in favor of exploring gesture and sound. However,

once I began research on different forms of mediated dance, I came to realize that the two
concepts were one in the same as long as I were willing to challenge my pre-conceived
notion of dance. In the end, the restrictions of the Leap Motion enriched my final product
as they forced me to focus on gesture, a type of movement that is just as complex and
beautiful as what most people consider dance but often goes unnoticed.
Once I had decided on the Leap Motion, the next step was to determine the
framework through which gesture would alter sound. Before focusing on a specific
performance concept, I began by writing a gestural DJ application for the Leap Motion
that could be used in practically any setting: on a desk at home, in a dance party, during a
theatrical performance, etc. My goal was to create a program that was not customized to
any specific performance concept but instead could be molded once I had done the
necessary research to craft a coherent performance. Writing in Processing, a Java-based
language designed for electronic art and visual design, I created an application that
allowed a user to play songs from a music library on their computer and add generic
audio effects in real-time to the songs using a series of discrete hand gestures. The list of
audio effects included volume control, playback rate and pitch control, a looping
function, and various low-pass and high-pass filters. The gestures that triggered the
effects were intended to be as intuitive as possible so as to make the program userfriendly. For example, increasing the volume of a song involved placing ones hands
above the Leap Motion with the palms facing each other and gradually moving the hands
further apart. I created a simple user interface that permitted the user to load an infinite
number of songs and transition from one to the next via hand gesture. When the program

was completed, I had created an easy-to-use system in which a users body felt attached
to the music that the computer was producing.
I named the program Ugoku, meaning to move in Japanese, to allude to the
programs attempt to redefine the relationship between movement and sound. However,
Ugoku was only a tool; deciding how the tool would be used in a performance was a
whole other challenge. My goal was to use Ugoku in a way that would add a new
perspective to interactivity as well as dance. I set out to depict gesture as a different form
of dance, one that is not a response to music. The performance that resulted went beyond
that goal and became an exploration of all the facets of gesture using both live and
mediated bodies.
II. THE CONCEPT
Since the Leap Motion specializes in the tracking of hand gestures, I began to sculpt the
concept for my performance by first trying to define what gesture is and what role it plays
in society. In Design, Digital Gestures, and the In[ter]ference of Meaning, McMeel,
Brown, and Longley, professors at the University of Auckland, quote the anthropologist
Andr Leroi-Gourhan as saying that gesture is the primary site of communication.7
The choreographer and scholar Kent de Spain also mentions the communicative aspect of
gesture while describing how the advent of writing managed to separate meaning from
movement: Anyone who read the words later would have to supply for themselves
whatever was lacking in communicative context: gestures8 Communication is
certainly a main component of gesture, but some forms of gesture are more
7 Dermott McMeel, Carol Brown, and Alys Longley, Design, Digital Gestures, and the In[ter]ference of
Meaning: Reframing Technologys Role Within Design and Place Through Performative Gesture,
International Journal of Performance Arts and Digital Media 7.1 (2001): 8.
8 Kent de Spain, Dance and Technology: A Pas de Deux for Post-Humans, Dance Research Journal 32.1
(2000): 12.

communicative than others. The most communicative gestures are observable during
lectures or speeches, in which they act as a non-verbal language intended to emphasize or
complement the language of the speaker. These gestures, seemingly arbitrary hand and
finger movements, have over time acquired very specific meaning. In watching a variety
of online lectures, I also noticed that a finite set of specific hand movements are repeated
frequently over the course of a lecture. This repetition bore a strong resemblance not
only to language, but also to choreography.
Aside from communicative gesture, there is also a far more visceral, unplanned type of
gesture, which I will refer to as emotional gesture. McMeel, Brown, and Longley
allude to this type of gesture with a quote from the German philosopher Edmund Husserl:
Gesturecreates an inference of meaning but does not have meaning.9 This form
seems to grow naturally from some internal passion rather than be logically constructed.
It is far less predictable and often harder to directly translate. I found the unplanned
quality of this gesture to be particularly beautiful. Despite the hand movements being
visceral, they are just as complex if not more so than communicative gesture.
Whereas communicative gesture is reminiscent of choreography, there is a strong
connection between emotional gesture and freestyle dance. In both types of gesture, the
thoughts of the speaker provide the music for the movement. However, communicative
gesture takes from a vocabulary of well-known movements, whereas emotional gesture is
often a spontaneous synthesis of new movements. In Notes on Gesture, part of Infancy
and History: The Destruction of Experience by the Italian philosopher Giorgio Agamben,
Agamben employs the words of the ancient Roman scholar Marcus Terentius Varro to
distinguish gesture from other types of movement: a poet makes a play, but does not
9 McMeel, 14.
6

act it; on the other hand the actor acts the play but does not make itWhereas the
imperator of whom the expression res gerere (to carry something out) is used neither
makes nor acts, but takes charge, in other words carries the burden of it.10 The Latin
word gerere is the origin of the word gesture, implying that a gesture is something
that is carried out from beginning to end.11 Agamben uses Varros quote to explain how
unlike a play, a gesture is performed by its author. Something I believe is missing in
Agambens analysis, however, is the fact that the authoring and performing of gesture
occur almost simultaneously. Varros comparison between gesture and
playwriting/playacting highlights in my mind the improvisatory element of gesture and
speaks to the connection between gesture and freestyle dance.
The resemblances between gesture and the common notion of dance indicated that
perhaps dance in a broader sense of the word could still enter into the performance,
despite the Leap Motions inability to record the full body. I was reassured of the
connection I drew between gesture and dance during a conversation with Yvonne Rainer,
a prominent American dancer, choreographer, and filmmaker. Rainer specializes in
experimental work in which she challenges all borders of dance, often employing
minimalist tactics.12 One of her films, entitled Hand Movie, is a five-minute film that
includes only a single shot of Rainers hand moving in intricate ways.13 So it should not
have been a surprise when I mentioned to her my hypothesis that gesture was in itself a
form of dance, and she responded immediately, But of course! With her vote of
10 Giorgio Agamben, Notes on Gesture, Infancy and History: The Destruction of Experience (London:
Verso, 1993) 135.
11 Gesture, Def. 1a, The Oxford English Dictionary, 2nd ed. 1989, Web.
12 Vincent Bonin, Yvonne Rainer, 2006, Daniel Langlois Foundation, 12 April 2014
<http://www.fondation-langlois.org/html/e/page.php?NumPage=1870>.
13 Kinodanz, Hand-Movie, de Yvonne Rainer (1966), 2 September 2010, YouTube, 12 April 2014
<https://www.youtube.com/watch?v=CuArqL7r1WQ>.

confidence, I decided that framing gesture as dance would be a focal point of the
performance.
Once I had parsed out the linguistic and dance-like qualities of gesture, I needed to
determine how to de-familiarize gesture in the performance and place it at the center of
the audiences attention. In order to gain insight from the lineage of artists who have
wrestled with the subject of gesture or any form of non-traditional movement in
performance, I began by researching the work of Paul Kaiser, a digital artist. Kaiser has
worked with Merce Cunningham and Bill T. Jones, two American dancers and
choreographers who were at the forefront of American modern dance and experimented
with incorporating technology and unorthodox movement into their work. In Kaisers
Sidewalk Project, the form of dance that Kaiser incorporates into the work is the
movement of crowds.14 He abstracts the movement so spectators can view it outside of
its cultural context and appreciate it for its intrinsic qualities. When describing Sidewalk
Project, Kaiser defines dance as a certain kind of grace that has been stripped to its
essentials.15 If I wanted to portray gesture as a form of dance, I had to in some way also
strip it to its essentials.
With Sidewalk Project as an example, I decided that some form of motion capture
would be necessary to abstract the gestures enough to de-familiarize them. I eventually
came across the ideal candidate for such a visual component: the LED wall of the Ground
Caf at the Center for Engineering and Innovative Design (CEID).16 By using a camera
14 Sidewalk Project focuses on crowd movement by using motion capture to record crowds and then
projecting the video onto the pavement of New York City using a streetlamp equipped with a high-powered
projector. Kaiser uses the projector and unconventional projecting surface to abstract and de-familiarize the
movement.
15 Kent de Spain and Paul Kaiser, Digital Dance: The Computer Artistry of Paul Kaiser, Dance
Research Journal 32.1 (2000): 19.
16 The Ground Caf is equipped with a partial wall and ceiling covered with 23,000 LED lights behind
frosted glass.

to record and display gestures on the LED wall, the frosted glass alone would add a layer
of abstraction. Given the sheer size and brightness of the wall, the gestures would also
automatically become a focal point of the performance. The Ground Caf was an ideal
setting for making gesture the center of attention and stripping it to its essentials, but I
still had to find some way to clearly convey gesture as dance and as language.
The McMeel, Brown, and Longley article introduced me to the option of
incorporating more than one body into the performance as a means of depicting gesture
as dance. The article discusses a gestural performance that took place within the
Business School building at the University of Auckland.17 In the performance, two
dancers conduct everyday gestures, such as checking a cell phone, stretching, or
yawning.18 If one performer were alone, the movements would not be conspicuous in any
way. However, the synchronization of both performers conducting the same gesture at the
same time creates a sense of choreography, a duet of sorts. I was interested in the
concept of incorporating gesture into a duet to convey its performative aspects, but I
wanted to highlight the freestyle component of gesture just as much as the choreographed
one. The University of Auckland performance gave me the idea to create a duet between
a gesturer and a traditional dancer, rather than having two bodies mirror one another.
Instead of highlighting the dance-like qualities of gesture by treating gesture explicitly as
choreography, I could visually link the movements of gesture to the movements of
traditional dance by having a dancer and gesturer interact with one another. The second
body would be crucial for depicting gesture as dance; however, it would also add new
layers to the performance that would not come without its complications.
17 McMeel, 8.
18 Ibid.
9

Despite the rich implications of the gesturer-dancer relationship, some of the

projects I had read about discouraged me from incorporating live dancers into the
performance. The first project I investigated that combined live dance and technology
was Ghostcatching, a collaboration between Paul Kaiser and Bill T. Jones.19 Kaiser,
when describing the effect of transferring the movements of a live dancer to a computer
or television screen, states in looking at performers on the stage, we are seduced by the
charisma of the body rather than by the beauty of the movement.20 The debate between
live and mediated bodies in performance has been prevalent ever since technology
entered the stage. Herbert Blau, a director and theoretician perhaps makes the strongest
argument for why live bodies are so charismatic: In a very strict sense, it is the actors
mortality which is the actual subject [of any performance], for he is right there dying in
front of your eyes.21 I did not want the liveness of the dancer to distract from the pure
movements of the gesturer. By including the live dancer, the performance would involve
two live bodies and one mediated body (the image of the gesturer on the LED wall). In
order to decide whether or not the live dancer was a good idea, I needed to analyze
Ghostcatching more closely.
As I watched clips of Ghostcatching, I often found myself forgetting that Jones
was a real person and therefore belittled the difficulty of his movements. For example,
although I might have marveled at the control with which Jones would steadily lift one
leg in the air without wavering, I was not phased when the animation of Jones
19 Kinodanz, Ghostcatching Bill T. Jones, 1 September 2010, YouTube, 13 April 2014
<https://www.youtube.com/watch?v=aL5w_b-F8ig>. In Ghostcatching, Kaiser animates the dancing body
of Jones as a three-dimensional collection of crudely drawn lines. Joness movements draw additional lines
in the artificial space, creating an intricate sketch that outlines the movements of the dance.
20 De Spain, 21.
21 Herbert Blau, Blooded Thought: Occasions of Theatre (New York: Performing Arts Journal
Publications, 1982) 134.

10

accomplished the same feat; only in the presence of the human would I see grace, and the
lack of a physical human in Ghostcatching troubled me at times. Kent de Spain touched
on a similar point in his discussion of the animations in Ghostcatching: They dont
quite communicate a sense of weight or momentum, but there is present a level of
movement detail, fragments of quirky and unintentional shifts and balances caught by the
motion-capture system, that lends them an eerie verisimilitude.22 De Spains mention of
verisimilitude would seem to be in favor of the animations effectively depicting Joness
movements. However, de Spain refers to the verisimilitude as being eerie because he is
acutely aware of the technology creating it. Despite the quirky and unintentional shifts
and balances, the animation still lacks a sense of weight or momentum, meaning that
the technology is not quite advanced enough to fade into the background. As a result, its
eeriness distracts from the movements just as much as a live body would. De Spains
comments pushed me more in favor of including a live dancer; however, I still had the
mediated body of the gesturer on the LED wall to consider, and I was unsure how the
combination of live and mediated bodies would play out.
In order to better understand the relationship between liveness and mediatization,
I looked to Philip Auslander, a professor at Georgia Tech, for insight. Auslander, in Live
Performance in a Mediatized Culture, discusses Herbert Blaus performance theory and
makes the claim that although media was originally created to mimic live performance,
now the live and the mediatized are becoming more and more similar: Blaus claim that
a performance combining live bodies and filmed images can produce a confusion of
realms presupposes that live and recorded images are perceived as belonging to different

22 De Spain, 9.
11

realms.23 In my performance, the dancer and gesturer interacting next to and under the
digital image of the gesturer would possibly create this confusion of realms. However,
Auslander states that the screen is becoming less and less of a barrier between the live
and the mediatized. Furthermore, my performance would not actually include any prerecorded images, so I was not concerned about the conflation of multiple periods of time
in one space. I became more attached to the idea of a live dancer performing next to the
image of the gesturer as a means of both depicting gesture as dance and also exploring
new methods of interactivity.
An article by Johannes Birringer, a choreographer and director, on the nature of
interactive dance helped clarify just how my performance would challenge current
paradigms of interactivity. Birringer begins by describing a concept he refers to as
movement-as-interface.24 This term refers to a system in which a dancers movements
trigger other media, such as video, audio, and motion capture.25 Movement-as-interface
is the essential element of Birringers definition of interactive dance.26 Although this
concept of sensors capturing movement and activating some other aspect of a
performance was exactly in line with my original objective of permitting dance to alter
music, I was curious as to why Birringers definition of interactive dance was so
unidirectional. Why did the interactivity have to come from the dancer side and not the
media side; what if a dancers movements were dictated by media that was being
simultaneously altered by a third party? By having a gesturer alter the music to which a

23 Philip Auslander, Live Performance in a Mediatized Culture (London: Routledge, 1999) 1.

24 Johannes Birringer, Interactive Dance, the Body, and the Internet, Journal of Visual Art Practice 3.3
(2004): 166.
25 Birringer, 166.
26 Ibid, 168.

12

dancer would in turn respond, Birringers concept of movement-as-interface would be

flipped.
The addition of live dancers would create a relationship between performers that had not
existed in the concept before. One performer would be in control of the music to which
the other performers would react. In one sense, this relationship would be that of a duet
as I stated earlier, in which one dancers movements would complement the others. This
aspect of the relationship would speak to the role of gesture as dance. The gesturer would
also be partially in control of the dancer, assuming that the dancer would respond directly
to the music that the gesture was manipulating. However, in a somewhat contradictory
way, the dancer would also be in control of the gesturer in that the dancers movements
would dictate the emotional meaning of the gesture. This complex relationship would be
conveyed in the performance both in how the dancers and gesturer communicated with
one another non-verbally as well as how the wall of LED lights would frame the dancer
with the hand movements of the gesturer.
I was confident in my portrayal of gesture as dance, but I had not yet fleshed out
how I was going to convey the linguistic element of gesture. The inspiration for this
aspect of the performance came from a description in the McMeel article about a gestural
computer application called the Gesture Machine. The Gesture Machine is a design
tool that records physical gestures to create unique three-dimensional images.27
Originally, the development team used discrete gestures for specific operations in the
program.28 However, the gestures were eventually scrapped in exchange for a more openended interface that allowed users to play around in the program and learn how gestures
27 McMeel, 14.
28 Ibid.
13

corresponded to manipulations of the 3D space.29 Such an open-ended interface gave me

the idea to create a learning environment in my performance in which audience members
acquired the gestures necessary to manipulate the sound.
In order to explore gesture as language, I decided it was important that audience
members and not myself use Ugoku in order to simulate the way in which specific
gestures acquire meaning. By having audience members control the Leap Motion and
also not provide them with prior instructions as to which gestures the Leap Motion could
recognize, they would have to try out different hand and finger movements in order to
gradually learn which movements corresponded to which audio effects, like a baby
experimenting with language. Also similar to language acquisition, once the audience
member grasped the proper movements, the emotional significance of those movements
would not be defined by the audience member but by others interacting with the audience
member, namely the dancer. The process of experimenting with different movements,
observing the resulting changes in the environment, and watching how those changes
affected others would illuminate the way in which gestures are formed and how they
acquire meaning.
Since audience members would not have prior instructions for how to use Ugoku, with
the exception of perhaps a demonstration I would conduct at the beginning of the show, I
had to rethink the specificity of the gestures that ran the program, similar to the Gesture
Machine. The final customization of the program involved mapping certain qualities of
the music being played to the position of one or both of the hands. For example, the
number of high frequencies permitted in the song was directly related to the z-axis
position of the right hand (with the z-axis pointing towards the user from the sensor). A
29 McMeel, 15.
14

clockwise circular movement of either hand resulted in the song speeding up, whereas a
counterclockwise circle slowed the song down. Perhaps the only highly specific gesture
was that of a single finger extended generally forward, which would cause the music to
shift from one speaker to the other based on exactly where the finger pointed. I hoped
that all of the gestures would not be too specific to pick up so that the audience members
would be inspired to learn them.
I eventually had to decide on what type of music was going to play during the
performance. I did not want the message of any particular genre to impact the
performance, so I was inclined from the start to play a variety of genres. In order to mix
genres, I needed to find dancers that had different dance backgrounds and would be able
to respond well to whatever was thrown at them. An eclectic combination of music and
dancers appealed to me more and more as the show concept developed further; the
inclusion of numerous techniques of expression spoke to the universality of gesture and
of dance.
In order to draw the audiences attention to gesture, I believed it was crucial that
as many elements of the performance be connected to gesture as possible. The gesturer
already influenced the music, the dancers, and the content of the LED screen. However, I
wanted the gestures not only to appear on the screen but also affect the overall mood of
the screen and performing space. I experimented with mapping the position of the hands
to parameters of various blurring, tessellating, and colorizing effects using the video
manipulation program Isadora. I added additional code to Ugoku so that information
collected by the Leap Motion was processed in Ugoku and sent directly to Isadora for
interpretation. Isadora would receive the values from Ugoku and use those values to

15

control the filters abstracting the image of the gesturer. In the end, because the frosted
glass in front of the LED lights was already abstracting the image slightly, the most
effective video filter was a simple colorizer. As an audience member would move his or
her hands around the Leap Motion, not only would the music in the room change, but
also the color of the lights would change. Finally, every aspect of the room was under the
control of gesture.
III. THE PERFORMANCE
The final performance occurred almost exactly as planned, with a few surprises
along the way. From the changing colored light that reflected off the walls to the
dramatic movements of the dancers, it was clear that everything in the performing space
responded to the hand movements of the person at the motion sensor. The dancers
performed spectacularly using styles that ranged from lyrical to hip-hop. One of the most
pleasing elements of the performance was that the audience felt a collective responsibility
to always have someone manipulating the music. As a result, the LED screen was always
changing faces, as were the audio effects that were activated. Some people chose to
conduct the music using purposeful and graceful movements, whereas others attempted to
test the limits of the program by moving their hands in a dramatic and, at times, wild
fashion. This variation, according to Grayson Cooke, a professor at Southern Cross
University, is the cornerstone of interactivity: live media performance is premised on
a dialectic between control and uncontrol, or a balance between intent and accident.30
The unpredictability of the performance was ultimately what made it special. Everyone

30 Grayson Cooke, Start Making Sense: Live Audio-Visual Media Performance International Journal of
Performance Arts and Digital Media 6.2 (2010) 200.

16

who sat by the sensor imbued their personality into their gestures, so that every set of
gestures had a unique character to it.
The most interesting discovery I made during the performance was of the power
dynamics that existed between the gesturer and the dancer. I knew beforehand that the
relationship between the two performers would be a layered one. However, I did not
anticipate how the relationship would change from person to person. By default, the
dancers played a subservient role, given the sudden changes in music that they had to
cope with and the gigantic display of the gesturer that loomed over them. Like a
puppeteer, if the gesturer decided to suddenly speed up a song, the dancer had no choice
but to follow along. That being said, the electric personality or, as Paul Kaiser would say,
charisma of the dancer often overpowered the timid audience members sitting by the
sensor. Some of the dancers would even stare at the gesturer and motion to him or her to
make the music more challenging. The dancers also had an advantage over the gesturer
in that they knew ahead of time what sorts of effects to anticipate, and the songs that
played during the show were songs with which the dancers were very familiar.
Throughout the performance, however, some of the most beautiful moments occurred
when both dancer and gesturer were sympathetic towards one another and a true duet
arose from it.
For the audience members who were not sitting at the sensor, it was informative
to watch where they cast their eyes. Occasionally, the audience members watched either
the person at the sensor or the LED screen. However, most of the attention was fixed on
the dancers. Although the dancers were placed in the center of the performing space, I
was surprised that more people were not looking at the LED screen, given its sheer size

17

and brightness. However, especially for the hip-hop dancers, there were so many fastpaced changes in motion that if one looked away even for a second, one would easily
miss something impressive. The way in which the dancers stole the show is perhaps a
testament to Kaisers hesitation about using live dancers. However, I believe the lack of
attention given to the LED lights was also a result of my earlier hypothesis that figures on
a screen appear less human and therefore harder to connect with. All things considered, I
was not disappointed that the live dancers were the focal point, since they provided the
emotional interpretation for the gestures that were underlying all aspects of the show.
IV. THE FUTURE
There are a few changes to the show that I would make if I wanted to continue
this exploration of gesture. Firstly, I would try to expand on the possible audio effects of
the program. The panning effect of moving the sound from one speaker to the other did
not register clearly in the performing space since the speakers were set up as a mono
channel instead of stereo. In addition, the volume control functioned well, but it did not
elicit many major changes in the dancers movements. I would have liked to experiment
with further manipulation of the musics frequencies to create a more distorted sound to
which the dancers could respond. I also would like to expand my concept to include
audio synthesis as well as manipulation. I would map gestures to specific sounds and
essentially convert the gesturers body into an instrument. I believe the effect of using
gestures to create the music would be truer to the concept of gesture as language. In
addition, the connection between dancer and gesturer would become even stronger; the
gesturer would have more control over the dancers movements, but the dancer would
more fully embody the emotional content of the gestures.

18

One final change to the concept that I would like to explore would be to somehow
track the gestures of everyone in a room simultaneously instead of singling out one
person. At one point in the process, I had toyed with the idea of trying to engage
audience members in conversation and use their natural gesticulation to control the music
in the room. This idea became too difficult to achieve in a practical sense, but capturing
gesture while audience members were not monitoring their movements would be ideal for
examining how gesture functions in everyday life. To that end, I would love to remove
all formal aspects of the performance and instead design a party or other social event in
which one or multiple sensors could pick up on all of the gestures occurring at the same
time. That way, all audience members would be implicit in changing the music and
lights, and no one would be self-conscious enough to filter their movements. This new
element would truly bring back Rokebys concept of virtual spillage. By masking the
virtual reality in an informal setting, participants would be able to fully give in to the
rules of the space and hopefully return to the real world with a slightly altered experience.

19

Bibliography
Agamben, Giorgio. Notes on Gesture. Infancy and History: The Destruction of
Experience. London: Verso, 1993.
Auslander, Philip. Live Performance in a Mediatized Culture. London: Routledge, 1999.
Birringer, Johannes. Interactive Dance, the Body, and the Internet. Journal of Visual
Art Practice 3.3 (2004): 165-178.
Blau, Herbert. Blooded Thought: Occasions of Theatre. New York: Performing Arts
Journal Publications, 1982.
Bonin, Vincent. Yvonne Rainer. 2006. Daniel Langlois Foundation. 12 April 2014
<http://www.fondation-langlois.org/html/e/page.php?NumPage=1870>.
Cooke, Grayson. Start Making Sense: Live Audio-Visual Media Performance.
International Journal of Performance Arts and Digital Media 6.2 (2010): 193208.
De Spain, Kent. Dance and Technology: A Pas de Deux for Post-Humans. Dance
Research Journal 32.1 (2000): 2-17.
De Spain, Kent and Paul Kaiser. Digital Dance: The Computer Artistry of Paul Kaiser.
Dance Research Journal 32.1 (2000): 18-23.
Gesture. Def. 1a. The Oxford English Dictionary. 2nd ed. 1989.
Kinodanz. Ghostcatching Bill T. Jones. 1 September 2010. YouTube. 13 April
2014 <https://www.youtube.com/watch?v=aL5w_b-F8ig>.
Kinodanz. Hand-Movie, de Yvonne Rainer (1966). 2 September 2010. YouTube. 12
April 2014 <https://www.youtube.com/watch?v=CuArqL7r1WQ>.

20

McMeel, Dermott, Carol Brown, and Alys Longley. Design, Digital Gestures, and the
Interference of Meaning: Reframing Technologys Role Within Design and Place
Through Performative Gesture. International Journal of Performance Arts and
Digital Media 7:1 (2001): 5-22.
Rokeby, David. The Construction of Experience: Interface as Content. Digital
Illusion: Entertaining the Future with High Technology. Ed. Clark Dodsworth.
New York: ACM, 1998.
Rokeby, David. Very Nervous System (1986-1990). 24 November 2010. David
Rokeby. 12 April 2014 <http://www.davidrokeby.com/vns.html>.
THEMUSEUMtv. David Rokeby Explains Dark Matter. 16 November 2011.
YouTube. 13 April 2014 <https://www.youtube.com/watch?v=QE9NE9n3HTI>.

21