E-Motions explores the animation and sonification of abstract
information spaces through real time motion capture data from a dancer
or group of dancers. Instead of visualizing the capture data as an
avatar animation, we chose to create abstract worlds that are visually
opposed to the simulated, but realistic, spaces with which motion
capture normally interacts. E-Motions uses aspects of the data, like the
coordinates of a hand or the velocity of a knee, to arrange elements in
these virtual realities. Thus, while the worlds behave differently based
on the performer's position in the performance space or how fast she
moves, the correspondance between virtual and real is indirect, forming
a puzzle to be deciphered by the audience.
Our methodology, when experienced within one of the E-Motions virtual
worlds, simultaneously obscures and reveals the performer's motion. Much
like a glyph in information visualization, the state of a world reveals
aspects of the dance, but still exists as something entirely other. By
eschewing realism, we gained the freedom to make the mapping between
performed motion and manipulated world arbitrarily complex, creating
problems for choreography. The performer must move to create interesting
sonic and visual effects, but the internal logic of a given world causes
certain movements to be more meaningful than others. Indeed, coding the
E-Motions worlds involved an iterative process, with the dancers
discovering what was possible, adapting their movements to the current
rules, and requesting code alterations to create new possibilities. In
many ways, choreography and coding became two aspects of the same
process.
E-Motions uses two parallel systems, one to drive the rendering of
the abstract worlds and another to drive the audio. Motion capture data
from a Motion Analysis optical tracking system is sent to a PC, which
interprets it into virtual world parameters and then renders the
resulting world via custom software. The data communication and world
rendering is based on the open source Syzygy library, originally written
for PC cluster-based virtual reality. On the other hand, the audio
system is based on the MAX programming environment. Two Mac computers,
each connected to a seperate firewire video camera, are used to generate
the audio events. Commercial Max plug-ins on each Mac process the
incoming video, producing MIDI events based on the amount of image
change present in different sections of the video input. These MIDI
streams are then converted to sound by a custom MAX program.
Using optical motion capture systems in a live artistic work has
disadvantages. In such systems, the captured individual wears small
infrared reflecting spheres called markers, and an array of cameras
registers their positions. Unfortunately, optical tracking systems
falter when confronted with marker occlusion, as often occurs during
human motion, perhaps when the dancer's limbs are drawn in close to her
body. At such times, the system can misidentify markers or lose them
altogether. Normally, artists use a post-processing step to clean up
motion capture data for character animation. However, E-Motions cannot
do so, being a real-time performance piece. Because we use non-realistic
spaces, where the observer has fewer preconceived notions of how things
should look, the flaws in the optical motion capture are less
objectionable.
The Motion Analysis capture system comprises the bulk of the
E-Motions equipment, with 10 cameras, tripods, some networking gear, and
a host computer. A small amount of additional computer equipment is
used. On the video side, a PC receives data over ethernet from the
Motion Analysis system and renders the virtual world, displaying on a
nearby wall via a portable projector. On the audio side, two Macs
receive video input, each from a seperate firewire camera, and create
sound using the MAX environment. A small sound board get the audio in
the performance space.
E-Motions has been presented twice, each time for about a week, as an
ambient performance piece in a museum setting. Each day, a cast of
dancers rotates in and out of the space. Two engineers control the
computer equipment, one adjusting parameters for the sonification and
the other working on the video side switching between virtual worlds.
The E-Motions performance is a mixture of advertised show times, when an
audience congregates to see a rehearsed sequence of worlds, music, and
movements, and other times when the dancers are experimenting with the
system, learning the behavior of the worlds, and adapting their
movements to that behavior. The process of learning how to physically
interact with an abstract space is an important part of the performance.
The first presentation of E-Motions used a single dancer, fully
markered, so that all parts of her body created motion data. The second
time, the fully markered dancer was joined by two others, each wearing
three markers. The Motion Analysis system was able to track them
seperately from the main dancer, creating an ensemble piece.
