Você está na página 1de 9

Jran Rudi

Norwegian Network for Technology, Acoustics,

Computer Music Video: A
and Music (NOTAM)
Nedre Gate 5
Composers Perspective
0515 Oslo, Norway

Computer music and video can be combined and Mapping can be considered a subset of visual rep-
made interdependent in works of art in a multitude resentation. In binary format, data can be ported
of ways. Works may either contain pre-rendered im- with relative ease from one domain to another.
agery, be generated on the fly from a set of rules, be However, the data may not make much sense
passively dependent on external events, or actively when ported into the domain of another medium,
seek interaction, requiring specific user activity for as for example experiments with the resynthesis
realization. Video imagery can be developed as digi- of graphics files in a phase vocoder such as used in
tal models that change according to spatial or tim- the software program Ceres (originally authored
bral relationships in the music, for example, and by yvind Hammer in 1994; www.notam02.no/
audio can be developed by mapping pixel values notam02/prod-prg-ceres.html). To represent music
from a static or moving image to parameter values or sound visually in a cross-media approach, deci-
for synthesis and sound processing. sions must be made about which aspects are to be
This article gives a brief overview of some issues represented; in other words, the mapping process
in music representation and mapping, and it dis- requires interpretation.
cusses the artistic strategies that I have employed in Cross-disciplinary representations may also in-
the cross-media approach to When Timbre Comes clude auditory display systems and sonification,
Apart (199295), Concrete Net (1996), Planet (Terra) although the goal for these might not necessarily be
(1988, 1996), and Construction Drive (2005). the artistic exploration of sound. Cross-disciplinary
representations are not trivial from a timbral point
of view, and it has been argued that they make most
Cross-Disciplinary Representation and Mapping sense when informed by psychoacoustics and when
aware of the implications they have for setting audi-
Visual representations of music, such as musical tory scenes. An example of this is the use of sound
notation, spectrograms, and other mappings of in computer games, in which the feedback and
sound data, may be considered cross-disciplinary. graphical user-interface sounds provide auditory in-
However, not all such forms of notations are in- formation on system status and user actions and the
tended as interesting visual objects in their own music changes dynamically according to position,
right, although several musical scores contain quali- action, and environment (Schtze 2003).
ties considered as visual art, such as Agglomaration New possibilities for narrative arise when other
(1960) by Anestis Logothesis, December 1952 by forms of representation and expression are added to
Earle Brown, and Five Piano Pieces for David Tudor sound, and the composer must consider which links,
(1959) by Sylvano Busotti. Musical notation, listen- if any, are desirable to establish between the forms of
ing/analysis scores, and plots in time and spectral representation in the specific work. There are many
domains represent different aspects of music. Each interesting examples from interactive dance and dra-
representation type focuses on different elements or matic performances in which different methods for
parameters of the musical idea or object, and these extracting data from the dancers movements have
representations serve as tools for the composer/ been employed (Siegel 1998; Bevilacqua, Ridenour,
artist, facilitating different manipulations of the and Cuccia 2002; see also Todor Todoroffs 1998
musical material. work En Jeu and his 2000 collaborative work In Be-
tween). In these examples, the data have been linked
Computer Music Journal, 29:4, pp. 3644, Winter 2005 to sound generation and processing, and in some in-
2005 Massachusetts Institute of Technology. stances to lighting and digital scenography. Of

36 Computer Music Journal

particular interest is video software such as BigEye from complex auditory scenes (Bregman 1990) based
from STEIM (www.steim.org/steim/bigeye.html) on the combination, presence, or absence of sounds
that calculates changes in pixel values and maps that we hear or expect to hear. Some sounds repre-
them to musical parameters. In the reverse direction, sent specific actions with enough clarity to be con-
video imagery can be based on the mapping of pa- sidered sound icons, whereas others are ambiguous
rameters taken directly from the music, as in Roger and may refer to a diversity of actions, leaving the
Dannenbergs work Uncertainty Principle (2000). listener free to connect the dots. Aesthetic and
analytic issues such as these are frequently dis-
cussed in the framework of spectromorphology
Points of Departure, Intention, and Approach (Smalley 1986, 1997), as a further development of
Pierre Schaeffers writings in Trait des Objects
Composition can be abstract storytelling, or the Musicaux (1966).
telling of abstract storiesperhaps both. Extending Composing for audio and animation entails a rela-
Varses definition of music as organized sound, tively new art process that involves issues of audio-
music may also be conceptualized as contextualiz- visual coherence. The process raises questions. How
ing sounds, often from a hermeneutic perspective, should the visuals relate to the sounds? Should they
where single components are meaningful only when be referential or non-referential, literal or sym-
considered as part of a whole, and where the context bolicshould they be concrete and recognizable as
is complex and allows for multiple interpretations representations we are familiar with, or should they
or understandings. In the compositions discussed be abstract? Should the visuals somehow be derived
later, several extra-musical ideas are interwoven from the sound? Following the storyline and time-
with the goal of mirroring and addressing the com- line, issues also arise pertaining to moving pictures:
plexities in nature and human interaction, which color, light, camera angles, and movement, as well
are context-dependent and difficult to predict. The as the editing to achieve speed and development.
objects of these compositions are often taken from In addition, there are thematic questions of poetry,
ideas of action and consequence, namely how per- politics, and program. Should the music be about
ceptions are structured and developed into patterns, something that can be clearly stated and easily un-
and how resonance (in the sense of emotional recog- derstood, and should the work attempt to convey a
nition) emergesin short, how our myths are sense of immersion, or should it make the illusion
formed. While myth today is often understood in (of reality or representation) explicit and thus avail-
terms of a false statement, the Greek mythos able for reflection? Such decisions are linked to the
refers to powerful truths about the world, often in artistic intentions that define the genre of computer
the form of stories. music video.
Electronic sound is the primary sound material in The weighing and balancing of contradictory
my compositions that are discussed in this article; pulls, visual and auditory, complicates the process
as a tool, it has allowed great freedom to create nar- of identifying the locus of artistic intention in com-
ratives with different levels of abstraction by play- puter music video, as the object of consideration
ing on the representational and symbolic values in may become pitted against the idea of some type of
the sounds themselves. Computer technology al- action, as in such game-type interactive composi-
lows sounds to be manipulated away from (and to- tions as my work in progress Construction Drive.
ward) clarity. To make the context work in these For many composers, making music is a type of re-
compositions, it was important to intentionally search, a personal exploration of ideas and how they
blur clarity in the music, thus fostering ambiguity, may be given audible form. Combining music with
avoiding one-dimensionality, and emphasizing visual media changes and recharges this notion of
choice. Blurring and ambiguity are familiar features composition as research process, presenting oppor-
in our everyday experience, as meaning is abstracted tunities to engage with recent and commercial tech-

Rudi 37
nologies, such as DVD and computer game platforms. finding their own patterns rather than waiting for
In regards to the latter, the currently available con- the composers message. In any case, a sense of
tent for game platforms suggests that this is a largely clarity might be deceptive on any level, as Czeslaw
unexploited niche for art, holding the promise of Milosz suggests in his 1951 novel The Captive
new audiences as well. Mind: When someone is honestly 55% right, thats
very good, and if someone is 60% right, its wonder-
ful and great luck, but if someone is 75% right, it is
Ideas and Techniques in Four Computer Music suspicious, and whoever says he is 100% right is a
Videos fanatic! Uncertainty, then, is a compositional ele-
ment that can be reinforced or quelled by either vi-
In the works discussed below, several ideas are com- sual or auditory meansor bothin the computer
bined, drawn from previous personal experiences, music video.
specific sounds, numerical descriptions of natural
phenomena, literature, visual art, and human mo-
tion. A consistent trait is nonetheless the desire to When Timbre Comes Apart
connect seemingly dissimilar ideas and to explore
these connections through composition. The com- In my work When Timbre Comes Apart (199295),
poser can tell several stories at the same time by meaning (timbre) is disassembled and reassembled,
modeling complex situations from many dissimilar as the piece investigates what happens when parts
elements. The totality is incomprehensible without are taken away from the whole and how the same
an understanding of the parts, and the parts lose elements can be used to reconstruct a new entity.
their significance if removed from the whole. In this The medium for this was timbre, the detailed ma-
complex weave, elements vie for attention, and what nipulation and blending of harmonics in the contin-
may considered particularly important parts of the uum between pitch and noise, and in-between
compositions are also merely a small part of this timbres that could be recognized and sounds where
larger sonic amalgam. The intention is that when no source or specific identity was perceptible. The
those elements in the music that are most difficult timbres were derived from additive and granulation
to hearthose in which the surrounding sounds synthesis algorithms, along with two sound record-
mask them and offer them the most resistanceare ings: one of my son (two years old at the time), and
noticed, the listener will be rewarded by a sense of one of a large tam-tam, approximately five feet in
direction or continuity. This element of resistance diameter. The ratios between the partials of the syn-
requires concentration, and listeners may need to thesized timbre that is presented in the first half-
overcome their own resistance as wellnot to suc- second of the piece are used throughout the work as
cumb to boredombecause it may take awhile be- structuring numbers, and the concrete pitches of
fore the idea is uncovered. the partials serve as a guiding harmonic grid (see
The possibilities for these kinds of reactions are Rudi 1998).
composed into the works, and as such, they enter At the time this piece was written, phase vocod-
into a tradition of works that employ a systematic ing had become easy to do, because the price of
stretching of the audiences patience. (I have per- computational power had dropped significantly.
sonal, vivid memories of this type of forced eartrain- The first three parts of the work were synthesized
ing from John Cages reading/singing of James with the Kyma System, as was the granulation in
Joyces Finnegans Wake. It was a humbling experi- the last two parts. NOTAMs powerful phase
ence to understand that my sense of time and com- vocoder program Ceres was the principal software
positional topic was insufficient for grasping the for the timbral manipulation. The music was writ-
work at first.) By stretching the listeners patience ten first, and the animation was developed directly
and expectation of musical development in time, from the music. At the time, graphical modeling
the composer demands that listeners be creative in was an exploding field owing to the hardware-based

38 Computer Music Journal

Figure 1. Three- front to back, and fre-
dimensional sonogram quency is distributed left
from a section of When to right, with higher fre-
Timbre Comes Apart quencies to the left.
(199295). Time runs from

graphic engine of Silicon Graphics (SGI) computers, compositional perspectives, elements that perhaps
and animation of a modeled spectrogram was easily would have otherwise passed unnoticed. An ex-
executed. The SGI featured the excellent modeling ample of this is shown in Figure 1, where the cam-
package Explorer, and the camera movement along era is located between partials that almost disappear
the model was scripted using a number of splines to in the total sound at that time. Another example of
achieve smooth camera movement with little or this is seen the closing of the first part, where the
few jumpy movements and to eliminate abrupt camera looks back while ducking below the model.
beginnings and endings of camera trajectories. The The intention is to strengthen a sense of closure for
model was normally viewed with a variable time- the listeners. The first part is composed to be in-
limited horizon to not reveal too much of the piece tense and long, in order to allow the listener a sense
at any time, and the camera was moved back and of release at the onset of the second part. Other ex-
forth, over and under the model. amples of how narrative was strategically employed
The narrative of When Timbre Comes Apart are the zooming effect leading into the fourth and
aimed to point out important sound elements and fifth parts, where the camera is kept close to the

Rudi 39
model, and when the listener experiences crashing tion, working through and experiencing the remains
through the representation at the beginning of the of something that has happened in the past. The lis-
fourth part as a mirroring of the strong onset sound. tener is awakened in the beginning of the piece,
The narrative was made possible by extensive which then passes through three parts with signifi-
camera movement, changing focus and perspec- cant sound events, and then winds down to the end
tive, looking forward, backward, sideways, over, of the piece during a long fourth part of reflection.
and under the model. The educational aspect of this The sounds in the piece emerge from a long fascina-
method of pointing to essential compositional ele- tion with sounds from long steel wires, sounds from
ments was further emphasized in another work And steel scraps, and the 1974 novel Concrete Island by
the Birds . . . ? (1997), written for NOTAMs educa- J. G. Ballard. The novel is about a highway network
tional software program called DSP (Sack 2003). and its excluding nature once one has exited it and
The even flow of sound samples determined the is no longer moving along smoothly. As long as the
speed of the camera along the time axis in both of principal character in the book follows conventions
these works, with the listening point situated di- and behaves according to the network rules, the
rectly under the camera. The even movement gave network is a good thing, but once off the highway, it
a drive to the works that was stronger than the is virtually impossible to get back on. As in much of
music would have provided on its own. The drama- Ballards writings, there are vivid descriptions of an
turgy of the animation followed the dramatic devel- interesting subculture with specific survival strate-
opment of the music through the use of focus, giesan alternative network. Networks are thus
distance to the model (which influences the percep- both inclusive and exclusive, and in the novel, it is
tion of speed), and color. Movement under the virtually impossible to get back on the highway
model was used to open and close compositional once one has exited. This duality was explored in
sections, and color was used to separate the differ- this musical piece through readings of selected pas-
ent parts of the composition. To me, for example, sages from Ballards book, deliberately obscured
orangey red suggested a warmer timbre than through signal processing where vocal samples ex-
metallic green, and other colors were used to vi- cite physical models of strings. The physical mod-
sually suggest different timbral qualities and musi- els were visualized and adapted in the video part,
cal expressions in the other parts of the compositions. and traffic sounds were used as sound material, fil-
As a perceptual tool, the spectrum was displayed tered and pitched to create an underlying, changing
opposite from what is common in three-dimensional harmony.
representations, with the higher partials closer to To construct the space of the piece, nine sets of
the camera than the lower partials. ratios were used, measurements of the distances be-
Given that the images and the sound are two tween the planets in our solar system. Each set
types of representation of the same data in this comprised measurements made from one planet to
work, When Timbre Comes Apart is a true cross- the others, and the sets were quasi-transposed to be
domain mapping. However, the camera was neither in the same range. These ratios were used as param-
neutral nor still, always aimed at an important fea- eter values for all signal processing in the piece and
ture of the music to influence the listeners appreci- to slice the vocal recordings to excite different
ation of it. One could say that the camera tried to be physically modeled strings with different signals. (If
loyal to the distribution of interesting and salient all of the strings had been excited with the same sig-
events in the composition. nal, the result would have been similar to that of an
analog spring reverberator; see Rudi 1998.)
The video part is quite simple, rendered in Persis-
Concrete Net tence of Vision (POV-Ray), a freeware raytracing
program. There are three main events in the piece
In Concrete Net (1996), the camera had another all represented by different and distinct models
function. Concrete Net is about memory and reflec- unlike When Timbre Comes Apart, in which the

40 Computer Music Journal

Figure 2. Image taken from
Concrete Net (1996), at the
entrance point to the sec-
tion of physically modeled
strings. There are nine sets
of strings.

spectrum served as a model for the entire piece. The string constructions, and slowly rotating from there
images are abstract, taken from space imagery: a toward the end of the piece. For the camera rotation
planet, debris, and a star. The planet is the first ob- to have the desired effect of visual recapitulation,
ject encountered, followed by a cloud of fog and the trajectory needed to follow a curve, and the ef-
metal debris, which is in turn followed by nine ro- fect is that the listener sees restatements and rein-
tating string constructions (see Figure 2). In between terpretations of events that took place earlier in the
the objects are smaller lumps of fog and metal scraps, piece as they move further and further away, even-
signifying closure and transition, to prepare the lis- tually disappearing behind the star. The sound in
tener and to recollect the past musical experience. this part is also formed of restatements and reinter-
Following the string constructions is a long stretch pretations. The slowness of the rotation brings calm
with visual noise in the form of metal scraps, placed and enables reflection on the material and the re-
mainly to give the viewer a sense of movement and minders of the events, and it returns the mood of
direction. The camera movement is simple, point- the piece to a tranquil state after having worked
ing forward in linear fashion until the end of the through the material several times.

Rudi 41
The camera is quite neutral throughout this piece, considered from all angles, becoming increasingly
with the exception of the beginning, when the cam- clearer as the piece plays and the model becomes
era vibrates to emphasize the sudden change in more exposed. The camera movement is indepen-
timbre that occurs after breaking through the at- dent from the time in the music: it speeds up and
mosphere of the planet. The piece is composed to be slows down, adding excitement and calm in keeping
unbalanced, with all important events occurring in with the ideas in the music. In this work, the cam-
the beginning. The subsequently relatively still, ro- era and the music indirectly complement one an-
tating camera in the fourth part encourages careful other; rather than following the idea or development
listening, as it becomes clear that the camera, hav- of the music, the camera is used to make more of a
ing shown what is coming, does not point out new philosophical point about communication and un-
elements in the composition. The idea was to create derstanding. When the image has been viewed from
space for the listeners reflection by creating a dis- all sides, the camera self-destructs; the idea has
tance to the previous events, mentally and physi- been exposed and its job done.
cally, thus reinforcing the original intent of the
piece related to notions of inclusion/exclusion.
Construction Drive

Planet (Terra) Construction Drive (see Figure 4) is a computer mu-

sic game in the final stages of completion. The player
Another work that belongs to this group of works is navigates in a virtual world, and the graphics are
Planet (Terra), an animation of music that was real- based on a well-known painting. Apart from the as-
ized in 19871988. The music originally attempted sociations engendered by the painting itself, the
to depict developing order in a chaotic sound world. image has been used directly in two ways. The
The piece is short (about 3 min 30 sec), merely sug- painting has been mapped onto a modeled, three-
gesting an idea rather than expanding on it, as is the dimensional landscape, with geological formations
case in the two previously discussed works. The an- that largely follow lines suggested in the image.
imation was based on mapping of the spectral data Geometric and amorphous shapes in the painting
from the piece, modeled onto the surface of a sphere. have been made three-dimensional and situated as
This sphere was enclosed in another sphere that had objects in the landscape according to their original
same base color as the first, and this sphere shrinks placement. A substantial part of the sounding mate-
as the music plays, allowing the mapped spectrum rial stems from recordings of readings of Guillaume
to emerge. The spectrum was colored to resemble a Apollinaire on themes related to the artistic style of
geological map, and the landscape materializes as a this particular painting. Together, the sounds and
mythical receding of water that bares mountains, images in the game are concerned with the nature
valleys, and plains, as is shown in Figure 3. of art and the riddle of perception. The player navi-
A piece of music consists of many parts, and in gates by driving through the landscape, which is
relation to the above discussion of contextualiza- strewn with sounds that are performed as the player
tion, the listener is not really able to grasp the enters their active zone. This is a compositional
whole piece until all parts have been played. The technique that has also been employed in a game for
idea is developed over time, and each moment has the Microsoft Xbox (Schtze 2003).
only limited significance when taken out of con- In such interactive works as games, the composer
text. The model and animation in Planet (Terra) at- relinquishes control of the distribution of sound
tempted to address this concept through a gradual events in time throughout the piece. In many com-
revelation of the entire spectrum (the content) and puter games, composers employ sequencing software
through a circling camera movementconstantly like Microsofts Direct Music, building soundtracks
moving, focusing, and refocusing different parts of from general MIDI patches and recorded sound, and
the sphere. The piece, mapped around the sphere, is changing orchestration through cues from user ac-

42 Computer Music Journal

Figure 3. A view of the
planet model showing the
quasi-geological map after
the waters have receded,
taken from Planet (Terra)

tions, game events, and locations in the game envi- areas of timbral and thematic spaces, enabling con-
ronment. This is an effective means of maintaining trol over the coherence in the piece. The visual
a high degree of real-time control of the soundtrack, elements reinforce the possibility of creating a the-
but is better-suited as a tool for creating a computer matic space, although the video narrative is nonlin-
game music track with mood changes and effects ear, given that the user has significant freedom to
than computer music, owing to the limited number choose where in the landscape to drive. An impor-
of available signal-processing routines and the high tant consideration has thus been to secure good
dependence on available RAM for loading material enough musical results for the players to encour-
that can be processed. age them to return, and certain concessions have
In Construction Drive, the sequence of sounds also been made to the game paradigm, which de-
resulting from player movement can be different mands that there be a point, or reason for playing.
every time, likewise the spacing of events. This As with any visual media, the eye performs the
type of situation is challenging for the composer. It image through the act of looking, and the geological
is nonetheless possible to group sounds as different underlay for the surface, volumes, and colors of the

Rudi 43
Figure 4. A preliminary ex-
ample from Construction
Drive of a three-
dimensionally modeled
shape from the painting.

Although video art requires unique sensitivities

and technical skills, it entails much of the same un-
derstanding of composition and dramaturgy as mu-
sic. The works discussed have primarily exploited
video as a complementary means of expressing mu-
sical ideas, by creating an additional layer of reso-
nance for the listeners and greater opportunity for
reflection or appreciation of complex interconnec-
tions of meaning and suggestion. It is perhaps in
this superimposition of layers of meaning and inter-
pretation that the most rewarding artistic aspects
lie in the genre of computer music video.


Ballard, J. G. 1974. Concrete Island. London: Granada

modeled objects suggest connections and leads the Publishing.
gaze in certain directions, encouraging initial stages Bevilacqua, F., J. Ridenour, and D. J. Cuccia. 2002. 3D
of interpretation. The color palette of the video re- Motion Capture Data: Motion Analysis and Mapping to
flects that of the painting, slightly filtered, and is in- Music. Proceedings of the 2002 Workshop/Sympo-
tended to play a thematically supporting role by sium on Sensing and Input for Media-centric Systems.
suggesting associations to the painters style, vocab- Santa Barbara, California.
ulary, and object types. The intention of the work is Bregman, Albert. 1990. Auditory Scene Analysis. Cam-
to tell a story about arts construction, identity, and bridge, Massachusetts: MIT Press.
Milosz, C. 1951. The Captive Mind. New York: Vintage.
Rudi, J. 1998. Computer Music Animations. Organised
Sound 3(3):193198.
Sack, B. 2003. DSP: Digital Sound Processing. Com-
Conclusion puter Music Journal 27(1):9293.
Schaeffer, P. 1966. Trait des Objects Musicaux. Paris:
Integration of video and computer music expands Editions du Seuil.
the composers opportunity for crafting multi- Schtze, S, 2003. The Creation of an Audio Environment
faceted narratives, and it changes the art-making As Part of a Computer Game World: The Design for
process through the use of new types of tools. The Jurassic ParkOperation Genesis on the Xbox As a
composer can use video to augment the musical Broad Concept for Surround Installation Creation.
idea or to develop a storyline not easily conveyed Organised Sound 8(2):170180.
Siegel, W. 1998. The Challenges of Interactive Dance:
through sound alone. As discussed, the artistic idea
An Overview and Case Study. Computer Music Jour-
may also be made apparent through the use of video nal 22(4):2943.
as a tool for meta-composition, or as an educational Smalley, D. 1986. Spectro-Morphology and Structuring
tool to bring salient (but not necessarily prominent) Processes. In S. Emmerson, ed. The Language of
elements to the listeners attention. In other words, Electroacoustic Music. London: Macmillan, pp. 6193.
in computer music video, imagery and audio are Smalley, D. 1997. Spectromorphology: Explaining
able to articulate and mirror their respective and Sound-Shapes. Organised Sound 2(2):107126.
mutual compositional and artistic characteristics
and intents.

44 Computer Music Journal