A Sense of Connection:

Synesthetic Metaphor, Intermedia, and the Possibility of

Acquired Synesthesia

Dave Wall

The remarkable thing is not that "synesthetic art" turns out to be a chimera, but that it is
such a compelling and useful chimera for the production of art.
Paul Hertz
Indeed, there is good evidence to suggest that our senses aren’t as separate as you might
think…
Jamie Ward

Introduction

The world of the senses is not what many of us imagine it to be. Research has uncovered
an array of senses that we use when we encounter the world; most of us think of five, but
researchers have discovered anywhere between twenty-one and thirty-three. There are
two views for the way in which we experience physical reality: the modularity thesis
which most people hold to that assumes five independently functioning senses; and the
unitary thesis which supports the idea of one integrated sensory organ with five
suborgans (van Campen 1999, 9). The concept of five senses comes from Aristotle, who
believed that each sense was linked to a sensory organ. In fact, there are a number of
specialized cell receptors for each sense that respond to specific signals. The retina, for
example, contains a cell that responds to light and another that responds to color, and the
color receptors can be subdivided into three types. Depending on how we count these, the
number of senses we come up with could be anywhere between twenty-one and thirty-
three (Ward 2008, 33).

If we now add synesthesia – the fusion of one or more senses – to this line of thought, the
possibilities for artistic work expands greatly. We may not experience the work of these
specialized cells, but we can consider their existence metaphorically when we produce
art. It makes sense that the more cell receptors a sense contains, the more range a
metaphor based on synesthesia has, and the deeper it can potentially be. Language, and
by extension metaphor, conditions how we ascribe meaning to experiences (Paul Hertz
1999, 9) and to works of art. The deeper the meaning, the more profound the artwork.

For artists, the fact of an existing neurological condition – synesthesia - is nowhere near
as interesting in terms of artistic possibilities as using synesthesia as a metaphor to create
new art. Chromesthesia – the fusing of color and sound – is the most common type of
synesthesia, and dropping that possibility alone into the idea stream has led artists to
metaphorically mix various sense combinations in the production of art.

It has been suggested that the synesthetic state can be produced by consciously creating
associations between different senses. Some artists have made a practice of creating these
associations so that, rather than working within the contrived situation of “making
synesthetic art”, the artist can use an actual, personal synesthetic reaction as a tool. The
fusion of the different senses becomes a natural, effortless act rather than a deliberate
contrivance (Ox 1999a, 7- 8).
2

At this point, however, synesthetic art produced by non-synesthetes using synesthetic

metaphor is for the most part a contrivance, something that needs to be worked out in the
mind of the artist, not something that occurs naturally. Naturally occurring synesthesia
means that connections are experienced spontaneously as personal reactions to the world.
Paul Hertz suggests that persons, not artworks, are synesthetic and offers the term “cross-
modal art” as a more precise term than “synesthetic art” when describing non-synesthetic
artists, while recognizing that the fusion of the senses represented by synesthesia is what
fuels the imagination of these artists (Hertz 1999, 400). In the case of the art producer
who is a synesthete, synesthetic art is an appropriate term for what they produce. While
keeping Hertz’s distinction in mind I will, for the purposes of this paper, be using
“synesthetic art” to describe the work of both synesthete and non-synesthete artists. Both
artists are capable of creating synesthetic artistic experiences, which I think of as the
simultaneous perception of multiple stimuli in a single experience. This type of
experience in art goes by the name of intermedia, which I will look at later in the paper.

Both synesthete and non-synesthete artists use synesthetic imagery in order to access
multiple, simultaneous meanings that can be freely combined, re-combined and un-
combined. As Hertz points out, “we are tied neither to deep structures nor to arbitrarily
determined symbolic codes, but free to play with the whole range between them” (Hertz
1999, 404). Synesthetic art gives us the ability “to use aleatoric juxtaposition, partial
mappings and associations, parallel coupling and complementarity freely as a
metalanguage of cross-modal composition, a counterpoint of rupture and cohesion”
(Hertz 1999, 404). This freedom has led to difficulties in establishing universal
correspondences between senses (i.e. this image equals that sound) and this is seen
clearly in the production of graphic music scores and the use of color organs. The
difficulty in finding a common visual language for extended instrumental technique in
conventional music notation is also related to this lack of correspondence between the
senses. While this is often seen as negative, it creates the sense of freedom outlined by
Hertz above and can lead to the creation of personal, idiosyncratic systems.

I have touched on the main issues that I will look at in this paper – synesthesia as
metaphor, intermedia, and the development of synesthesia in non-synesthetes. The latter
is somewhat controversial and has been likened to a musician developing good relative
pitch. It may well become that banal someday, but for now it exists more as an idea than
as reality, and points toward an artist who is naturally adept at inhabiting whichever
artworld he/she finds necessary for the effective communication of ideas.

A brief survey of synesthesia research and experimentation will serve to contextualize

synesthesia and introduce some of the machines that were produced through the synesthetic
metaphor, machines that helped keep the idea alive in the minds of artists and the public in
general. I will also consider two painters, Paul Klee and Wassily Kandinsky in some depth,
as contrasting examples of approach in using the synesthetic metaphor. I will also look at
abstract cinema pioneer John Whitney as an example of someone who used the synesthetic
metaphor to create a color music system of great depth and complexity.
3

Research and Experimentation: Synesthetic Metaphor in History

Pitch and color correspondences have been a source of fascination for philosophers and
artists for centuries. Aristotle concentrated on lightness, ordering colors on a scale from
black to white and linking pitches to corresponding gray values. Pythagoras demonstrated
that pitch could be measured by the length of strings, inspiring other philosophers to
consider the possibility of a similar system of measurement for color.

John Locke, in his Essay Concerning Human Understanding of 1694, raised a

hypothetical example of a blind man who saw scarlet when a trumpet was played. While
he clearly believed that vision was necessary in order to experience color and was not
using this as an example of synesthesia, the example nevertheless prompted heated
international discussions during the eighteenth and nineteenth centuries concerning the
possibility of correspondence between color and sound (Peacock 1988, 398).

In his Optiks of 1704, Isaac Newton, in an effort to create a quantifiable way to connect
mathematical systems of optics with musical harmony, linked sound oscillation to the
wavelength of light. He posited that the distribution of white light in the color spectrum is
analogous to how tones are distributed in the octave, identifying seven discrete “light
entities” which corresponded to the seven notes of the octave, an somewhat arbitrary
correspondence since there is no natural relationship between the two systems.

Not long after, the synesthetic metaphor becomes actualized in the work of French
mathematician Louis-Bertrand Castel who started to develop the first color harpsichord
(clavecin oculaire) around 1720, inspired by Newton’s work. Though impractical in
terms of the concert hall – strips of paper lit by candles did not have the power to create a
convincing visual experience in such a large space – it was the first in a line of color
organs that would stretch into the twentieth century, culminating in the work of Thomas
Wilfred in the 1920s and 30s.

These instruments did not play sound (Peacock 1988, 402). They employed synesthetic
metaphor by simply assigning color to each key on the keyboard, using music as a way of
structuring a temporal visual art. Temporality is perhaps the most obvious musical
attribute not present in painting, and many artists in the first decades of the twentieth
century - futurists, members of De Stijl and Der BlaueReiter – were looking for ways to
add it to their art through visual suggestions of movement (van Campen 1999, 11).

Thomas Wilfred was perhaps most influential figure in developing a color - music
correspondence, and had the most sustained popularity of any previous color organist,
developing what he called the art of Lumia. The quality of his work and his association
with the Museum of Modern Art in New York contributed to his notoriety and brought
his work to the attention of a great many other artists, and helping to keep the synesthetic
metaphor alive.

After initially using the musical analogy to explain his work, Wilfred rejected the idea of
a correspondence between light and sound, yet his art resembled music by including
factors of time and rhythm in live performance. Music existed in his work as a structural
idea and strongly influenced the production of his art even though he felt that “attempts
to design Lumia instruments in imitation of musical ones will prove as futile as attempts
to write Lumia compositions by following the conventional rules laid down for music”
(Eskilon 203, 67).

………………

Paul Klee and Wassily Kandinsky are important examples of the synesthetic metaphor
influencing painting at the highest level. Both of these artists experimented with music as
a way of influencing and structuring their work, and both developed a critical language
based on music to describe their experimentation and to communicate a new approach to
visual art.

Klee focused on developing an abstract visual language based on historical musical

models, but his approach did not include overt references to synesthesia as did
Kandinsky’s. He was reluctant to fully embrace the musical metaphor in his painting, but
he nevertheless moved towards it. He concentrated mainly on rhythm, which he
considered to not only mark the movement of time in music, but also in painting. His
early works demonstrate his approach with “the translation of motifs taken from nature
into free, rhythmical linear structures and tonal values…” (Duchting 2004, 16). Rejecting
the idea that color could be mapped onto the musical scale, he struggled to find a
technique that would communicate his sense of color.

Influenced by Robert Delauney, he began using contrasting colors to create structural

features and continued to explore the relationship between music and painting. He
created ‘fugue’paintings as a way of solving compositional problems involving
temporality. Fugue in Red (fig. 1) demonstrates the way in which temporality is achieved
as the various forms seem to emerge from dark to light. This piece is related to abstract
cinema artists such as Hans Ruttman, who knew Klee, and Ludwig Hirschfeld-Mack who
visited the Bauhuas where Klee was teaching and displayed a succession of colored
forms expanding and contracting through the use of lamps of varying strength.

5
 fig. 1

With the piece In Bach’s Style (fig. 2), he creates a linear framework reminiscent of a
musical score, the fermata in the left-hand corner becoming more a physiognomic
hieroglyphic (in this case an eye) than a musical symbol.

fig. 2

Rose Garden (fig. 3) is another example of what could easily be interpreted as a musical
score, the roses becoming notes and the use of line and color creating rhythmic values
and/or formal attributes.

6
 fig. 3

The use of color as a metaphor for music continued to elude him. While he was able to
quantify line and light/dark (note duration and pitch), color, for him, corresponded to the
expressive, unexplainable, and irrational content of musical tones, not the tones
themselves. As such, he felt they could not be quantified. As he said: “Color has always
had something secretive about it that is difficult to grasp…The colors are the most
irrational elements in painting” (Duchtung 2004, 45).

Our knowledge of Klee’s history as a musician along with the names he gave to some of
his paintings encourage us to find musical meaning in them. In paintings like Ancient
Sounds (fig. 4), shifting shades of darker color seem to imply harmonic structure while
the lighter colors emerge as single lines of melody. His “polyphonic” paintings, “in
which as many as eight colored planes are laid on top of one another” (Duchtung 2004,
67), create depth and the implication of musical polyphony. Dots, lines, and color
combine in Rising Star (fig. 5) to imply a composition of many voices. Klee was after an
“ecstatic truth” beyond the simple assignation of color to pitch, and used music as a
“mediator between the world of appearances and artists’ efforts to fathom the laws of the
universe” (Duchtung 2004, 88).

7
fig. 4

fig. 5

8
Wassily Kandinsky, by contrast, was seeking this ecstatic truth by blending the arts into a
spiritually felt, synesthetic meshing of sensations. He considered colors to be living
energies with their own intrinsic ‘sounds’. In his production of Mussorgsky’s Pictures at
an Exhibition, he included abstract forms and colors, as well as lighting effects and
movement in an effort to create a synesthetic art. In discussing the way in which different
art forms affect different people, Kandinsky says, in Concerning the Spiritual in Art,
“There reside...in arts which are outwardly different, hidden forces equally different, so
that they may all work in one man towards a single result, even though each art maybe
working in isolation” (Kandinsky 1977, 43). Kandinsky argues for what is in essence a
synesthetic art, a fusion that results in “a single result,” pointing toward the development
of intermedia.

Deeply affected by Arnold Schoenberg’s music, he conceived of an independence of

forms related to the perceived independence of the instruments in Shoenberg’s music,
freed from traditional harmony. Inspired by Shoenberg, he claimed that objects in a
painting could be defined exclusively in terms of “tensions between forces” (Leggio
2002, 109). Kandinsky understood from music theory that the idea of art is the
exteriorization of these tensions, which he thought of as the artist’s inner necessity
leading, in Schoenberg’s music, to an experience that is purely spiritual.

Kandinsky’s love of the circle owes much to Schoenberg’s replacing of linear harmony
with a circular conception in which the tensions created are not resolved in terms of
harmonic closure. This is also evident in Wagner’s “endless melody” in Tristan und
Isolde. Schoenberg saw this lack of closure as an “analogy with infinity” which “could
hardly be made more vivid than through a fluctuating…unending harmony” (Leggio
2002, 115). Here we see the importance of the spiritual aspect so important to both these
men, a goal Kandinsky saw being attained through synesthetic art or, as mentioned
earlier, “a single result,” implying a “oneness.” His painting Several Circles demonstrates
a sense of this spiritual aspect, evoking a sense of the cosmos and infinity (fig. 6).

9
 fig. 6

In his book Point and Line to Plane, Kandinsky theorized on the individual graphic mark
– the dot or point - which he feels is directly equivalent to the fundamental musical tone.
This led to a graphic translation of passages from Beethoven’s Fifth Symphony (fig. 7).

fig. 7

10
The point’s “inner qualities…take on an increasingly powerful sound. These inner
qualities…emerge one after another from the depth of its being...the dead point becomes
a living being” (Leggio 2002, 105). This is, of course, a personal interpretation of the
relationship between image and sound and as such is outside of a common, agreed upon
language, much like Newton’s correspondence of the color spectrum to musical pitches,
or the numerous color-tone analogies that have been created since Castel’s first color
organ (Daniels 2010, 345).

Kandinsky’s painting Composition VIII (fig. 8) is a good example of his concept of the
living point with different energy levels radiating from each point/circle in the painting.
Kandinsky’s work is seen as an influence on composers producing graphic scores; this
particular piece was sonified by Todd Barton, composer and composition teacher at
Southern Oregon University, and posted to the Vimeo video site
(http://vimeo.com/6766060).

fig. 8

John Whitney’s obsession with translating the “liquid architecture” of music into film led
him to question the nature of music and to search for a visual code that would match that
of music’s, and allow him to structure visual material in time. As implied earlier,
however, the physical parameters of visual and auditory media provide us with no clear
system of correspondences. We can invent things, but the fact remains that
correspondences among media are arbitrary and are conditioned by “fuzzy cultural
practices and psychological preferences” (Hertz 1999, 403). Nevertheless, Whitney was
determined to overcome these obstacles and establish a system by which music
corresponded logically to image. When attempts to map Shoenberg’s concept of the tone
row onto filmstrip failed, he abandoned the movie camera and began an effort to invent a
machine that he could use to create images that would accurately reflect music’s

11
abstraction. This was an effort to create a non-arbitrary language/system for producing
synesthetic art.

Identifying certain musical truisms, such as the fact that music has motion and that the
relationships between notes are what create music, provided insight into how to approach
visual motion. He found that musical resolution of the tension created by musical motion
and note relationships is directly related to a resolution to a “visual harmonic resonance,”
which he described as follows: “First, motion becomes pattern if objects move
differentially. Second, a resolution to order in patterns of motion occurs at points of
resonance [or resolution]. And third, this resolution at resonant events…characterize
differential resonant phenomena of visual harmony” (Whitney 1980, 40). “Differential
resonant phenomena” refers to resonance (resolution) achieved through differential, or
independent, ordered movement. This led to his detection of charge and discharge of
tension in graphic pattern arrays and his connection of harmonic forces/tensions in music
being related to the same in graphic display (Whitney 1980, 43). This recalls Kandinsky’s
tensions between forces.

Rather than an up-down, left-right orientation that color organists had previously used to
evoke the musical scale, Whitney proposed his concept of differential motion pattern,
mentioned above. This involved defining sixty points along a line, each point moving a
different amount. For instance, point #1 moves one step up, point #2 moves 2 steps up,
point #3 moves 3 steps up until point #60 is reached. If point #1 moves 2 steps, then #2
moves 4 steps, #3 moves 6 steps. This could also be applied to a set of elements, such as
an array of triangles. Whitney sums it up:

This is the differential principle, which is applied in various

ways to all motion of typical scales of a hypothetical
instrument. Any scale ascends a ladder of tensional
structure in the time dimension. The progressive rise and
fall of tension…is characterized by recurrent nodes, where
tension is resolved to equilibrium (resonance) (Italics mine)
(Whitney 1980, 49).

While this is idiosyncratic to Whitney, it represents a system less arbitrary than any
presented thus far if only because of its detail, and its inclusion of musical tension in
relation to visual tension. The depth and authority of the work would make it easy to
accept as a standard operating procedure for marrying image with music.

This system is very personal to him, and it is difficult to see the relationship between
sound and image in his films. Being able to determining his success in fusing the two is
dependant on being able to watch his work with new eyes. Tensional structure is the clue.
Watching for – and being able to see - changes in the images that are related to the rising
and falling of musical tension/energy is key. When encountering his work we have to
discard conventional notions of pitch/color relationships, or the conventional expectation

12
of music corresponding to visual movement in obvious ways.

This scratches the surface of Whitney’s thought and work in this domain, but
demonstrates a practical, systematic approach to the synesthetic metaphor that goes
deeper than any we have yet encountered. Two of his films – Matrix III and Arabesque –
are included in the filmography.

Becoming Synesthetic

It is known that we are born as auditory-visual synesthetes. Until 6 months of age,

responses to spoken language are recorded in the areas of the brain that process both
hearing and sight. This ability gradually disappears until it is gone completely at 36
months of age (Afra, P. 2009, 35).

In a limited sense, however, we are all still synesthetes. Jamie Ward describes multi-
sensory neurons in the brain that fuse sound and vision. These neurons become amplified
in certain situations, and where we look becomes where we hear. This explains why, in a
movie theatre, we experience the sound coming from the screen when the speakers are
placed on the sides of the room. Similarly, a voice that goes unheard in a noisy room can
be heard if we watch the speaker’s lips. These examples of sound-sight fusion represent
an indirect, yet physical link to synesthesia. In these cases, entire areas of the brain that
privilege sight or sound aren’t fused; individual neurons are. Our brains contain a type of
sense fusion, indicating a possible path we can take to acquire synesthesia. Is the
acquisition of synesthesia possible?

There are indications that it is. The new neuroscience of optogenetics, using, as the name
implies, a combination of genetics and optics, is providing the tools for brain mapping
with great precision. Essentially, neuroscientists can control brain circuits in laboratory
animals, turning cells on or off with laser light, and replacing damaged circuitry with new
wiring from a different location. As Dr. Tim Murphy of the Brain Research Centre in
British Columbia says, “There is an actual sprouting of connections and structural
rearrangements.” (Anne McIlroy, “Shining a light inside the brain,” The Globe and Mail,
April 2, 2011). This research is very new and is being used to investigate brain function
in stroke patients and the neural circuitry involved in mental illness. Given the ability to
create connections and rearrange structure, it does not seem to me to be a stretch to say
that this technology could be used to connect different areas of the brain that involve the
senses.

As it stands now, synesthesia can be naturally occurring or, more interestingly for the
purposes of this paper, acquired. It can be acquired through sensory loss or temporarily
through the use of certain drugs. Ward states that, “…synesthesia is effectively an
adaptation of the normal processes of multisensory perception that is universal among
humans…” (Ward, p. 62) indicating the possibility of switching on pathways in the brain
that permit synesthesia.
13
LSD and mescaline are known to activate serotonin 2A receptors. Auditory visual
synesthesia has been reported with the activation of serotonin 2A receptors in the brain,
also known as the “synesthesia receptor”. (Afra, P. 2009, 35) These drugs can produce
profound results within an hour. Since it is inconceivable that the brain could rewire itself
so quickly, the suggestion is that our brains are already intertwined and connected
synesthetically (Ward 2008, 21). The drug effectively unblocks the connection. The
research of Tor Wager, psychology professor at Columbia University indicates that, “all
the drugs in the world are already in our brains; that’s why we have receptors that are
able to make sense of synthetic or artificially introduced versions [of sensory events]. The
real trick is figuring out how to prompt our brains to release the right stuff at the right
time” (Hanus 2008, 1). Wager’s research indicates the possibility of voluntarily releasing
the drugs necessary for unblocking connections in the brain that produce synesthesia.

Attempts to enable blind people to “see” by using software to convert visual images to
sound using a hidden camera mounted on the wearer’s head have had success. Higher
sounds denote objects higher up in the image and loudness is associated with brightness.
The visual image is heard as a soundscape scanned from left to right. Pat Fletcher,
blinded in an industrial accident and one of the first users, insists that it is like seeing, not
hearing. She sees a visual representation of the soundscape, thus representing a person-
made synesthesia (http://www.artificialvision.com/).

Many artists have stated it is possible to develop and/or increase synesthetic awareness
simply by making it part of their practice while creating art. For example, Jack Ox has
been experimenting for over twenty years with how to combine different media into one,
and claims that it is now natural for her to see sonic forms (Ox, 1999, 7). This supports
Ward’s conclusion that a slow reorganization of multisensory pathways is possible.
(Ward 2008, 62) This reorganization often occurs after sensory loss of some kind, but the
experiences of Ox and other artists indicate that sustained effort may also produce
synesthesia. Kandinsky is perhaps the most famous example of this, having, as is
commonly believed, “developed his abilities through associative techniques aimed at
enhancing sensory exchange, much like one might develop relative pitch” (Ione 2004,
154). My own work as a musician has enabled me to develop a specialized type of sound
– symbol synesthesia that every trained musician develops. If I see a melodic passage in
musical notation, I can hear it. If I hear a melodic passage, I can see it in musical
notation.

But why would we want to develop synesthesia? Would it create better artists?
Ramachandran and Hubbard have this to say:

Synesthesia causes excess communication amongst brain

maps…Depending on where and how widely in the
brain the trait was expressed, it could lead to both
synesthesia and to a propensity toward linking
 14

seemingly unrelated concepts and ideas – in short,

creativity (Ward 2008, 127).

Developing synesthesia would mean, essentially, breaking down or “eliding the division
and hierarchies we apply to the senses” (Barker 2008, 2), creating a larger ground for the
production of ideas. To recycle a statement from earlier in this paper, it would produce
artists “who are naturally adept at inhabiting whichever artworld” (or simultaneous
artworlds) “they find necessary for the effective communication of ideas.” This leads
nicely to a discussion of intermedia.

Intermedia
Richard Wagner’s aesthetic approach of Gesamtkunstwerk, the uniting of the visual, the
aural, and narrative text into a single artistic unity, is directly related to a synthesis of the
senses and is an early example of multimedia. However, his idea of a single artistic unity
– influencing Kandinsky and his idea of a “single result” - points towards an intermedial
art with its ideas of “commingled, shared structural patterns coming from completely
different worlds” (Ox 2001, 47). In other words, once someone thought of combining art
forms as autonomous constructs in one space, it was only a matter of time before artists
started combining different artistic syntaxes into a single thing.

Dick Higgins named and described intermedia in the 1960s (Higgins 1984, 2), and
provided examples; Gene Youngblood defined a necessary component when he spoke of
synergy in which the sum of the whole is greater than its parts. When these “parts behave
with integrity and without self-consciousness” (Youngblood 1970, 110), intermedia is
possible. The parts must fuse to create a stronger whole; independent parts existing in one
space without fusing – as in Gesamtkunstwerk - is multimedia. Youngblood, writing
about cinema in 1970, was convinced that the current environment involving a multi-
dimensional, simulsensory network of information sources could most effectively
function using a synesthetic aesthetic language (Youngblood, p. 77), resulting in
intermedia. Refining the definition further, Yvonne Spielman believes that an obligatory
part of intermedia is a “combinatory structure of syntactical elements that come from
more than one medium but are combined into one and are thereby transformed into a new
entity” (Ox 2001, 48).

The possibilities for the combining of different media into a new form have increased
exponentially with the computer, making it much easier to perform intermedia. The
popularity of chaos and complexity theories in the latter part of the twentieth century has
contributed to this process of fusion. Systems are more easily connected and we also have
a very precise collage tool. We can cut and paste multiple layers; our artistic procedures
can achieve greater and greater complexity, creating new possibilities (Ox 1999b, 391).
As Paul Hertz points out "computers make it possible to use these deep structures to
build cross-modal works of art from the smallest perceptible element up” (Hertz 1999,
15
403).

In 1996, intermedia/synesthetic artist Jack Ox began work on the visual translation of

Quanta and Hymn to Matter, a large-scale symphonic and choral work by the American
composer, Dary John Mizelle. The visual representation was produced by what she
named “The 21st Century Virtual Reality Color Organ.” I present this work as an
example of contemporary intermedia being dependent on the use of technology for its
realization.

The 21st Century Virtual Reality Color Organ is a

computational system for translating musical compositions
into visual performance. It uses supercomputing power to
produce 3D visual images and sound from Musical
Instrument Digital Interface (MIDI) files and can play a
variety of compositions. Performances take place in
interactive, immersive, virtual reality environments such
as the Cave Automatic Virtual Environment (CAVE),
VisionDome, or Immersadesk. Because it’s a 3D
immersive world, the Color Organ is also a place—that is,
a performance space (Ox,
http://www.jackox.net/IEEE.pdf).

Ox mixed over 130 RGB colors to express the timbres of the orchestra, which included
strings (fig. 9), woodwinds (fig. 10), brass (fig. 11), percussion, and vocals (fig. 12). Each
section is represented by sets of colors in a graduated series of hues, with additions of
other colors expressing the sound resulting from the application of different instrumental
technique. Below are her charts for strings, woodwinds, brass, and vocals. She does not
provide specific explanations of which colors in each chart express which parameters.
She does, however, provide a page of the score over which she has laid color
corresponding to the appropriate parameters. Timbre is represented by hue and dynamics
are represented by saturation (fig. 13). Assuming a standard score with woodwinds at the
top, followed by brass, percussion and strings gives us a little more insight into how the
system works, but in the end we would need three things to truly understand her system:
the score with overlaid color and instrument sections clearly marked; the original score
without color so that specific musical indications are not obscured by overlaid colors; and
a performance of the score. She has not provided a color chart for percussion.
 16

fig. 9 strings fig. 10 woodwinds

 17

fig. 11 brass

fig. 12 vocals
 18

fig. 13

The poem on which the piece is based describes a very harsh landscape. In order give this
landscape expression in the work, Ox travelled to deserts in California and Arizona and
photographed landscapes that she felt expressed in their sculptural qualities sound
making concepts of different instrument families. She then made drawings which were
transformed into 3D through modeling and placement in the VR world of the CAVE, a
virtual reality theater originally developed by the Electronic Visualization Lab at the
University of Illinois, Chicago. This hand drawn and modeled world is the canvas and the
image source for surface textures on which the timbral colors lay.

This is a sketch that Ox made in order to show how the individual landscapes would be
put together (fig. 14). Closest to the front is brass, followed by strings, with the ledge on
which are placed the smaller units of percussion. Behind this ledge are woodwinds with
vocals in the very back. Individual landscape images for each instrumental group can be
found here: http://home.bway.net/jackox/colororgansite/.

19
 fig. 14

In the virtual reality environment it looks like what we see in fig. 15. Referring back to
the color and landscape charts for each instrumental section allows for a kind of
synesthetic experience as the colors translate to sound. This takes a certain amount of
study and interpretation. For instance, the green could either be strings, woodwinds or
vocals, or a meshing of the three. The final product can be seen as a transcription
translated into 3D in a virtual environment, but it is more than that. It is a work designed
to evoke strong emotional responses in the viewer.

20
 fig. 15

The music is represented by MIDI. The following is Ox’s description of how the music is
realized in virtual reality:

Each note as it is played creates a rectangular strip located

in a specific place over the landscape, which is determined
by time and pitch. Pitch is the y axis and determines how
high it is placed in space. Time is the x axis, going from
left to right, so time says where the strip begins and ends in
space. The initial attack of the note, which is how loud it
begins, determines the width, or z axis of the rectangular
strip, which remains constant until the note ends. Further
changes in loudness or softness are reflected in the
saturation of the timbre color which is part of the surface of
the strip (Ox, http://home.bway.net/jackox/colororgansite/).

This piece falls under Higgins definition of intermedia as a work that exists between
mediums (Higgins 1984, 49), combining the syntaxes of music, color, movement, and
sculpture in a 3D virtual environment.

This quick time movie demonstrates the approach, using a chamber piece by Clarence
Barlow. It is important to note that the viewer experiences the piece from within a world

21
of hand-drawn, black and white landscapes modeled in 3D. Imagine, while viewing the
movie, that you are on the ground looking up at the colored strips, instead of floating
above them (see fig. 15). As I understand it, this is one of the possible ways of
experiencing pieces conceived in this environment. The video gives an idea of the
environment, and demonstrates some of the parameters mentioned above: pitch, timbre,
and loudness (http://www.jackox.net/media/Barlow_apple.mov).
Time, rather than being expressed on a left to right x axis as described above, moves in a
parabolic spiral path in order to visualize the structure Barlow used in his compositional
process (http://www.jackox.net/pages/introImJan.html). Imposing the structure of the
music on the visual medium combines the two syntaxes and provides another example of
intermedia.

Conclusion

I have provided a limited exploration of the history of synesthesia research and

experimentation along with a consideration of the synesthetic metaphor in order to
demonstrate its influence on art, and to determine a way forward in terms of using it as a
viable tool in the production of art. It has clearly had an influence, but research shows us
that it is possible to take what it offers to a higher level, to use it as a naturally occurring
tool in our neurobiological makeup rather than as a contrived strategy for creating art.
This is particularly relevant to intermedia since its aim is to create any number of single,
multifaceted entities out of a variety of the seemingly discrete sources that we presently
think of as information received from individual sense organs.

With acquired synesthesia, these discrete sources of information become a single

multifaceted entity if we voluntarily decide that it be so. With acquired synesthesia, we
would not be forced to construct communication systems before creating art. We would
simply be communicating felt experience. I do not mean to imply that felt experience is
necessarily better than systems creation since engaging the intellect in the creation of
systems is part of effective art. Since acquired synesthesia would allow us to use a
synesthetic response voluntarily, we would continue to be able to exercise choice in this
domain. We can make use of the intellect, immediate felt experience, or both. I believe
that more options, wisely used, are beneficial in general term as well a in the creation of
art.

22
BIBLIOGRAPHY

1. Afra, P., Funke, M. and Matsuo, F., ``Acquired auditory-visual synesthesia: A window
to early cross-modal sensory interactions,'' Psychology Research and Behavior
Management, Vol. 2, January 2009.

2. Barker, J. M. “Out of Sync, Out of Sight: Synaesthesia and Film Spectacle.”

Paragraph, Volume: 31, Issue: 2 (July 1, 2008).

3. Baron-Cohen, Simon. “Is There a Normal Phase of Synaesthesia in Development?,"

Psyche,Vol. 2 (psyche.cs.monash.edu.auf/i,l ename:p syche-96-2-
27-syn_development-baron_cohen). Quoted in Hertz, Paul. “Synesthetic Art: An
Imaginary Number?” Leonardo, Vol. 32, No. 5, Seventh New York Digital Salon (1999),
pp. 399-404.

4. Berman, Greta. "Synesthesia and the Arts," Leonardo 32, No. 1 (1999), pp. 15-22.

5. Brougher, Kerry, Jeremy Strick, Ari Wiseman, Judith Zilcher and Olivia Mattis. Visual
Music: Synesthesia in Art and Music Since 1900. Los Angeles: Thames and Hudson,
2005.

6. Carpenter, S. “Everyday fantasia. The world of synesthesia.” APA Monitor on

Psychology, vol. 32, No. 3 (2001).

7. Carrà, Carlo. "The painting of sounds, noises and smell."

http://www.unknown.nu/futurism/paintsound.html

8. Conrad, Daniel. "The dichromaccord: Reinventing the elusive color organ." Leonardo,
32, no. 5 (1999), 393-398.

9. Cytowic, Richard. Synesthesia: A Union of the Senses. Cambridge, Mass: MIT Press,
2002.

10. Dailey, A., Martindale, C., Borkum, J. "Creativity, synesthesia and physiognomic
perception," Creativity Research Journal 10, 1-8 (1997).

11. Daniels, Dieter and Sarah Neuman. Audiovisuology: A Compendium. Koln: Walther
Konig, 2010.

12. Duchtung, Hajo. Paul Klee: Painting Music. Munich: Prestel Verlag, 2004.

23
13. Eskilon, Stephen. “Thomas Wilfred and Intermedia: Seeking a Framework for
Lumia.” Leonardo, Vol. 36, No. 1 (2003), pp. 65-68.

14. Galeyev, Bulat. ”What is synaesthesia: myth and reality” "Leonardo Electronic
Almanac, vol.7 nr. 6 (1999). http://prometheus.kai.ru/mif_e.htm.

15. Galeyev, Bulat. “Synaesthesia and musical space. On Yavorsky's forgotten

hypothesis "Leonardo, vol. 26, nr. 1 (1993), pp. 76 - 78.

16. Hertz, Paul. “Synesthetic Art: An Imaginary Number?” Leonardo, Vol. 32, No. 5,
Seventh New York Digital Salon (1999), pp. 399-404.

17. Higgins, Dick. Horizons: The Poetics and Theory of the Intermedia. U.S.: Southern
Illinois University Press, 1984.

18. Howes, David. Scent, Sound and Synaesthesia: Intersensoriality and Material Culture
Theory. In: Tilley, Christopher et al. (eds.);. Handbook of Material Culture (pp. 161-72).
London, England: Sage, 2006.

19. Ione, A. "Kandinsky and Klee: Chromatic Chords, Polyphonic Painting and
Synesthesia." Journal of Consciousness Studies 11, no. 3-4 (2004), pp. 148-58.

20. James, David E. editor. Stan Brakhage: Filmmaker. Philadelphia: Temple University
Press, 2005.

21. Kandinsky, Wassily. Concerning the Spiritual in Art. New York: Dover, 1977.

22. Kandinsky, Wassily. Point and Line to Plane. New York: Dover, 1979. Quoted in
Leggio, James, editor. Music and Modern Art. New York: Routledge, 2002.

23. Leggio, James, editor. Music and Modern Art. New York: Routledge, 2002.

24. Li, X., Tao, D., Maybank, S.J. and Yuan, Y. “Visual music and musical vision.”
Neurocomputing, Volume: 71, Issue: 10-12 (June 2008), pp: 2023-2028.

24
25. Mallary, Robert. "Spatial-synesthetic art through 3-D projection: The requirements of
a computer-based supermedium," Leonardo 23, no.1 (1990), pp. 3-16.

26. Miller, Dorothy, curator. 15 Americans, exh. cat., (New York: Museum of Modern
Art, 1952). Quoted in Eskilon, Stephen. “Thomas Wilfred and Intermedia: Seeking a
Framework for Lumia.” Leonardo, Vol. 36, No. 1 (2003), pp. 65-68.

27. Moritz, William. “The dream of color music, and machines that make it possible.”
http://www.awn.com/mag/issue2.1/articles/moritz2.1.html

28. Mueller, Robert Emmet. "Visic: A proposal for true color music." Leonardo, 32, no. 3
(1999) 177-179.

29. Ox, Jack and Peter Frank. “The Systematic Translation of Musical Compositions into
Paintings.” Leonardo, Vol. 17, No. 3 (1984), pp. 152-158.

30. Ox, Jack "Color me synesthesia " Leonardo 32, no. 1 (1999a), 7-8.

31. Ox, Jack. "Synesthetic fusion in the digital age." Leonardo, 32, no. 5 (1999b), 391-
392

32. Ox, Jack. "Creating a visual translation of Kurt Schwitters's 'Ursonate'," Leonardo
Musical Journal 3 (1993), 59-61.

33. Poast, Michael. "Color music. Visual color notation for musical expression".
Leonardo, 33, 3 (2000), 215-221.

34. Peacock, Kenneth. “Instruments to Perform Color-Music: Two Centuries of

Technological Experimentation.” Leonardo, Vol. 21, No. 4 (1988), pp. 397-406

35. Ramachandran, V.S. and Hubbard, E.M. “Synaethesia – a window into perception,
thought and language.” Journal of Consciousness Studies, 8, 3 – 34, (2001). Quoted in
Jamie Ward. The Frog Who Croaked Blue. UK: Routledge, 2008.

36. van Campen, Crétien . The Hidden Sense: Synesthesia in Art and Science.
Cambridge: MIT Press, 2007.

37. van Campen, Crétien .“Artistic and Psychological Experiments with Synesthesia.”
Leonardo, Vol. 32, No. 1 (1999), pp. 9-14.

25

38. Ward, Jamie. The Frog Who Croaked Blue. UK: Routledge, 2008.
39. Whitney, John. Digital Harmony: On the Complementarity of Music and Visual Art.
Tennessee: Kinsport Press, 1980.

40. Wilfred. Thomas. “Composing in the Art of Lumia” Author(s): The Journal of
Aesthetics and Art Criticism, Vol. 7, No. 2 (Dec., 1948).

41. Youngblood, Gene. Expanded Cinema. New York: E.P Dutton and Co., 1970.

FILMOGRAPHY

1. Arabesque by John Whitney and Larry Cuba. http://www.youtube.com/watch?

v=HnlQa74H7is&feature=player_embedded#at=45 (accessed April 1, 2011).

2. John Whitney – Matrix III (1972) http://www.youtube.com/watch?

v=ZrKgyY5aDvA&feature=player_embedded#at=134 (accessed April 1, 2011).

WEBSITES

1. Artificial Vision for the Totally Blind. http://www.artificialvision.com/

(accessed March 16, 2011).

2. Center for Visual Music. http://www.centerforvisualmusic.org/

(accessed March 17, 2011).

3. Colour Music. http://home.vicnet.net.au/~colmusic/ (accessed March 18, 2011).

4. Carpenter, Siri. “Everyday fantasia: The world of synesthesia.” American

Pschological Association Monitor. Vol. 32, no. 3 (March 2001)
http://www.apa.org/monitor/mar01/synesthesia.aspx (accessed March 24, 2011).

5. Galeyev, Bulat M. “Synesthesia: Myths and Reality.”

http://prometheus.kai.ru/mif_e.htm (accessed March 25, 2011).

6. Hanus, Julie. Your Brain is a Pharmacy. The Utne Reader, September – October,
2008. http://www.utne.com/2008-09-01/Science-Technology/Your-Brain-Is-a-
Pharmacy.aspx?page=2#ixzz1I7ZMgRfU (accessed April 2, 2011).

26

7. Heyrman, Hugo. “Dr. Hugo Heyrman (Motions of the Mind).”

http://www.doctor. hugo.org/synaesthesia/index.htm
 (accessed March 20, 2011).

8. McDonnell, Laura. “Colour and

Sound.” .http://homepage.eircom.net/~musima/visualmusic/visualmusic.htm#top
(accessed March 18, 2011).

9. Moritz, William. “The Dream of Color Music and the Machines that Made it
Possible.” http://www.awn.com/mag/issue2.1/articles/moritz2.1.html (accessed
March 18, 2011).
10. MULTI: Multi-user Laser Texture Interface.
http://www.vacuumwoman.com/multi/index.html (accessed March 18, 2011).

11. Natalia Sidler. http://www.nataliasidler.ch/index.php (accessed March 30, 2011).

12. Ox, Jack. “The 21st Century Virtual Reality Color Organ.”
http://home.bway.net/jackox/colororgansite/ and http://www.jackox.net/ and
http://www.jackox.net/IEEE.pdf (accessed March 16, 2011).

13. Ox, Jack. “Im January am Nil.” Music by Clarence Barlow.

http://www.jackox.net/pages/introImJan.html (accessed April 7, 2011)

14. RhythmicLight.com. http://rhythmiclight.com/index.html (accessed March 20,

2011).

15. Stadnick, George O. “The Art of Lumia.” http://www.photonlightguitars.com/

(accessed March 17, 2011).

16. Synesthesia Artforum. http://synesthesia-artforum.blogspot.com/ (accessed

March 22, 2011).

17. The Online Visual Music Forum: Natural and Invented Visual Embodiments of
Music.
http://www.ronpellegrinoselectronicartsproductions.org/Pages/VisualMusicForum
.html (accessed March 20, 2011).

18. Thorisson, K. R. and K. Donoghue. “Synthetic Synesthesia.”

http://xenia.media.mit.edu/~kris/synthsyn.html (accessed March 22, 2011).

27

19. Wikipedia. “Color Organ.” http://en.wikipedia.org/wiki/Color_organ (accessed

March 17, 2011).
20. Wikipedia. “Gesamtkunstwerk” http://en.wikipedia.org/wiki/Gesamtkunstwerk
(accessed March 26, 2011).

21. Wikipedia. “History of Synesthesia Research.”

http://en.wikipedia.org/wiki/History_of_synesthesia_research (accessed March
26, 2011).

22. Vimeo. “Kandinsky Sonified by Todd Barton.” (http://vimeo.com/6766060).

(Accessed March 3, 2011).