Timbre & Musical Orchestration

Marcelo Caetano

Mster en Audio Musical

E.T.S.I. de Telecomunicacin
Universidad de Mlaga
Overview

Timbre
Timbre perception
Multidimensional scaling
Musical timbre
Computer-aided musical orchestration
Timbre
Origins, early studies, timbre perception
Timbre Etymology

Tympanon (Byzantine Greek, Latin, 12th century Old English):

kettledrum, tambourine
Timbre (13th century Old French, 14th century Middle English): small
drum, bell without a clapper struck with a hammer (also a weight, a
helmet, a skullcap, a heraldic crest)
Timbre (French): sound of a bell, sonorous quality of any
instrument or of a voice, characteristic quality of a sound
Timbre

Encompasses a complex set of auditory attributes, psychological and

musical concepts
Timbre covers many parameters of perception not accounted for by
pitch, loudness, spatial position, duration, and various
environmental characteristics such as room reverberation
Timbre is an umbrella term for many spectro-temporal attributes
 Helmholtz
Waveform shaped by amplitude envelope
Periodicity evokes sensation of pitch
Assumes temporal invariance of spectrum
Used Fourier analysis to explain differences

H von Helmholtz. On the Sensations of Tone, 1885.

 Helmholtz
Perception of pitch depends on fundamental frequency
Perception of loudness depends on amplitude
Therefore, perception of timbre must depend on relative amplitude
of partials
Increase in loudness adds energy to higher partials
Nonlinearities of sound production mechanism

H von Helmholtz. On the Sensations of Tone, 1885.

 Helmholtz
Spectral envelope determines timbre

H von Helmholtz. On the Sensations of Tone, 1885.

 Helmholtz
Sinusoids cannot have timbre
Low frequency sinusoids are dull
High frequency sinusoids are sharp
Ignores the influence of phase
Fourier analysis is not a good approximation for the frequency
response of the human ear
Fourier analysis
Linearly spaced center frequencies
Constant bandwidth
Ear (constant Q transform)
Nonlinearly spaced center frequencies
Bandwidth depends on frequency

H von Helmholtz. On the Sensations of Tone, 1885.

 Timbre

Timbre is that attribute of auditory sensation in terms of which a

listener can judge that two sounds similarly presented and having the
same loudness and pitch are dissimilar [American National Standards
Institute, 1960]

Defines of timbre is NOT

Relies on dissimilarity
Implies that timbre, loudness, and pitch are independent
Implies that unpitched sounds do not evoke sensation of timbre
[Bregman 1990]

A Bregman. Auditory Scene Analysis, 1990.

The Perceptual Complexity of Timbre

Timbre is multidimensional
o Continuous dimensions (brightness, attack sharpness)
o Discrete dimensions (Pluck in harpschord, blat in sforzando trombone)
Timbre is one of the primary vehicles for the recognition,
identification, and tracking over time of a sound source
o Involves absolute categorization of sounds
Musical timbre as basis for perceptual dimensions that bear musical
form
o Musical form: relations among perceptual qualities
o Auditory stream segregation
Timbre Research

Isolate dimensions
o How many dimensions?
o What are the dimensions?
Associate verbal labels
o Verbal attributes
o Semantic labels
Find physical variables associated to these dimensions
o Acoustic correlates of timbre perception
o Sound features
Verbal Attributes of Timbre

Independent features that can be described by verbal attributes

Classify sounds according to pairs of attributes
o Dull vs Sharp
o Dark vs Bright
o Rough vs Smooth
o Hollow vs Solid
o Scattered vs Compact
Bias of Verbal Descriptions

Pre-selection of scales and number of dimensions do not necessarily

match the attributes the listener would use
Lack of familiarity with pre-selected attributes
Listeners previous experience influences the vocabulary used
Semantic Differential Analysis

Each sound is rated along many category scales

Dimensional analysis to find small number of orthogonal
dimensions
Factor analysis uses correlation to find small number of dimensions
(factors or unobserved variables) that explain most of the variance
of the observed variables
 Verbal Attributes and Scales of Timbre

Dull vs Sharp (44% variance): Sharpness

Scattered vs Compact (26%): Compactness
Empty vs Full (9%): Fullness
Colorless vs Colorful (2%): Sound Color

G von Bismarck. Timbre of Steady Sounds: A Factorial Investigation of its Verbal Attributes. Acustica,
1974.
Multidimensional Scaling (MDS)
Introduction to MDS, MDS and
psychoacoustics, MDS and timbre spaces
Multidimensional Scaling (MDS)

MDS represents measurements of (dis)similarity among pairs of

objects as distances between points in a low-dimensional space
MDS enables the data analyst to literally look at the data and
explore their structure visually
MDS can also be used as a mathematical model for (dis)similarity
judgements
MDS helps uncover the hidden psychological dimensions
implicitly used to assess (dis)similarity
Multidimensional Scaling (MDS)

MDS used to represent (dis)similarity data as distances in a low-

dimensional space for visual inspection
MDS used to test if and how criteria used to distinguish among
objects are mirrored in empirical differences of these objects
MDS used to discover dimensions that underlie judgments of
(dis)similarity
MDS used as a psychological model that explains judgements of
(dis)similarity in terms of a distance function
MDS: Exploratory technique

Statistics on the rate of different crimes in the 50 US states

Can you predict a high crime rate of murder by knowing that the
crime rate of burglary is high?
Correlations of the crime rates helps answer the question
MDS: Exploratory technique

Small correlation matrix but

Still difficult to understand the structure of the data
MDS provides a visual representation of the correlation matrix
MDS: Exploratory technique

Correlations 2-D MDS representation

MDS: Exploratory technique

Distribution of points represents the correlations

Points close together correspond to crime rates that are highly
correlated
MDS reveals two dimensions: horizontal (violence vs property) and
vertical (hidden vs street)
MDS: Psychological structures

Similarity ratings 2-D MDS representation

Matlab examples
Open timbre_mds.m
Multidimensional scaling
of timbre
Dissimilarity judgments, timbre spaces,
psycholgical dimensions
 Multidimensional Scaling of Timbre

Dissimilarity ratings between pairs of sounds

Find low dimensional orthogonal space where distances represents
dissimilarity
Correlate each dimension of timbre space with acoustic attributes
extracted from sounds
Assign verbal labels to dimensions

G von Bismarck. Timbre of Steady Sounds: A Factorial Investigation of its Verbal Attributes. Acustica,
1974.
 Multidimensional Scaling of Timbre

S McAdams. Perspectives on the Contribution of Timbre to Musical Structure, 1999.

 Multidimensional Scaling & Timbre Spaces

J Grey. Multidimensional Perceptual Scaling of Musical Timbres. JASA, 1977.

 MDS Timbre Spaces

Assumes all listeners use the same underlying dimensions

Assumes high correlation with acoustic features capture physical
cues used in the mental representation
Cannot handle specificities

S McAdams. The Perception of Musical Timbre. Oxford Handbook Music of Psychology, 2008
Timbre Spaces

Recorded musical instrument sounds [Iverson 1993, Lakatos 2000]

Recorded and modified instrument sounds [Grey 1977, Grey 1978]
FM synthesis simulation of orchestral musical instruments
[Krumhansl 1989, McAdams 1995]
 Recorded Modified Instruments

J Grey. Multidimensional Perceptual Scaling of Musical Timbres. JASA, 1977.

 Timbre Space

Spectral centroid
Attack synchronicity/Spectral flux
Attack centroid

J Grey. Multidimensional Perceptual Scaling of Musical Timbres. JASA, 1977.

 Iverson

Spectral centroid
Ampitude envelope
o Different portions (attack/decay) for different sound sets

P Iverson. Isolating the Dynamic Attributes of Musical Timbre. JASA, 1993.

 Recorded Harmonic Instruments

S Lakatos. A Common Perceptual Space for Harmonic and Percussive Timbres, 2000.
 Recorded Percussive Instruments

S Lakatos. A Common Perceptual Space for Harmonic and Percussive Timbres, 2000.
 Harmonic and Percussive Combined

S Lakatos. A Common Perceptual Space for Harmonic and Percussive Timbres, 2000.
 Harmonic and Percussive Combined

Spectral centroid
Rise time
Richness

S Lakatos. A Common Perceptual Space for Harmonic and Percussive Timbres, 2000.
 FM Synthesis Instruments

C Krumhansl. Why is Music Timbre so Hard to Understand? Perspec Electroac Mus 1989.
 FM Synthesis Instruments

Temporal envelope
Spectral envelope
Spectral flux

C Krumhansl. Why is Musical Timbre so Hard to Understand? Perspec Electroac Mus 1989.
Musical Timbre
Musical instruments, musical
orchestration, timbral combinations,
computer-aided musical orchestration
Musical Orchestration

Refers to composing music for an orchestra

Initially orchestration was simply the assignment of instruments to
pre-composed parts of the score
Gradually orchestration has become part of the compositional
process
Nowadays musical orchestration involves timbral combinations
Timbral combinations arise when playing multiple instruments
simultaneously to achieve desired effects

42
Timbral Combinations

I ask you Can you play the instruments of an orchestra to resemble

a person screaming?
Why would anyone want to do that? You ask me back
Well

43
Timbral Combinations

44
Computer-Aided Musical Orchestration (CAMO)

Large databases of musical instrument sounds

Find combination of notes from musical instruments that best
approximates a given target sound perceptually
Desired output is musical parameters: pitch, dynamics, instrument
Timbral combinations to match a target sound

Target Orchestrations

45
 Formalization of CAMO

Autonomous exploration of instrument combinations

Complexity of timbre perception [1]
o Multidimensional perceptual phenomenon
o Composers use non formalized knowledge
o Codify perceptual similarity between sounds
Combinatorial optimization problem
o Combinatorial explosion
o Knapsack problem
o NP-complete
Constraints
o What instruments are available
o How many of each instrument

[1] S. McAdams, B. Giordano (2009). The perception of musical timbre. In S. Hallam, I. Cross, & M. Thaut (Eds.), The Oxford handbook of music psychology, 7280. New York,
NY: Oxford University Press.
46
Musical Instrument Timbre

Timbre Perception Timbre Space [2]

The ways in which sounds are
perceived to differ
Perceptual difference not
accounted for by pitch, loudness,
spatial position, duration, and
environmental characteristics
Categorical view: sound
source recognition,
identification, tracking in time
Sensory view:
multidimensional set of
attributes associated with timbre
spaces

[2] J Grey. Multidimensional Perceptual Scaling of Musical Timbres. Journ. Acoust. Soc. Am., 61(5):12701277, 1977. 47
CAMO with Genetic Algorithms (GAs)

Orchide [3] is the state of the art for CAMO

Orchide uses GAs and local search
Local search explores the neighborhood of a solution
Orchide performs constrained optimization

Target Strings Woodwinds

[3] G. Carpentier, G. Assayag, and E. Saint-James. Solving the Musical Orchestration Problem using Multiobjective Constrained Optimization with a Genetic Local Search
Approach. Journ. Heuristics, 16(5), pp. 681714 (2010).
48
CAMO with an Artificial Immune System

Preliminary results

Target: Car Horn Orchestrations

49
CAMO with an Artificial Immune System

Preliminary results

Target: Ahh Orchestrations

50
CAMO with an Artificial Immune System

Preliminary results

Target: Scream Orchestrations

51