AES121heyser Smith

History and Practice of Digital Sound Synthesis
Julius Smith
CCRMA, Stanford University
AES-2006 Heyser Lecture
October 6, 2006
Julius Smith
AES-2006 Heyser Lecture 1 / 84
Overview
Early Digital Synthesis
Spectral Modeling
Physical Modeling
Summary
Overview
Julius Smith
Outline
Overview
Outline
CCRMA Perspective
Spectral Modeling
Physical Modeling
Summary
Digital sound synthesis approaches in approximate historical order:
Wavetable (one period)

Subtractive
Additive
Frequency Modulation (FM)
Sampling
Spectral Modeling
Physical Modeling
Some connections with audio coding will be noted
Emphasis:
Sound examples
Block diagrams
Historical notes
Julius Smith
CCRMA Perspective
Overview
Outline
CCRMA Perspective
Spectral Modeling
Physical Modeling
Summary
The Knoll, Stanford University
Julius Smith
Overview
Spectral Modeling
Physical Modeling
Summary
Early Digital Sound Synthesis
Julius Smith
Wavetable Synthesis in Music I-V (1957-1969)

Overview
Music V
KL Music
Daisy
Additive Analysis
Additive Synthesis
FM Synthesis
FM Formula
FM Patch
FM Spectra
FM Examples
FM Voice
Sampling Synthesis
Modern Example
Spectral Modeling
Physical Modeling
Summary
Julius Smith
Music V Scripting Language (Note Cards)

Overview
Music V
KL Music
Daisy
Additive Analysis
Additive Synthesis
FM Synthesis
FM Formula
FM Patch
FM Spectra
FM Examples
FM Voice
Sampling Synthesis
Modern Example
Spectral Modeling
Physical Modeling
Summary
Essentially Supported in MPEG-4 Structured Audio Orchestra
Julius Smith
Language (SAOL) (Music V csound SAOL)

Encoding sounds as instruments is hard, in general
Kelly-Lochbaum Vocal Tract Model

Overview
Music V
KL Music
Daisy
Additive Analysis
Additive Synthesis
FM Synthesis
FM Formula
FM Patch
FM Spectra
FM Examples
FM Voice
Sampling Synthesis
Modern Example
Spectral Modeling
Physical Modeling
e(n)
Glottal Pulse
Train or Noise
e(n)
1 + k1
z 1 2
k1
(Unused
Allpass
Output)
k1
R1
z 1 2
1 k1
y(n)
1 + kM
z 1 2
kM
kM
Speech
Output
y(n)
RM
z 1 2
1 kM
Kelly-Lochbaum Vocal Tract Model (Piecewise Cylindrical)
Summary
John L. Kelly and Carol Lochbaum (1962)
Julius Smith
Sound Example
Overview
Bicycle Built for Two: (WAV) (MP3)
Music V
KL Music
Daisy
Additive Analysis
Additive Synthesis
FM Synthesis
FM Formula
FM Patch
FM Spectra
FM Examples
FM Voice
Sampling Synthesis
Modern Example
Spectral Modeling
Vocal part by Kelly and Lochbaum (1961)

Musical accompaniment by Max Mathews
Computed on an IBM 704
Based on Russian speech-vowel data from Gunnar Fants book
Probably the first digital physical-modeling synthesis sound
example by any method
Inspired Arthur C. Clarke to adapt it for 2001: A Space Odyssey
the computers first song
Physical Modeling
Summary
Julius Smith
Classic Additive-Synthesis Analysis (Heterodyne Comb)

Overview
Music V
KL Music
Daisy
Additive Analysis
Additive Synthesis
FM Synthesis
FM Formula
FM Patch
FM Spectra
FM Examples
FM Voice
Sampling Synthesis
Modern Example
Spectral Modeling
Physical Modeling
Summary
John Grey 1975 CCRMA Tech. Reports 1 & 2

(CCRMA STANM reports available online)
Julius Smith
Classic Additive-Synthesis (Sinusoidal Oscillator Envelopes)

Overview
Music V
KL Music
Daisy
Additive Analysis
Additive Synthesis
FM Synthesis
FM Formula
FM Patch
FM Spectra
FM Examples
FM Voice
Sampling Synthesis
Modern Example
Spectral Modeling
Physical Modeling
Summary
John Grey 1975 CCRMA Tech. Reports 1 & 2

(CCRMA STANM reports available online)
Julius Smith
Classic Additive Synthesis Diagram

Overview
A1 (t) f1 (t) A2 (t) f2 (t) A3 (t) f3 (t) A4 (t) f4 (t)
Music V
KL Music
Daisy
Additive Analysis
Additive Synthesis
FM Synthesis
FM Formula
FM Patch
FM Spectra
FM Examples
FM Voice
Sampling Synthesis
Modern Example
Spectral Modeling
noise
FIR
Physical Modeling
Summary
y(t) =
4
X
i=1
Julius Smith
Ai (t) sin
Z

i (t)dt + i (0)
Classic Additive-Synthesis Examples

Overview
Music V
KL Music
Daisy
Additive Analysis
Additive Synthesis
FM Synthesis
FM Formula
FM Patch
FM Spectra
FM Examples
FM Voice
Sampling Synthesis
Modern Example
Spectral Modeling
Physical Modeling
Bb Clarinet
Eb Clarinet
Oboe
Bassoon
Tenor Saxophone
Trumpet
English Horn
French Horn
Flute
All of the above

Independently synthesized set
Summary
(Synthesized from original John Grey data)
Julius Smith
Frequency Modulation (FM) Synthesis

Overview
FM synthesis is normally used as a spectral modeling technique.
Music V
KL Music
Daisy
Additive Analysis
Additive Synthesis
FM Synthesis
FM Formula
FM Patch
FM Spectra
FM Examples
FM Voice
Sampling Synthesis
Modern Example
Spectral Modeling
Physical Modeling
Summary
Julius Smith
Discovered and developed (1970s) by John M. Chowning
(CCRMA Founding Director)

Key paper: JAES 1973 (vol. 21, no. 7)
Commercialized by Yamaha Corporation:
DX-7 synthesizer (1983)

OPL chipset (SoundBlaster PC sound card)
Cell phone ring tones
On the physical modeling front, synthesis of vibrating-string
waveforms using finite differences started around this time:
Hiller & Ruiz, JAES 1971 (vol. 19, no. 6)
FM Formula
Overview
x(t) = Ac sin[c t + c + Am sin(m t + m )]
Music V
KL Music
Daisy
Additive Analysis
Additive Synthesis
FM Synthesis
FM Formula
FM Patch
FM Spectra
FM Examples
FM Voice
Sampling Synthesis
Modern Example
where
(Ac , c , c ) specify the carrier sinusoid

(Am , m , m ) specify the modulator sinusoid
Can also be called phase modulation
Spectral Modeling
Physical Modeling
Summary
Julius Smith
Simple FM Brass Patch (1970)

Overview
Music V
KL Music
Daisy
Additive Analysis
Additive Synthesis
FM Synthesis
FM Formula
FM Patch
FM Spectra
FM Examples
FM Voice
Sampling Synthesis
Modern Example
Jean-Claude Risset observation (19641969):

Brass bandwidth amplitude
fm = f0
g
A
fc = f0
Spectral Modeling
Physical Modeling
Summary
Out
Julius Smith
FM Harmonic Amplitudes (Bessel Function of First Kind)

Overview
Harmonic number k , FM index :
Spectral Modeling
0.5
J ()
Music V
KL Music
Daisy
Additive Analysis
Additive Synthesis
FM Synthesis
FM Formula
FM Patch
FM Spectra
FM Examples
FM Voice
Sampling Synthesis
Modern Example
0
0
Physical Modeling
Summary
2
0.5
0
4
5
10
15
20
25
30
10
Order k
Argument
Julius Smith
Frequency Modulation (FM) Examples

Overview
All examples by John Chowning unless otherwise noted:
Music V
KL Music
Daisy
Additive Analysis
Additive Synthesis
FM Synthesis
FM Formula
FM Patch
FM Spectra
FM Examples
FM Voice
Sampling Synthesis
Modern Example
Spectral Modeling
Physical Modeling
Summary
FM brass synthesis
Low Brass example
Dexter Morrils FM Trumpet
FM singing voice (1978)
Each formant synthesized using an FM operator pair
(two sinusoidal oscillators)
Chorus
Voices
Basso Profundo
Other early FM synthesis
Clicks and Drums
Big Bell
String Canon
Julius Smith
FM Voice
Music V
KL Music
Daisy
Additive Analysis
Additive Synthesis
FM Synthesis
FM Formula
FM Patch
FM Spectra
FM Examples
FM Voice
Sampling Synthesis
Modern Example
FM voice synthesis can be viewed as compressed modeling of

spectral formants
Carrier 2
Carrier 1
Carrier 3
Magnitude
Overview
Spectral Modeling
Physical Modeling
Summary
Frequency
Modulation Frequency (all three)
Julius Smith
Sampling Synthesis History

Overview
1979 - Fairlight Computer Music Instrument - 8-bit
Music V
KL Music
Daisy
Additive Analysis
Additive Synthesis
FM Synthesis
FM Formula
FM Patch
FM Spectra
FM Examples
FM Voice
Sampling Synthesis
Modern Example
Spectral Modeling
First commercial sampler

Eight voices, 8 bits, 64 KB (4 sec) RAM, 16 kHz (mono)
Editing, looping, mixing
One could draw waveforms and additive-synthesis amplitude
envelopes (for each harmonic) with a light pen
$25,000$36,000!
1981 - E-mu Systems Emulator

First affordable sampler ($10,000)
Eight voices, 128 K RAM, 8-bit, 80 lb.
Physical Modeling
Summary
1986 - Ensoniq Mirage

Breakthrough price-point ($1695)
Eight voices, 144 K RAM, 8-bit
Julius Smith
Modern Sampled Piano

Overview
Example:1
Music V
KL Music
Daisy
Additive Analysis
Additive Synthesis
FM Synthesis
FM Formula
FM Patch
FM Spectra
FM Examples
FM Voice
Sampling Synthesis
Modern Example
40 Gigabytes on ten DVDs (three sampled pianos)

Every key sampled
410 velocity layers
Separate recordings with soft pedal down
Separate release recordings, for multiple striking velocities
Spectral Modeling
Physical Modeling
Summary
Julius Smith
Synthogy Ivory, $349 (Electronic Musician, October 2006)

Fundamental Problem with Sampling Synthesis

Overview
Piano timbre is determined by
Music V
KL Music
Daisy
Additive Analysis
Additive Synthesis
FM Synthesis
FM Formula
FM Patch
FM Spectra
FM Examples
FM Voice
Sampling Synthesis
Modern Example
key number (1 byte)

key velocity (2 bytes more than enough)
pedal state (1 bit [or byte] per pedal)
Piano control is relatively low-dimensional:
Less than six bytes of information per note played

No continuous controls (typically)
Ratio of total sampled data to one note of control data
one billion
Spectral Modeling
Physical Modeling
Summary
Julius Smith
Now consider bowed strings

Overview
Control parameters:
Music V
KL Music
Daisy
Additive Analysis
Additive Synthesis
FM Synthesis
FM Formula
FM Patch
FM Spectra
FM Examples
FM Voice
Sampling Synthesis
Modern Example
Left-hand finger position(s)

Left-hand vibrato
Bow velocity
Bow force
Bow position
Bow angle
Shoulder damping
Instrument orientation
Player motion (within a room)
Spectral Modeling
Physical Modeling
Summary
Julius Smith
Difficulty of sampling bowed strings

Overview
Music V
KL Music
Daisy
Additive Analysis
Additive Synthesis
FM Synthesis
FM Formula
FM Patch
FM Spectra
FM Examples
FM Voice
Sampling Synthesis
Modern Example
Bowed-string control is infinite-dimensional in principle

Many time-varying functions gestures
(we counted more than 10)

Complete sampling of bowed strings on the level of pianos has
apparently never been done
Rule-driven navigation of the most useful recorded playing
regimes has worked well (e.g., Synful Orchestra)
Model-based approaches greatly reduce data requirements:
Spectral models (inspired by sound perception)

Physical models (model the sound source)
Spectral Modeling
Physical Modeling
Summary
Julius Smith
Overview
Spectral Modeling
Physical Modeling
Summary
Spectral Modeling Synthesis

(Historical Summary)
Julius Smith
Classic Vocoder Analysis & Resynthesis (Dudley 1939)

magnitude, or
magnitude and
phase extraction
Overview
Spectral Modeling
Vocoder
Vocoder Examples
Sinusoidal Modeling
Spectral Trajectories
Sines + Noise
S+N Examples
S+N FX
S+N XSynth
Sines + Transients
S + N + Transients
S+N+T TSM
S+N+T Freq Map
S+N+T Windows
HF Noise Modeling
HF Noise Band
S+N+T Examples
SM Summary
Spec Future
Physical Modeling
x0 (t)
x
0 (t)
x(t)
x1 (t)
f
Data Compression,
Transmission,
Storage,
Manipulation,
Noise reduction, ...
xN 1
x
1 (t)
x
(t)
x
N 1
Analysis
Processing
Synthesis
Summary
Julius Smith
Phase Vocoder Channel Model

Analysis Model
Overview
Synthesis Model
Vocoder
Vocoder Examples
Sinusoidal Modeling
Sines + Noise
S+N Examples
S+N FX
S+N XSynth
Sines + Transients
S + N + Transients
S+N+T TSM
S+N+T Freq Map
S+N+T Windows
HF Noise Modeling
HF Noise Band
S+N+T Examples
SM Summary
Spec Future
Physical Modeling
Summary
Julius Smith
ak
Channel Filter
Response
Spectral Modeling
ak(t)
A
F
Sine Osc
k
0
k+k(t)
Out
Early channel vocoder implementations (hardware) only
measured amplitude ak (t) (Dudley 1939)

The phase vocoder (Flanagan and Golden 1966) added phase
tracking in each channel
Portnoff (1976) developed the FFT phase vocoder,
which replaced the heterodyne comb in computer-music
additive-synthesis analysis (James A. Moorer)
Inverse FFT synthesis (Rodet and Depalle 1992) gave faster
sinusoidal oscillator banks
Amplitude and Frequency Envelopes

Overview
ak (t)

Spectral Modeling
Vocoder
Vocoder Examples
Sinusoidal Modeling
Sines + Noise
S+N Examples
S+N FX
S+N XSynth
Sines + Transients
S + N + Transients
S+N+T TSM
S+N+T Freq Map
S+N+T Windows
HF Noise Modeling
HF Noise Band
S+N+T Examples
SM Summary
Spec Future
k (t) = k (t)
Physical Modeling
Summary
Julius Smith
Channel Vocoder Sound Examples

Overview
Original

Spectral Modeling
Vocoder
Vocoder Examples
Sinusoidal Modeling
Sines + Noise
S+N Examples
S+N FX
S+N XSynth
Sines + Transients
S + N + Transients
S+N+T TSM
S+N+T Freq Map
S+N+T Windows
HF Noise Modeling
HF Noise Band
S+N+T Examples
SM Summary
Spec Future
10 channels, sine carriers

10 channels, narrowband-noise carriers
26 channels, sine carriers
26 channels, narrowband-noise carriers
26 channels, narrowband-noise carriers, channels reversed
Phase Vocoder: Identity system in absence of modifications
The FFT Phase Vocoder next transitioned to the Short-Time
Fourier Transform (STFT) (Allen and Rabiner 1977)
Physical Modeling
Summary
Julius Smith
Tracking Spectral Peaks in the Short-Time Fourier Transform

Overview
atan

Spectral Modeling
Vocoder
Vocoder Examples
Sinusoidal Modeling
Sines + Noise
S+N Examples
S+N FX
S+N XSynth
Sines + Transients
S + N + Transients
S+N+T TSM
S+N+T Freq Map
S+N+T Windows
HF Noise Modeling
HF Noise Band
S+N+T Examples
SM Summary
Spec Future
Phases
s(t)
FFT
Frequencies
dB mag
Peak
tracking
Quadratic
Peak
Interpolation
window w(n)
Amplitudes
STFT peak tracking at CCRMA: mid-1980s (PARSHL program)

Motivated by vocoder analysis of piano tones
Influences: STFT (Allen and Rabiner 1977),
ADEC (1977), MAPLE (1979)

Independently developed for speech coding by McAulay and
Quatieri at Lincoln Labs
Physical Modeling
Summary
Julius Smith
Example Spectral Trajectories

Overview

Spectral Modeling
Vocoder
Vocoder Examples
Sinusoidal Modeling
Sines + Noise
S+N Examples
S+N FX
S+N XSynth
Sines + Transients
S + N + Transients
S+N+T TSM
S+N+T Freq Map
S+N+T Windows
HF Noise Modeling
HF Noise Band
S+N+T Examples
SM Summary
Spec Future
Physical Modeling
Summary
Julius Smith
Sines + Noise Synthesis (1989)

PSfrag
Overview
A1 (t) f1 (t) A2 (t) f2 (t) A3 (t) f3 (t) A4 (t) f4 (t)

Spectral Modeling
Vocoder
Vocoder Examples
Sinusoidal Modeling
Sines + Noise
S+N Examples
S+N FX
S+N XSynth
Sines + Transients
S + N + Transients
S+N+T TSM
S+N+T Freq Map
S+N+T Windows
HF Noise Modeling
HF Noise Band
S+N+T Examples
SM Summary
Spec Future
Physical Modeling
Summary
Julius Smith
white noise
u(t)
y(t) =
filter(t)
ht ( )
4
P
i=1
Ai (t) cos
hR
t
i
0 i (t)dt + i (0) + (ht u)(t)
Sines + Noise Sound Examples

Overview
Xavier Serra 1989 thesis demos (Sines + Noise signal modeling)

Spectral Modeling
Vocoder
Vocoder Examples
Sinusoidal Modeling
Sines + Noise
S+N Examples
S+N FX
S+N XSynth
Sines + Transients
S + N + Transients
S+N+T TSM
S+N+T Freq Map
S+N+T Windows
HF Noise Modeling
HF Noise Band
S+N+T Examples
SM Summary
Spec Future
Piano
Original
Sinusoids alone
Residual after sinusoids removed
Sines + noise model
Voice
Original
Sinusoids
Residual
Synthesis
Physical Modeling
Summary
Julius Smith
Musical Effects with Sines+Noise Models (Serra 1989)

Overview
Piano Effects

Spectral Modeling
Vocoder
Vocoder Examples
Sinusoidal Modeling
Sines + Noise
S+N Examples
S+N FX
S+N XSynth
Sines + Transients
S + N + Transients
S+N+T TSM
S+N+T Freq Map
S+N+T Windows
HF Noise Modeling
HF Noise Band
S+N+T Examples
SM Summary
Spec Future
Pitch downshift one octave

Pitch flattened
Varying partial stretching
Voice Effects
Frequency-scale by 0.6
Frequency-scale by 0.4 and stretch partials
Variable time-scaling, deterministic to stochastic
Physical Modeling
Summary
Julius Smith
Cross-Synthesis with Sines+Noise Models (Serra 1989)

Overview
Voice modulator

Spectral Modeling
Vocoder
Vocoder Examples
Sinusoidal Modeling
Sines + Noise
S+N Examples
S+N FX
S+N XSynth
Sines + Transients
S + N + Transients
S+N+T TSM
S+N+T Freq Map
S+N+T Windows
HF Noise Modeling
HF Noise Band
S+N+T Examples
SM Summary
Spec Future
Creaking ships mast carrier

Voice-modulated creaking mast
Same with modified spectral envelopes
Physical Modeling
Summary
Julius Smith
Sines + Transients Sound Examples (Serra 1989)

Overview
In this technique, the sinusoidal sum is phase-matched at the

cross-over point only (with no cross-fade).
Spectral Modeling
Vocoder
Vocoder Examples
Sinusoidal Modeling
Sines + Noise
S+N Examples
S+N FX
S+N XSynth
Sines + Transients
S + N + Transients
S+N+T TSM
S+N+T Freq Map
S+N+T Windows
HF Noise Modeling
HF Noise Band
S+N+T Examples
SM Summary
Spec Future
Marimba
Original
Sinusoidal model
Original attack, followed by sinusoidal model
Piano
Original
Sinusoidal model
Original attack, followed by sinusoidal model
Physical Modeling
Summary
Julius Smith
Multiresolution Sines + Noise + Transients (Levine 1998)

Overview
Why Model Transients Separately?

Spectral Modeling
Vocoder
Vocoder Examples
Sinusoidal Modeling
Sines + Noise
S+N Examples
S+N FX
S+N XSynth
Sines + Transients
S + N + Transients
S+N+T TSM
S+N+T Freq Map
S+N+T Windows
HF Noise Modeling
HF Noise Band
S+N+T Examples
SM Summary
Spec Future
Sinusoids efficiently model spectral peaks over time

Filtered noise efficiently models spectral residual vs. t
Neither is good for abrupt transients in the waveform
Phase-matched oscillators are expensive
More efficient to switch to a transient model during transients
Need sinusoidal phase matching at the switching times
Transient models:
Original waveform slice (1988)

Wavelet expansion (Ali 1996)
MPEG-2 AAC (with short window) (Levine 1998)
Frequency-domain LPC
(time-domain amplitude envelope) (Verma 2000)
Physical Modeling
Summary
Julius Smith
Time Scale Modification of Sines + Noise + Transients Models
original signal
Spectral Modeling
sines +
noise
sines +
noise
sines +
noise
transients
Physical Modeling
time-scaled
signal
sines +
noise
transients
Vocoder
Vocoder Examples
Sinusoidal Modeling
Sines + Noise
S+N Examples
S+N FX
S+N XSynth
Sines + Transients
S + N + Transients
S+N+T TSM
S+N+T Freq Map
S+N+T Windows
HF Noise Modeling
HF Noise Band
S+N+T Examples
SM Summary
Spec Future
sines +
noise
transients
transients
Overview
sines +
noise
time
Time-Scale Modification (TSM) becomes well defined:
Transients are translated in time

Sinusoidal envelopes are scaled in time
Noise-filter envelopes also scaled in time
Dual of TSM is frequency scaling
Summary
Julius Smith
Sines + Noise + Transients Time-Frequency Map

16
Overview
14
Physical Modeling
frequency [kHz]
Vocoder
Vocoder Examples
Sinusoidal Modeling
Sines + Noise
S+N Examples
S+N FX
S+N XSynth
Sines + Transients
S + N + Transients
S+N+T TSM
S+N+T Freq Map
S+N+T Windows
HF Noise Modeling
HF Noise Band
S+N+T Examples
SM Summary
Spec Future
12
10
8
6
4
2
0
50
100
150
200
250
50
100
150
200
250
amplitude
Spectral Modeling
0.5
0
0.5
1
time [milliseconds]
(Levine 1998)
Summary
Julius Smith
Overview
transient
Corresponding Analysis Windows
low octave
amplitude
Vocoder
Vocoder Examples
Sinusoidal Modeling
Sines + Noise
S+N Examples
S+N FX
S+N XSynth
Sines + Transients
S + N + Transients
S+N+T TSM
S+N+T Freq Map
S+N+T Windows
HF Noise Modeling
HF Noise Band
S+N+T Examples
SM Summary
Spec Future
middle octave high octave
Spectral Modeling
Physical Modeling
0
Summary
Julius Smith
50
100
150
200
250
time [milliseconds]
Quasi-Constant-Q (Wavelet) Time-Frequency Map

16
Overview
14
frequency [kHz]
Vocoder
Vocoder Examples
Sinusoidal Modeling
Sines + Noise
S+N Examples
S+N FX
S+N XSynth
Sines + Transients
S + N + Transients
S+N+T TSM
S+N+T Freq Map
S+N+T Windows
HF Noise Modeling
HF Noise Band
S+N+T Examples
SM Summary
Spec Future
12
10
8
6
4
2
0
4
0
x 10
50
100
150
200
250
50
100
150
200
250
amplitude
Spectral Modeling
1
0
1
2
time [milliseconds]
Physical Modeling
Summary
Julius Smith
Bark-Band Noise Modeling at High Frequencies (Levine 1998)

Overview
85
80
Spectral Modeling
Physical Modeling
Summary
Julius Smith
75
70
65
magnitude [dB]
Vocoder
Vocoder Examples
Sinusoidal Modeling
Sines + Noise
S+N Examples
S+N FX
S+N XSynth
Sines + Transients
S + N + Transients
S+N+T TSM
S+N+T Freq Map
S+N+T Windows
HF Noise Modeling
HF Noise Band
S+N+T Examples
SM Summary
Spec Future
60
55
50
45
40
35
30
500
1000
1500
2000
2500
3000
3500
4000
4500
frequency [Hz]
Amplitude Envelope for One Noise Band

Overview
Vocoder
Vocoder Examples
Sinusoidal Modeling
Sines + Noise
S+N Examples
S+N FX
S+N XSynth
Sines + Transients
S + N + Transients
S+N+T TSM
S+N+T Freq Map
S+N+T Windows
HF Noise Modeling
HF Noise Band
S+N+T Examples
SM Summary
Spec Future
Original Mag. [dB]
Spectral Modeling
70
60
50
40
50
100
150
200
250
300
350
50
100
150
200
250
300
350
75
70
65
60
55
50
45
Physical Modeling
80
LSA Mag. [dB]
80
time [milliseconds]
Summary
Julius Smith
For more information, see Scott Levines thesis.2

2
http://ccrma.stanford.edu/~scottl/thesis.html
Sines + Noise + Transients Sound Examples

Overview
Scott Levine Thesis Demos (Sines + Noise + Transients at 32 kbps)

(http://ccrma.stanford.edu/~scottl/thesis.html)
Spectral Modeling
Vocoder
Vocoder Examples
Sinusoidal Modeling
Sines + Noise
S+N Examples
S+N FX
S+N XSynth
Sines + Transients
S + N + Transients
S+N+T TSM
S+N+T Freq Map
S+N+T Windows
HF Noise Modeling
HF Noise Band
S+N+T Examples
SM Summary
Spec Future
Mozarts Le Nozze di Figaro
Original
Compressed using MPEG-AAC at 32 kbps
Compressed using sines+transients+noise at 32 kbps
Multiresolution sinusoids alone

Residual Bark-band noise
Transform-coded transients (AAC)
Bark-band noise above 5 kHz
Physical Modeling
Summary
Julius Smith
Rock Example
Overview

Spectral Modeling
Vocoder
Vocoder Examples
Sinusoidal Modeling
Sines + Noise
S+N Examples
S+N FX
S+N XSynth
Sines + Transients
S + N + Transients
S+N+T TSM
S+N+T Freq Map
S+N+T Windows
HF Noise Modeling
HF Noise Band
S+N+T Examples
SM Summary
Spec Future
It Takes Two by Rob Base & DJ E-Z Rock
Original
MPEG-AAC at 32 kbps
Sines+transients+noise at 32 kbps
Multiresolution sinusoids
Residual Bark-band noise
Transform-coded transients (AAC)
Bark-band noise above 5 kHz
Physical Modeling
Summary
Julius Smith
Time Scale Modification using Sines + Noise + Transients

Overview

Spectral Modeling
Vocoder
Vocoder Examples
Sinusoidal Modeling
Sines + Noise
S+N Examples
S+N FX
S+N XSynth
Sines + Transients
S + N + Transients
S+N+T TSM
S+N+T Freq Map
S+N+T Windows
HF Noise Modeling
HF Noise Band
S+N+T Examples
SM Summary
Spec Future
Time-Scale Modification (pitch unchanged)
S+N+T time-scale factors [2.0, 1.6, 1.2, 1.0, 0.8, 0.6, 0.5]
S+N+T Pitch Shifting (timing unchanged)
Pitch-scale factors [0.89, 0.94, 1.00, 1.06, 1.12]
Physical Modeling
Summary
Julius Smith
Spectral Modeling History Highlights
Fouriers theory (1822)

Teleharmonium (1906)
Hammond organ (1930s)
Channel Vocoder (1939)
Phase Vocoder (1966)
Additive Synthesis (1969)
FFT Phase Vocoder (1976)
Sinusoidal Modeling
(1977,1979,1985)
Sines+Noise (1989)
Sines+Transients (1989)
Sines+Noise+Transients (1998)
Julius Smith
Perceptual audio coding:
Princen-Bradley filterbank (1986)

K. Brandenburg thesis (1989)
Auditory masking usage
Dolby AC2
Musicam
ASPEC
MPEG-I,II,IV
(incl. S+N+T parametric sounds)
Future Prospects
Overview
Observations:

Spectral Modeling
Vocoder
Vocoder Examples
Sinusoidal Modeling
Sines + Noise
S+N Examples
S+N FX
S+N XSynth
Sines + Transients
S + N + Transients
S+N+T TSM
S+N+T Freq Map
S+N+T Windows
HF Noise Modeling
HF Noise Band
S+N+T Examples
SM Summary
Spec Future
Sinusoidal modeling of sound is Unreasonably Effective

Basic auditory masking discards 90% information
Interesting neuroscience observation:
... most neurons in the primary auditory cortex A1
are silent most of the time ...
(from Sparse Time-Frequency Representations, Gardner and

Magnesco, PNAS:103(16), April 2006)
What is a true and correct psychospectral model for sound?
The cochlea of the ear is a real-time spectrum analyzer

How is the ears spectrogram represented at higher levels?
Physical Modeling
Summary
Julius Smith
Overview
Spectral Modeling
Physical Modeling
Summary
Physical Modeling Synthesis

(Historical Summary)
Julius Smith
Kelly-Lochbaum Vocal Tract Model

Overview

Spectral Modeling
e(n)
y(n)
Physical Modeling
KL Music
Digital Waveguide
Signal Scattering
Plucked String
Struck String
Karplus Strong
EKS Algorithm
Clarinet
Wind Examples
Bowed Strings
Distortion Guitar
Acoustic Strings
Sound Examples
Linearized Violin
Commuted Piano
Pulse Synthesis
Complete Piano
Sound Examples
Phy Audio Coding
Phy Audio Coding?
Glottal Pulse
Train or Noise
e(n)
1 + k1
z 1 2
k1
(Unused
Allpass
Output)
k1
R1
z 1 2
1 k1
1 + kM
z 1 2
kM
kM
Speech
Output
y(n)
RM
z 1 2
1 kM
Kelly-Lochbaum Vocal Tract Model (Piecewise Cylindrical)
John L. Kelly and Carol Lochbaum (1962)
Summary
Julius Smith
Digital Waveguide Models (1985)

Overview
Lossless digital waveguide
= bidirectional delay line
at some wave impedance R
Spectral Modeling
Physical Modeling
KL Music
Digital Waveguide
Signal Scattering
Plucked String
Struck String
Karplus Strong
EKS Algorithm
Clarinet
Wind Examples
Bowed Strings
Distortion Guitar
Acoustic Strings
Sound Examples
Linearized Violin
Commuted Piano
Pulse Synthesis
Complete Piano
Sound Examples
Phy Audio Coding
Phy Audio Coding?
zN
R
zN
Useful for efficient models of
strings
bores
plane waves
conical waves
Summary
Julius Smith
Signal Scattering
Overview
Signal scattering is caused by a change in wave impedance R:
R R1
k1 = 2
R2 + R1
Spectral Modeling
Physical Modeling
KL Music
Digital Waveguide
Signal Scattering
Plucked String
Struck String
Karplus Strong
EKS Algorithm
Clarinet
Wind Examples
Bowed Strings
Distortion Guitar
Acoustic Strings
Sound Examples
Linearized Violin
Commuted Piano
Pulse Synthesis
Complete Piano
Sound Examples
Phy Audio Coding
Phy Audio Coding?
1 + k1
zN
R1
zN
k1
k1
zN
R2
zN
1 k1
If the wave impedance changes every spatial sample, the

Kelly-Lochbaum vocal-tract model results.
Summary
Julius Smith
Ideal Plucked String (Displacement Waves)

Overview
Spectral Modeling
y (n)
y (n-N/2)
Physical Modeling
KL Music
Digital Waveguide
Signal Scattering
Plucked String
Struck String
Karplus Strong
EKS Algorithm
Clarinet
Wind Examples
Bowed Strings
Distortion Guitar
Acoustic Strings
Sound Examples
Linearized Violin
Commuted Piano
Pulse Synthesis
Complete Piano
Sound Examples
Phy Audio Coding
Phy Audio Coding?
Bridge
-1
y-(n)
(x = 0)
(x = Pluck Position)
-1
Nut
y-(n+N/2)
(x = L)
Load each delay line with half of initial string displacement

Sum of upper and lower delay lines = string displacement
Summary
Julius Smith
Ideal Struck String (Velocity Waves)

Overview
Spectral Modeling
v+(n)
v+(n-N/2)
Physical Modeling
KL Music
Digital Waveguide
Signal Scattering
Plucked String
Struck String
Karplus Strong
EKS Algorithm
Clarinet
Wind Examples
Bowed Strings
Distortion Guitar
Acoustic Strings
Sound Examples
Linearized Violin
Commuted Piano
Pulse Synthesis
Complete Piano
Sound Examples
Phy Audio Coding
Phy Audio Coding?
Bridge
-1
v-(n)
(x = 0)
(x = Hammer Position)
-1
Nut
v-(n+N/2)
(x = L)
Hammer strike = momentum transfer = velocity step:
mh vh (0) = (mh + ms )vs (0+)
Summary
Julius Smith
Karplus-Strong (KS) Algorithm (1983)

Overview
Spectral Modeling
Output y (n)
y (n-N)
N samples delay
Physical Modeling
KL Music
Digital Waveguide
Signal Scattering
Plucked String
Struck String
Karplus Strong
EKS Algorithm
Clarinet
Wind Examples
Bowed Strings
Distortion Guitar
Acoustic Strings
Sound Examples
Linearized Violin
Commuted Piano
Pulse Synthesis
Complete Piano
Sound Examples
Phy Audio Coding
Phy Audio Coding?
1/2
1/2
z -1
Discovered (1978) as self-modifying wavetable synthesis

Wavetable is preferably initialized with random numbers
Summary
Julius Smith
Karplus-Strong Sound Examples

Overview
Vintage 8-bit sound examples:

Spectral Modeling
Physical Modeling
KL Music
Digital Waveguide
Signal Scattering
Plucked String
Struck String
Karplus Strong
EKS Algorithm
Clarinet
Wind Examples
Bowed Strings
Distortion Guitar
Acoustic Strings
Sound Examples
Linearized Violin
Commuted Piano
Pulse Synthesis
Complete Piano
Sound Examples
Phy Audio Coding
Phy Audio Coding?
Original Plucked String: (WAV) (MP3)

Drum: (WAV) (MP3)
Stretched Drum: (WAV) (MP3)
Summary
Julius Smith
EKS Algorithm (Jaffe-Smith 1983)

Overview
Hp (z)
z N
H (z)
HL (z)
Spectral Modeling
H (z)
Physical Modeling
KL Music
Digital Waveguide
Signal Scattering
Plucked String
Struck String
Karplus Strong
EKS Algorithm
Clarinet
Wind Examples
Bowed Strings
Distortion Guitar
Acoustic Strings
Sound Examples
Linearized Violin
Commuted Piano
Pulse Synthesis
Complete Piano
Sound Examples
Phy Audio Coding
Phy Audio Coding?
Hs (z)
Hd (z)
= pitch period (2 string length) in samples

1p
= pick-direction lowpass filter
Hp (z) =
1
1 pz
N
H (z) = 1 z N = pick-position comb filter, (0, 1)

Hd (z) = string-damping filter (one/two poles/zeros typical)
Hs (z) = string-stiffness allpass filter (several poles and zeros)
H (z) =
HL (z) =
(N ) z 1
= first-order string-tuning allpass filter
1 (N ) z 1
1 RL
= dynamic-level lowpass filter
1 RL z 1
Summary
Julius Smith
STK EKS Sound Examples

Overview
Spectral Modeling
Synthesis Tool Kit (STK) by Perry Cook, Gary Scavone, and

others distributed by CCRMA:
Google search: STK ToolKit
Physical Modeling
KL Music
Digital Waveguide
Signal Scattering
Plucked String
Struck String
Karplus Strong
EKS Algorithm
Clarinet
Wind Examples
Bowed Strings
Distortion Guitar
Acoustic Strings
Sound Examples
Linearized Violin
Commuted Piano
Pulse Synthesis
Complete Piano
Sound Examples
Phy Audio Coding
Phy Audio Coding?
STK Plucked String: (WAV) (MP3)
Plucked String 1: (WAV) (MP3)

Summary
Julius Smith
EKS Sound Example (1988)

Overview
Bach A-Minor ConcertoOrchestra Part: (WAV) (MP3)

Spectral Modeling
Physical Modeling
KL Music
Digital Waveguide
Signal Scattering
Plucked String
Struck String
Karplus Strong
EKS Algorithm
Clarinet
Wind Examples
Bowed Strings
Distortion Guitar
Acoustic Strings
Sound Examples
Linearized Violin
Commuted Piano
Pulse Synthesis
Complete Piano
Sound Examples
Phy Audio Coding
Phy Audio Coding?
Executed in real time on one Motorola DSP56001

(20 MHz clock, 128K SRAM)
Developed for the NeXT Computer introduction at Davies

Symphony Hall, San Francisco, 1988
Solo violin part was played live by Dan Kobialka of the San
Francisco Symphony
Summary
Julius Smith
Digital Waveguide Single Reed, Cylindrical Bore Model (1986)

Overview
Spectral Modeling
Physical Modeling
KL Music
Digital Waveguide
Signal Scattering
Plucked String
Struck String
Karplus Strong
EKS Algorithm
Clarinet
Wind Examples
Bowed Strings
Distortion Guitar
Acoustic Strings
Sound Examples
Linearized Violin
Commuted Piano
Pulse Synthesis
Complete Piano
Sound Examples
Phy Audio Coding
Phy Audio Coding?
Mouth
Pressure
pm ( n )
2
pb ( n )
hm
Reed Table
Reflection
Filter
h+
Embouchure
Offset
Output
Filter
Reed to Bell Delay
Bell to Reed Delay
p+b ( n )
Reed
Bore
Bell
Digital waveguide clarinet
Control variable = mouth half-pressure

Total reed cost = two subtractions, one multiply, and one table
lookup per sample
Summary
Julius Smith
Digital Waveguide Wind Instrument Sound Examples

Overview
STK Clarinet: (WAV) (MP3)
Google search: STK clarinet
Spectral Modeling
Synthesis Tool Kit (STK) by Perry Cook, Gary Scavone, and
Physical Modeling
KL Music
Digital Waveguide
Signal Scattering
Plucked String
Struck String
Karplus Strong
EKS Algorithm
Clarinet
Wind Examples
Bowed Strings
Distortion Guitar
Acoustic Strings
Sound Examples
Linearized Violin
Commuted Piano
Pulse Synthesis
Complete Piano
Sound Examples
Phy Audio Coding
Phy Audio Coding?
others distributed by CCRMA:

Google search: STK ToolKit
Staccato Systems Slide Flute
(based on STK flute, ca. 1995): (WAV) (MP3)

Yamaha VL1 Virtual Lead synthesizer demos (1994):
Shakuhachi: (WAV) (MP3)

Oboe and Bassoon: (WAV) (MP3)
Tenor Saxophone: (WAV) (MP3)
Summary
Julius Smith
Digital Waveguide Bowed Strings (1986)

Bow Force
Bow Velocity
Overview
Nut to Bow Delay

vb
Physical Modeling
-1
Nut
v+
Bow to Nut Delay
String
Body
Filter
Bow to Bridge Delay
Spectral Modeling
KL Music
Digital Waveguide
Signal Scattering
Plucked String
Struck String
Karplus Strong
EKS Algorithm
Clarinet
Wind Examples
Bowed Strings
Distortion Guitar
Acoustic Strings
Sound Examples
Linearized Violin
Commuted Piano
Pulse Synthesis
Complete Piano
Sound Examples
Phy Audio Coding
Phy Audio Coding?
vs,
r
+
vs,
l
Bow Table
+
vs,
r
vs,
l
Bow
Reflection
Filter
Bridge to Bow Delay
String
BridgeBody
Air
Reflection filter summarizes all losses per period
(due to bridge, bow, finger, etc.)

Bow-string junction = memoryless lookup table
(or segmented polynomial)
Summary
Julius Smith
Electric Cello Sound Examples (Peder Larson)

Overview
Staccato Notes: (WAV) (MP3)

Spectral Modeling
(short strokes of high bow pressure, as from a bouncing bow)

Bachs First Suite for Unaccompanied Cello: (WAV) (MP3)
Physical Modeling
KL Music
Digital Waveguide
Signal Scattering
Plucked String
Struck String
Karplus Strong
EKS Algorithm
Clarinet
Wind Examples
Bowed Strings
Distortion Guitar
Acoustic Strings
Sound Examples
Linearized Violin
Commuted Piano
Pulse Synthesis
Complete Piano
Sound Examples
Phy Audio Coding
Phy Audio Coding?
Summary
Julius Smith
Soft Clipper
Overview
f (x) =
Spectral Modeling
Physical Modeling
x 1
x3
3 ,
2
3,
1 x 1
x1
3
x=1:0.01:1; plot([(2/3)*ones(1,100), xx. /3, (2/3)*ones(1,100)])

0.8
0.6
0.4
0.2
f(x(n))
KL Music
Digital Waveguide
Signal Scattering
Plucked String
Struck String
Karplus Strong
EKS Algorithm
Clarinet
Wind Examples
Bowed Strings
Distortion Guitar
Acoustic Strings
Sound Examples
Linearized Violin
Commuted Piano
Pulse Synthesis
Complete Piano
Sound Examples
Phy Audio Coding
Phy Audio Coding?
2
,
0.2
0.4
0.6
0.8
2
1.5
0.5
0
x(n)
0.5
1.5
Summary
Julius Smith
Amplifier Distortion + Amplifier Feedback

Overview
Sullivan 1990

Spectral Modeling
Physical Modeling
KL Music
Digital Waveguide
Signal Scattering
Plucked String
Struck String
Karplus Strong
EKS Algorithm
Clarinet
Wind Examples
Bowed Strings
Distortion Guitar
Acoustic Strings
Sound Examples
Linearized Violin
Commuted Piano
Pulse Synthesis
Complete Piano
Sound Examples
Phy Audio Coding
Phy Audio Coding?
Pre-distortion output level

String 1
Nonlinear Distortion
..
.
Output Signal
Pre-distortion gain
String N
Distortion output level

Amplifier
Feedback
Gain
Amplifier Feedback Delay
Distortion output signal often further filtered by an amplifier cabinet

filter, representing speaker cabinet, driver responses, etc.
Summary
Julius Smith
Distortion Guitar Sound Examples

Overview
(Stanford Sondius Project, ca. 1995)

Spectral Modeling
Physical Modeling
KL Music
Digital Waveguide
Signal Scattering
Plucked String
Struck String
Karplus Strong
EKS Algorithm
Clarinet
Wind Examples
Bowed Strings
Distortion Guitar
Acoustic Strings
Sound Examples
Linearized Violin
Commuted Piano
Pulse Synthesis
Complete Piano
Sound Examples
Phy Audio Coding
Phy Audio Coding?
Distortion Guitar: (WAV) (MP3)

Amplifier Feedback 1: (WAV) (MP3)
Amplifier Feedback 2: (WAV) (MP3)
Summary
Julius Smith
Commuted Synthesis of Acoustic Strings (1993)

Overview
e(t)
Trigger
y(t)
s(t)
String
Excitation
Resonator
Output
Spectral Modeling
Physical Modeling
KL Music
Digital Waveguide
Signal Scattering
Plucked String
Struck String
Karplus Strong
EKS Algorithm
Clarinet
Wind Examples
Bowed Strings
Distortion Guitar
Acoustic Strings
Sound Examples
Linearized Violin
Commuted Piano
Pulse Synthesis
Complete Piano
Sound Examples
Phy Audio Coding
Phy Audio Coding?
Schematic diagram of a stringed musical instrument.
Trigger
Excitation
Resonator
String
Output
Equivalent diagram in the linear, time-invariant case.
Trigger
Aggregate
Excitation
a(t)
x(t)
String
Output
Use of an aggregate excitation given by the convolution of original

excitation with the resonator impulse response.
Summary
Julius Smith
Commuted Components
Overview
Trigger
Spectral Modeling
Aggregate
Excitation
a(t)
x(t)
String
Output
Physical Modeling
KL Music
Digital Waveguide
Signal Scattering
Plucked String
Struck String
Karplus Strong
EKS Algorithm
Clarinet
Wind Examples
Bowed Strings
Distortion Guitar
Acoustic Strings
Sound Examples
Linearized Violin
Commuted Piano
Pulse Synthesis
Complete Piano
Sound Examples
Phy Audio Coding
Phy Audio Coding?
Plucked Resonator driving a String.
s(t)
Bridge
Coupling
Guitar
Body
Air
Absorption
Room
Response
y(t)
Output
Possible components of a guitar resonator.
Summary
Julius Smith
Sound Examples
Overview
Electric Guitar (Pick-Ups and/or Body-Model Added) (Stanford

Sondius Project Staccato Systems, Inc. ADI, ca. 1995)
Spectral Modeling
Physical Modeling
KL Music
Digital Waveguide
Signal Scattering
Plucked String
Struck String
Karplus Strong
EKS Algorithm
Clarinet
Wind Examples
Bowed Strings
Distortion Guitar
Acoustic Strings
Sound Examples
Linearized Violin
Commuted Piano
Pulse Synthesis
Complete Piano
Sound Examples
Phy Audio Coding
Phy Audio Coding?
Example 1: (WAV) (MP3)

Virtual wah-wah pedal: (WAV) (MP3)
STK Mandolin
STK Mandolin 1: (WAV) (MP3)

STK Mandolin 2: (WAV) (MP3)
Summary
Julius Smith
Sound Examples
Overview
More Recent Acoustic Guitar

Spectral Modeling
Physical Modeling
KL Music
Digital Waveguide
Signal Scattering
Plucked String
Struck String
Karplus Strong
EKS Algorithm
Clarinet
Wind Examples
Bowed Strings
Distortion Guitar
Acoustic Strings
Sound Examples
Linearized Violin
Commuted Piano
Pulse Synthesis
Complete Piano
Sound Examples
Phy Audio Coding
Phy Audio Coding?
Bach Prelude in E Major: (WAV) (MP3)

soundexamplewavBach silenceLoure in E Major: (WAV) (MP3)
Virtual performance by Dr. Mikael Laurson, Sibelius Institute
Virtual guitar by Helsinki Univ. of Tech., Acoustics Lab3
http://www.acoustics.hut.fi/
Summary
Julius Smith
Commuted Synthesis of Linearized Violin

a)
Overview
Spectral Modeling
Physical Modeling
KL Music
Digital Waveguide
Signal Scattering
Plucked String
Struck String
Karplus Strong
EKS Algorithm
Clarinet
Wind Examples
Bowed Strings
Distortion Guitar
Acoustic Strings
Sound Examples
Linearized Violin
Commuted Piano
Pulse Synthesis
Complete Piano
Sound Examples
Phy Audio Coding
Phy Audio Coding?
Amplitude(n)
Frequency(n)
Impulse
Train
b)
Amplitude(n)
Frequency(n)
Impulse
Train
s(n)
e(n)
String
e(n)
c)
Amplitude(n)
Frequency(n)
Output
x(n)
Resonator
Impulse-Response
Train
Resonator
Output
x(n)
a(n)
String
Output
x(n)
a(n)
String
Assumes ideal Helmholtz motion of string

Sound Examples (Stanford Sondius project, ca. 1995):
Violin 1: (WAV) (MP3)
Bass: (WAV) (MP3)
Violin 2: (WAV) (MP3)
Cello: (WAV) (MP3)
Ensemble: (WAV) (MP3)
Viola 1: (WAV) (MP3)
Viola 2: (WAV) (MP3)
Summary
Julius Smith
Commuted Piano Synthesis (1995)

Overview
Hammer-string interaction pulses (force):

Spectral Modeling
Force
0.5
Physical Modeling
KL Music
Digital Waveguide
Signal Scattering
Plucked String
Struck String
Karplus Strong
EKS Algorithm
Clarinet
Wind Examples
Bowed Strings
Distortion Guitar
Acoustic Strings
Sound Examples
Linearized Violin
Commuted Piano
Pulse Synthesis
Complete Piano
Sound Examples
Phy Audio Coding
Phy Audio Coding?
0.4
0.3
0.2
0.1
5
10
15
Time
20
Summary
Julius Smith
Synthesis of Hammer-String Interaction Pulse

Overview
Impulse
Impulse Response

Spectral Modeling
Lowpass
Filter
Physical Modeling
KL Music
Digital Waveguide
Signal Scattering
Plucked String
Struck String
Karplus Strong
EKS Algorithm
Clarinet
Wind Examples
Bowed Strings
Distortion Guitar
Acoustic Strings
Sound Examples
Linearized Violin
Commuted Piano
Pulse Synthesis
Complete Piano
Sound Examples
Phy Audio Coding
Phy Audio Coding?
Time
Time
Faster collisions correspond to narrower pulses

(nonlinear filter )
For a given velocity, filter is linear time-invariant

Piano is linearized for each hammer velocity
Summary
Julius Smith
Multiple Hammer-String Interaction Pulses

Overview
Superimpose several individual pulses:

Spectral Modeling
Physical Modeling
KL Music
Digital Waveguide
Signal Scattering
Plucked String
Struck String
Karplus Strong
EKS Algorithm
Clarinet
Wind Examples
Bowed Strings
Distortion Guitar
Acoustic Strings
Sound Examples
Linearized Violin
Commuted Piano
Pulse Synthesis
Complete Piano
Sound Examples
Phy Audio Coding
Phy Audio Coding?
Force
Impulse 1
Impulse 2
Impulse 3
LPF1
LPF2
LPF3
String
Input
Time
Summary
Julius Smith
Multiple Hammer-String Interaction Pulses

Overview
Superimpose several individual pulses:

Spectral Modeling
Physical Modeling
KL Music
Digital Waveguide
Signal Scattering
Plucked String
Struck String
Karplus Strong
EKS Algorithm
Clarinet
Wind Examples
Bowed Strings
Distortion Guitar
Acoustic Strings
Sound Examples
Linearized Violin
Commuted Piano
Pulse Synthesis
Complete Piano
Sound Examples
Phy Audio Coding
Phy Audio Coding?
Force
Impulse 1
Impulse 2
Impulse 3
LPF1
LPF2
LPF3
String
Input
Time
As impulse amplitude grows (faster hammer strike), output pulses

become taller and thinner, showing less overlap.
Summary
Julius Smith
Complete Piano Model

Natural Ordering:
Overview
vc
Spectral Modeling
Physical Modeling
KL Music
Digital Waveguide
Signal Scattering
Plucked String
Struck String
Karplus Strong
EKS Algorithm
Clarinet
Wind Examples
Bowed Strings
Distortion Guitar
Acoustic Strings
Sound Examples
Linearized Violin
Commuted Piano
Pulse Synthesis
Complete Piano
Sound Examples
Phy Audio Coding
Phy Audio Coding?
Impulse 1 Tapped 1
GenerDelay
Trigger
ator
Line 2
LPF1
LPF3
LPF2
String
Sound Board
& Enclosure Output
Commuted Ordering:
vc
Sound Board
& Enclosure
Trigger Impulse Response
Tapped
Delay
Line
LPF1
LPF2
String
Output
LPF3
Soundboard and enclosure are commuted

Only need a stored recording of their impulse response
An enormous digital filter is otherwise required
Summary
Julius Smith
Piano and Harpsichord Sound Examples

Overview
(Stanford Sondius Project, ca. 1995)

Spectral Modeling
Physical Modeling
KL Music
Digital Waveguide
Signal Scattering
Plucked String
Struck String
Karplus Strong
EKS Algorithm
Clarinet
Wind Examples
Bowed Strings
Distortion Guitar
Acoustic Strings
Sound Examples
Linearized Violin
Commuted Piano
Pulse Synthesis
Complete Piano
Sound Examples
Phy Audio Coding
Phy Audio Coding?
Piano: (WAV) (MP3)

Harpsichord 1: (WAV) (MP3)
Harpsichord 2: (WAV) (MP3)
Summary
Julius Smith
More Recent Harpsichord Example

Overview
Harpsichord Soundboard Hammer-Response: (WAV) (MP3)

Musical Commuted Harpsichord Example: (WAV) (MP3)
Spectral Modeling
Physical Modeling
KL Music
Digital Waveguide
Signal Scattering
Plucked String
Struck String
Karplus Strong
EKS Algorithm
Clarinet
Wind Examples
Bowed Strings
Distortion Guitar
Acoustic Strings
Sound Examples
Linearized Violin
Commuted Piano
Pulse Synthesis
Complete Piano
Sound Examples
Phy Audio Coding
Phy Audio Coding?
Reference:
Sound Synthesis of the Harpsichord Using a Computationally
Efficient Physical Model,
aki,
Henri Penttinen, Jonte Knif, Mikael Laurson,
by Vesa Valim
and Cumhur Erkut
JASP-2004
Google search: Harpsichord Sound Synthesis
Summary
Julius Smith
Physical Modeling in Audio Coding

Overview
Spectral Modeling
Spectral modeling synthesis is finding application in audio coding.

Can physical modeling synthesis be used as well?
MPEG-4/SAOL already supports essentially all sound synthesis
Physical Modeling
KL Music
Digital Waveguide
Signal Scattering
Plucked String
Struck String
Karplus Strong
EKS Algorithm
Clarinet
Wind Examples
Bowed Strings
Distortion Guitar
Acoustic Strings
Sound Examples
Linearized Violin
Commuted Piano
Pulse Synthesis
Complete Piano
Sound Examples
Phy Audio Coding
Phy Audio Coding?
methods
Ability to encode sounds automatically is limited
Codebook-Excited Linear Prediction (CELP) is a successful
source-filter model (not quite physical)

There are many isolated examples of model-fitting to
recorded data
Good model-based denoising results have been obtained
Coder problem much harder when many sources are mixed
Summary
Julius Smith
Best Known Model-Based Audio Coders

Overview
A cover band can put together a very convincing facsimile of

popular music performance
Spectral Modeling
Physical Modeling
KL Music
Digital Waveguide
Signal Scattering
Plucked String
Struck String
Karplus Strong
EKS Algorithm
Clarinet
Wind Examples
Bowed Strings
Distortion Guitar
Acoustic Strings
Sound Examples
Linearized Violin
Commuted Piano
Pulse Synthesis
Complete Piano
Sound Examples
Phy Audio Coding
Phy Audio Coding?
JOS high-school band Bittersweet
Summary
Julius Smith
Future Physical Modeling in Audio Coding?

Overview
A Cover Band Approach to Model-Based Audio Coding:

Spectral Modeling
1.
Physical Modeling
KL Music
Digital Waveguide
Signal Scattering
Plucked String
Struck String
Karplus Strong
EKS Algorithm
Clarinet
Wind Examples
Bowed Strings
Distortion Guitar
Acoustic Strings
Sound Examples
Linearized Violin
Commuted Piano
Pulse Synthesis
Complete Piano
Sound Examples
Phy Audio Coding
Phy Audio Coding?
2.
3.
Recognize individual audio streams in a mix (CASA)

(I hear a trap set, electric bass, Fender Rhodes, and a strat)
For each stream, calibrate its model heuristically
(Here is what I hear the bass part doing: ...)
Fine-tune the synthetic mix to the real mix
(joint maximum likelihood estimation)
Features of Cover-Band Coding (CBC):
The playing experience of each virtual performer prevents
artifacts musically unreasonable parameters are made

unlikely (Bayesian priors)
An incorrect instrument must imitate its assigned stream
New arrangements can be synthesized by deliberately choosing
a new ensemble!
Summary
Julius Smith
Overview
Spectral Modeling
Physical Modeling
Summary
Summary
Julius Smith
Summary
Overview
We have reviewed a CCRMA-centric slice of the history of digital

sound synthesis (usually starting with results from Bell Labs):
Spectral Modeling
Physical Modeling
Summary
Julius Smith
Wavetable (one period)

Subtractive
Additive
FM
Sampling
Spectral Modeling
Physical Modeling (more in tomorrows 4:30 PM masterclass)
Connections to audio coding
Overview
Spectral Modeling
Physical Modeling
Summary
Sound Acknowledgment
Julius Smith
Overview
Thanks to Emu / Creative Labs for providing a superb-quality

external D/A converter for this talk (an E-Mu 0404/USB)
Spectral Modeling
Physical Modeling
Summary
Julius Smith

AES121heyser Smith

Enviado por

Dados do documento

Direitos autorais

Formatos disponíveis

Compartilhar este documento

Compartilhar ou incorporar documento

Opções de compartilhamento

Você considera este documento útil?

Este conteúdo é inapropriado?

Direitos autorais:

Formatos disponíveis

AES121heyser Smith

Enviado por

Direitos autorais:

Formatos disponíveis

History and Practice of Digital Sound Synthesis

AES-2006 Heyser Lecture 1 / 84

AES-2006 Heyser Lecture 2 / 84

Digital sound synthesis approaches in approximate historical order:

Wavetable (one period)

Some connections with audio coding will be noted

AES-2006 Heyser Lecture 3 / 84

The Knoll, Stanford University

AES-2006 Heyser Lecture 4 / 84

Early Digital Sound Synthesis

AES-2006 Heyser Lecture 5 / 84

Wavetable Synthesis in Music I-V (1957-1969)

AES-2006 Heyser Lecture 6 / 84

Music V Scripting Language (Note Cards)

Essentially Supported in MPEG-4 Structured Audio Orchestra

Language (SAOL) (Music V csound SAOL)

AES-2006 Heyser Lecture 7 / 84

Kelly-Lochbaum Vocal Tract Model

Early Digital Synthesis

Kelly-Lochbaum Vocal Tract Model (Piecewise Cylindrical)

John L. Kelly and Carol Lochbaum (1962)

AES-2006 Heyser Lecture 8 / 84

Bicycle Built for Two: (WAV) (MP3)

Early Digital Synthesis

Vocal part by Kelly and Lochbaum (1961)

AES-2006 Heyser Lecture 9 / 84

Classic Additive-Synthesis Analysis (Heterodyne Comb)

John Grey 1975 CCRMA Tech. Reports 1 & 2

AES-2006 Heyser Lecture 10 / 84

Classic Additive-Synthesis (Sinusoidal Oscillator Envelopes)

John Grey 1975 CCRMA Tech. Reports 1 & 2

AES-2006 Heyser Lecture 11 / 84

Classic Additive Synthesis Diagram

A1 (t) f1 (t) A2 (t) f2 (t) A3 (t) f3 (t) A4 (t) f4 (t)

Early Digital Synthesis

Classic Additive-Synthesis Examples

All of the above

(Synthesized from original John Grey data)

AES-2006 Heyser Lecture 13 / 84

Frequency Modulation (FM) Synthesis

FM synthesis is normally used as a spectral modeling technique.

Early Digital Synthesis

Discovered and developed (1970s) by John M. Chowning

(CCRMA Founding Director)

DX-7 synthesizer (1983)

AES-2006 Heyser Lecture 14 / 84

x(t) = Ac sin[c t + c + Am sin(m t + m )]

Early Digital Synthesis

(Ac , c , c ) specify the carrier sinusoid

AES-2006 Heyser Lecture 15 / 84

Simple FM Brass Patch (1970)

Jean-Claude Risset observation (19641969):

AES-2006 Heyser Lecture 16 / 84

FM Harmonic Amplitudes (Bessel Function of First Kind)

Harmonic number k , FM index :

Early Digital Synthesis

AES-2006 Heyser Lecture 17 / 84

Frequency Modulation (FM) Examples

All examples by John Chowning unless otherwise noted:

Early Digital Synthesis

AES-2006 Heyser Lecture 18 / 84

Early Digital Synthesis

FM voice synthesis can be viewed as compressed modeling of

Modulation Frequency (all three)