670

IEEE TRANSACTIONS ON ACOUSTICS, SPEECH,

AND SIGNAL PROCESSING, VOL.

ASSP-27, NO. 6 , DECEMBER 1979

nal Processin
JAMES H. JUSTICE

AbNtruct-Digital synthesis of music has led to the consideration of models other than the usual additive (Fourier) synthesis of waveforms. One of these methods, based on the FM equation, has been found to be of particular value due to its easy implementation and the richness and evolutionary character of its harmonic structure in time. Any useful synthesis procedure should be accompanied by a corresponding analytic procedure. In this paper we lay the foundations for such a procedure based on the discreteHilbert transform.

I.INTRODUCTION STIMULATING, challenging, and relatively new application of digital signal processing is to be found in the area of music generation via digital computer. By music generation, we mean the production of purposefully structured sounds in the broad sense. That is, wewish to control not only the means of sound synthesis itself, including the shaping of timbre and other tone characteristics, but we wish to provide the means to orchestrate sounds into larger structures or compositions with a minimum of restrictions in achieving whatever sort ofgoal we may have in mind.Inshort, our goalis nothing less than complete control of the sound environment including the warp and woof of sound textures, rhythms, .and pitches; the entire spectrum of sound characteristics from which the patterned cloth of musiciswoven. Indeed, tomorrows composer/performer, the weaver, may find himself as much at home with CRT terminals, graphic tablets, and joy sticks as he now does with his instruments, paper, pen, and baton. Since the creation of the first music generation program in 1958 by M. Mathews of Bell Labs [ 6 ] ,an ever growing number of engineers, mathematicians, computer scientists, and artists, a l l with a common goal, have been intent on converting the digital computer into a responsive and sensitive instrument capable of enhancing and extending our musical powers. The setting was eminently suited for, and indeed demanded the use of digital signal processing principles and techniques of all kinds, and few stones are being left unturned. And, just as any new area of inquiry draws heavily on the tools of the past, its own unique challenges will hopefully always contribute something new which may find use in other areas as well, or perhaps pave a road to new and as yet unseen areas of application. We do not need to look very deeply into the complexities of digital music generation to appreciate the need to find
Manuscript received January 10, 1979;revised June 17,1979. The author is with the Department of Mathematical Sciences, University of Tulsa, Tulsa, OK 74104, and with the Amoco Production Company, Tulsa, OK 74104.

algorithms and data base siructures which provide maximal versatility with minimal externalcontrol. To provide even the privileges that traditional music enjoys presents a computational burden that without sufficient forethought might become almost unmanageable or at best impractical. My purpose here is not to deal with the larger issue of digital production of music, but with only one, and perhaps the most basic aspect of it: the question of synthesis itself. How do we make a sound like a piano, clarinet, or perhaps some heretofore unknown hybrid? I read the same math, physics, acoustics, and music books you did, and it really was not all that hard. One simply takes the ingredients; a proper collection of fundamentals and harmonics, a dash of proper weights, and perhaps anenvelope for flavor, and it is done. It is at this point that we learn how immensely complex the realworld reallyis and how adeptthe earis in discerningevery little aspect of its complexity. Very soon we learned that this recipe was oversimplified in amyriad of ways. It is the nature ofmusical instruments thatno two harmonics of a single tone willevolve dynamically in exactly the same way; much less is this structure preserved as we move from pitch to pitch on a given instrument. The multiplication of the complexities of simple Fourier-type additive synthesis requiring different envelopes on each harmonic, which must changeas wepass from pitch to pitch, already begins to become overwhelming, particularly if real-time generation is being contemplated, and these observations only scratch the surface of the underlying reality. Those of us concerned with problems of synthesis, then, have looked for better and more compact means to mimic nature and in particular to fool our ears into thinking that we really did all that, and more. In 1973, J . Chowning of Stanford University [2] made the observation that truly complex spectra which could evolve in complex ways could be relatively easily obtained using the FM equation with suitable parameters and envelopes. The amazing quality of the sounds obtainable in this way, and the compact means of their production, led many of us to look to FM as a viable synthesis procedure. There was just one problem that we had to face, and I think it is summed up by the fact that when a student in my music class approaches me with the inevitable question, HOWdo I make a ____ (instrument) sound?, 1 am afraid I come as close as perhaps I will ever be to the ministry as I place hand on shoulder and dispense the best advice I can sometimes give. . . Indeed FM works very well, if we just knew how to control all those little parameters, and that is what this paper [5] is about.

0096-3518/79/1200-0670$00.75 01979 IEEE

USIC TION

IN PROCESSING JUSTICE: SIGNAL ANALYTIC

67 1

11. BASICs The goal of research in sound synthesis is to store a minimal amount of information to be combined with minimal computation to produce sufficiently complex sound structures to be of interest. The FM equation, in itsoriginal form, requires the storage of only a few parameters, some coarsely sampled envelopes, and a single cycle of a sine or cosine waveform, suitably sampled. The success of Chownings FM techriique [2] for generating a wide variety of sounds with sophisticated spectra which evolve in time and with great economy of means led us to investigate the underlying principles to determine how far we might go in analyzing and synthesizing sound by this technique. In particular, we were interested in two questions. First, is therean analysis technique which will yield the parameters which generate a givenFM signal? Second, we ask the question, to what extent might FM synthesis be used to regenerate or to approximate a given signal? The latter question is of great interest as it is extremely difficult to synthesize certain types of signals using techniques that are practical given limited core space and computation time.

$(t)= arctan (SI (t)/sR(t)).

(4b)

It is easy to see now that

S R (t)= I ( t ) cos $(t).

(5)

That is, our original signal has been decomposed into an envelope Z ( t ) and aphase function $(t).

B. The DiscreteAnalytic Signal The analysis in Section 11-A has a discrete analog. Let SR (n) be a real, discrete signal. Then we may interpretit as the sampled part of the real part of an analytic signal. The signal sI(n) may be obtained in a manner analogous to the method outlined in Section 11-A. Let sR(n), 0 < n < N be our real signal (take N to be a power of 2). Perform a fast Fourier transform on sR(n) to obtain S, (n). Define S(n) by the relations
N/2 < n < N n = 0 , N/2. An inverse fast Fourier transform of S yields our analytic discrete signal s(n) = sR(n) + isI(n),
0 <n <N.

A . The Analytic Signal Let us suppose that we are given a real signal which we denote by s , (t). The Fourier transform of SR , s,(~) will be conjugate symmetric since SR is real. If q ( t )is another signal then the Fourier transform of SR( t )+ isl(t) is given by s , (a) f i S I ( o ) where SI is the Fourier transform of SI. If the Fourier transform S(w) = S,(W) f iSz(w) vanishes for w < 0, then it is the Fourier transform of an analytic signal. AS long as SR is absolutely integrable there is a signal sz(t) for which S ( w ) = S R ( w ) t iSI(w), vanishes for w < 0 and SO ~ ( = t) SR (t)t i ~ I ( t ) is analytic. We may call s ( t ) the analytic signal associated with sR(t). We may now ask, how m i k t we obtain sz(t)? It is easy to verify that

Again, we may write s in the form s(n) = Z(n>exp (i@(n)) from which we obtain sR(n)=l(n)cos($(n))* (8) We should point out that sI is obtainable directly as the inverse discrete Fourier transform of the Hilbert transform of S , given by [3] (-iSR(n)
0 <n < N / 2

(7)

n = 0,N / 2
using thefactthat sR and sI arerealsignals.If S(w) = 0 for w < 0, then we obtain we require (n) N/2

<n <N .

As a result, we may Fourier transform s, multiply by a suitable weighting function as indicated in (2), and inverse Fourier transform to obtain sI as the imaginary part of the signal s so obtained. Having gotten
s(t) = sR(t) t isI(t)

We now have the ability to write a causal discrete signal in the form (8) with time-variant envelope and phase terms. We proceed to a further analysis of this relationship. At this point, let us remark that in general, if we plot the modulating angle @(t) in ( 9 , then the instantaneous frequency of the signal S R ( ~ at ) time to is proportional to the slope of the curve defined by $(t) at to. 111. USE OF THE ANALYTIC SIGNAL Let us write the equation for a frequency modulated sinusoid in the form

we may write (3) in the form

s (t)= I ( t ) exp (i$(t))

f(t) = M ) cos (%t + @ r n (t))

(1 0)

where I(t) =( S i ( t ) and

SI(t))Z

where Zc is the carrier amplitude or envelope, w, is the carrier frequency which is being modulated, and &(t) is the angle modulator and is proportional to the integral of the frequency of the modulating function with time.

612

IEEE TRANSACTIONS

ON ACOUSTICS, SPEECH,

AND SIGNAL PROCESSING, VOL. ASSP-27, NO. 6 , DECEMBER 1979

If the modulating angle is determined by an equation of the form

@m(t)= Im ( t )COS Pm t

(1 1 )

then, substituting into (10) we obtain the first equation considered by Chowning [2]

f ( t )=I&) cos (0,t -I- Im (t)cos p , t)

(1 2)

where we choose to use the cosine in place of the sine in view of Section 11. It is clear that not all possible modulating functions could be easily represented in the form ( 1 l ) ,but this is avery simple and natural form to work with, particularly in view of Section 11. A. Analysis of FM Signals Let us begin by supposing that we have a signal + ( t ) generated by an FM process given by (12). We shall consider the problem of determining the parameters in the equation. Applying the results of Section I1we may obtain sR(t) in the form
SR(t)= I ( t ) COS (6m (t)).

questions of synthesis fora later section. Inthe remainder of the paper we assume that a l l envelopes areslowly varying so that they may be coarsely sampled and stored or that they may be generated by simple algorithms. This assumption is made for practical considerations. Without it, almost any signal could be represented (in theory, at least) in almost any of the forms considered. Let us begin by considering our analysis for a large class of signalsusedinmusic computation. Consider a signal of the form
N

SR(t)=a(t)
k=O

C k COS k a t

( 1 5)

where we have a periodic signal represented by the summation being amplitude modulated by the function a(t). We do not consider this type of signal because we need another means for synthesizing it, but rather because it is of interest to learn how it relates to the class of FM signals. Writing SR(t) in complex exponential form,
N

(13)

If we now extract the linear trend from , 6 , we obtain the carrier angular frequency w,. This may be donebyfitting a least-squares straight line through the phase @,(t), and calculating its slope. Subtracting the linear trend from @ leaves us with a new signal, the modulator s m ( f ) = @,(t) - a,t. Applying the analysis of Section I1 again to the modulator yields
Srn (t)= Im (t)COS Ym (t).

sR(t) = 1/2a(t)
k=O

Ck(exp (&at+ exp (-&at))

(16)

we easily obtain the Fouriertransform of sR(t),

S,(W) = 1/2

2 Ck [A
k=O

(O

- kQ) A(u -I- k a ) ]

( 1 7)

( 14)

where A(w) is the Fourier transform of a(t). Our analysis in Section I1 tells us that the desired analytic signal is givenby
S(t) =

If now, the originalsignal obeyed ( 1 2), the function y,(t) w i l be a straight line whoseslope will be .0 , In the event that this is not so, our signal simply did not obey ( 1 2), and we are ready for further analysis. Under our original assumption that the signal obeyed (12), we have now determined all the parameters in that equation by a four step process which we may summarize here. 1 ) Obtain I,(t) and @,(t) from Section I1 analysis. 2 ) Extract linear trend from Gm (t)and obtain w, . 3) Obtain I,(t) and y,(t) from @,(t) - w,t using Section I1 analysis. 4) Extract linear trend from ym to obtain p , .

1/2n

2c k
k=O

im

[(A(W - k Q ) t A ( w t k a ) ]

. exp (iwt) dw
which becomes, after change of variable
N
S(t)

(18)

= 1/2n
k=O

Ck

lkC2

A(U) exp (i(0 k a ) t ) d W

t 1/2n

k=O

2 Jm A(o)
Ck
kSl

exp (i(w

k a ) t) dw.

The residual part of y m can only be a numerical error if (12) (1 9 ) were satisfied by our signal. We note now that if y,(t) were not a straight line, then the At this point we are faced with a possible aliasing problem for modulator is in turn being modulated and we may continue A(w). For simplicity in carrying out our analysis, let us supthe above analysis ad infinitum or ad nauseum if we so desire. pose that. A ( w ) = 0 for 1 w I > a2/2 (this is not much of a restriction, since envelopes may be assumed to be slowly B. Signal Analysis varying). We have indicated how the analysis technique discussed in We shall further make the simplifying assumption that our Section I1 can be applied to determine the parameters of an signal is unbiased, which is to say that the term Co in our exFM signal. We should now like to pursue the question of more pansion is zero. If this is not satisfied, the multiplier of Co general signal analysis. Specifically, we ask, what kind of in- after conversion will not be a(t) but rather a complex term formation can we get about more arbitrary signals and how whose real part is a(t)/2. might it be utilized for sound synthesis? W e shall reserve Under this simplifying assumption, (19) becomes

JUSTICE: ANALYTICSIGNALPROCESSINGINMUSICCOMPUTATION
0. 0 1

673

-1
~~

.........

...................................................................................................... . . ...,.. .. . . ... .. . .. ... . . ... .. . .. ... . .. , ..,,.,-.. ......,. ,,,.., ..... ,,. . ..... ............................ . ... .. ... ... . . ... . . ... .. . .. ... . . ... . . ... .. ... ,.. .. . .. .. . . .. . .. ....,.,,., . . . . .. ... .. .. ..
Example 1. Exponentiallydecayingcosine.Theenvelopecalculated is the true envelope.

If we now assume a(t) 2 0 so that no phase shifts were introduced by the envelope, we see that the envelope calculated by the methodsof Section I1 will be given by

forO<n<512

and N = 1 , 5 ,

(23)

where we recall, now, that (21) will be more complicated if A(w)#Ofor I o I > a / 2 . The angle modulator calculated in Section I1 will be given by $(t)= arctan( c k sin k n t / x C k cos kat) (22)

which we see can be quite a complicated term. A number of comments based on the above analysis are now in order. First,it is easy to see that if our originalsignal sR(t) had consisted of a single term with envelope a(t), then N f ( f )=a(t) c k COS k a t . (21) and (22) would yield the exact original envelope and the k=O exact angular frequency of the originalsignal. In any other case this statement is false. The summation in (21) is periodic, We have shown that the angle moduiator of this signal is of i l l oscillate between futed values. This in- the form (22), (25): however, and so w dicates that I ( t ) i n (21) will follow the original envelope in maximum deviation from zero. A multiple of a ( t ) could be sampled simply by taking the samples 2n/a time units apart. This fact can be illustrated by examples given in Examples 1 The argument in this expression is clearly a periodic function and 2. The signal used in these examples was of the form (15), with period 2n/Q. It follows that $(t) will be periodic (we restrain $ to the interval -n/2 < $ < n/2) with period 2n/Q2. (231,

respectively. a(t) was taken to be exponentially decaying. For the purpose of graphing, the signalwas then sampled at every fifthpoint (as a result, the signal graphs in some cases do notresemble their true form). It can be clearly seen from the examples that the envelope follows the original and in the case of a single cosine duplicates it towithin obtainable numerical accuracy. Let us now recall that we have obtained an analysis of the signal (15) in the quite different form (5). If it were true that the phase had a simple form [recall (22)], then we might hope to resynthesize the signal from (5). Again, let us turn to an example. Consider once more a signal of the form (15), (24):

6 74

IEEE TRANSACTIONS ON ACOUSTICS,SPEECH, AND SIGNAL PROCESSING, VOL. ASSP-27, NO. 6 , DECEMBER 1979

(b)

Example 2. (a) Exponentially decaying Fourier series (15), (23). (b) Linear phase

from (a).

JUSTICE:ANALYTIC SIGNAL PROCESSINGINMUSICCOMPUTATION

As a result, @ could be represented by a Fourier series and the original signal regenerated from an expression of the form [see (511

The question now arises, can we stopafter some finite number of steps, discard the remaining angle modulation, and not appreciably distort the original signal? Certainly for FM-type signals this process terminates after a few steps. f ( t ) = I ( t ) cos (at t f k sin k a t ) (26) Is it rapidly convergent forother signals? We have had no l i where we have removed the constraint - 4 2 < @ < 7r/2 and ex- time to obtaina complete answer to his question but w take a slightly different look at it before leaving the topic. tracted the linear trend, Let us consider the harmonic structtire of signalsof the We have been successful, then, in analyzing signals of the form (lo), (29): form (24) using the techniques of Section 11. However, we have considered in our example (24) only a small class of all f ( t )=I@) cos (at @(t)). (29) possible signals. Since (24) is not the most general form possible for a signal (under the assumption that a(t) is slowly It is well known that if the modulating angle @ has the form varying), the analysis of it may be too restrictive for general @(t) =1 1 ( 0 sin ( 0 1 0 , (3 0) application. Needless to say, we would not wish to reconstruct a signalof the form (24) using this analysis, but the then f has the expansion example is instructive. m At this point, an interesting possibility suggests itself. Havf(t) = Z(t)C&(I1 (t))cos (ut-I. n o 1 t). (3 1) ing obtained a representation of (15) in the form (5) where -m @ may be quite complicated, it seems reasonable to continue The harmonic structure offis seen to consist of all frequencies the analysis. Extracting the linear trend from @ and analyzing the result w t n o l for integer n, positive or negative. As Chowning points out, folding in discrete systems results in very comas outlined in Section 111-A, we obtain plex spectra from such signals.The functions J,(x) are, of @(t) = w1 t -I. I 2 ( t ) cos @ 2 ( t ) . (27) course, the Bessel functions of order n. Extending this analysis astepfurther, suppose that the Our original signal (1 5) is now in theform modulator @ in (29) has the form SR ( t )= I ( t ) cos (&I 1 ,(t)cos $2 (t)). (28)

We may continue this process as long as we like obtaining deeper and deeper layers of modulators.

@(t) =
k =1

Ik(t) sin ( W k t ) .

(32)

676

IEEE TRANSACTIONS OM ACOUSTICS, SPEECH, AND SIGNAL

PROCESSING, VOL. ASSP-27, NO. 6 , DECEMBER 1979

The expansion of the signal f ( t ) then takes the form

'cos [ ( u t n l u l t n 2 u 2 t - . . t n N w N ) t ] . (33) (Observe that the signal (15 ) which we analyzed earlier in (26) turns out to be a special case of (29) and (32) given by making the particular choice W k = k a , Ik = f k , where N may be infinite.) The spectrum of such a signal can clearly achieve a high level of complexity and surely represents an interesting class of signals. However, our suggested analysis of a signal into the form of nested layers of modulators departs quickly from the above analysis. Let us consider the harmonic structure of an FM signal with modulated modulator (where we have gone one extra step in applying analytical signal analysis to a signal as suggested above). Our signal will be of the form
f (t)= I ( t ) cos (utt Il (t)sin (ul t t I , ( t ) . sin w 2t)) (34)

drops below some critical level. The bandwidth of an FM signal is usually defined as the width of the frequency spectrum which contains all components having a weight greater than or equal to 1 percent of the amplitude of the unmodulated signal. Since the weights in (37) are given by values of Bessel functions evaluated at the modulation index Zl,we need only consider the behavior of the Bessel functions. Tables of values of Bessel functions reveal that for small values of 11,only a few of the low-order Bessel functions willbe significant. As Il increases, more and more of the weights in (37) become significant and so the spectrum becomes correspondingly richer. This general principle should be kept in mind aswe consider synthesis techniques. Tables relating modulation index to weights may be found in standard FM treatises. Having formed some idea of the bandwidth of a signal, we should recall the Shannon sampling theorem which says that a signal whose spectrum is contained in the frequency range I f 1 < can be perfectly reconstructed by sampling at a rate not more than 1 / 2 a s/sample. This condition must not be violated if we hope to faithfully reproduce a signal.

B. Signals with Periodic Modulators where we are using sines in place of cosines to facilitate the Let us suppose that we aregiven a signal f(t) which we use of Bessel functions in obtaining an expansion of the funcanalyze bythe methods of Section 111-A. We shall further tion. Carrying out the expansion, we obtain suppose thatat some level of the analysis, the modulator I m \ under consideration is essentially periodic so that we may f ( t ) = I COS ut t Il Jfl(Iz) sin ((a1 t n u 2 )t ) approximate it with a Fourier Series. For ease in writing, -m suppose that this occurs a t the first level. We then have a Whenwe realize that this signal (35) is similar in form (but representation of our signal in the form (29), (32), (38): with infinitely many terms) to the signalgiven in (29) and f ( t ) = I ( t ) COS U t t 4 1 (t) c kS hkul t . (32) which resulted in the expansion (33), it seems clear that (38) the harmonic structure of a signal of the form (34), given by We may now read off the bandwidth of the signal from the (35), may well defy comprehension. However, it is on the analysis given in (33) anddetermine a suitable sampling rate. basis of this general form that we suggest the utility of the To recreate the signal, the standard procedure would be to form (34) andthe even more complicated forms obtained store one period of the cosine and the modulator which are from it by the methods of Section 111-A. It seems reasonable sampled at appropriate rates and the envelope factors applied to suggest that many signals may be represented in this easily either from equation or from stored sampled values as in the obtained and relatively compact form. case of usual FM synthesis. In consideration of thefact that (38) contains as a very IV. ANALYSIS AND SYNTHESIS OF SIGNALS special case all signals of the form (24), (39) (where I ( t ) may In this section we wish to look at the practical application take on amore complex form), of the principles developed in the preceding sections. In par-

ticular, we shall be concerned with methods of analysis and synthesis, and with bandwidth and sampling considerations.

f ( t ) = a(t)

c k COS kat,

(39)

A. General Comments on Bandwidth and Sampling Letus begin by recalling the standard equationfor quency modulation (29), (30), (36):
f ( t ) = I ( t ) cos ( w t + 1 1 (t)sin 0
1 t)

it is the author's opinion that this method of synthesis is an important one and should be considered as such. freThe most general form of signal to which this discussion applies is given by (3 6) t Il(t)sin (a1 t+ * .
3

which has the expansion (31), (37)

m

tIfl-l(t)>sin a f l - l t + l ncck ( t ) sinku,) -..).

(37)
C . Signals with Quasi-PeriodicModulaton

f (t)= I ( t ) C J,(Il (t))cos ( u t + n u l t)).

-m

(40) We move now to a generalization of the periodic modulator case, which again, we have considered earlier. We shall define

The harmonic structure is seen to consist of frequency components u t n u 1 with weights J , ( Z 1 ( t ) ) for --03 < n < -03. A frequency component in a signal may be ignored if its weight

ANALYTIC JUSTICE: COMPUTATION MUSIC

SIGNAL IN PROCESSING

617

amodulator (41):

to be quasi-periodic if it is of the form (32),

@(t) =

Ik(t) sin ( u k t ) .

(41)

The modulator may become periodic if,for example, the angular frequencies o k have rational ratios to each other, and the Iks are constant. Again we shall suppose that we have analyzed a signal obtaining nested modulators until finally a modulator turns outto be quasi-periodic. For simplicity we may takeour signal to have only one modulator,

f ( t )= I ( t ) cos (at +

Ik(t) sin (cdkt)) ,

(42)

in which case, we have obtained an analysis of the signal in (33). The quasi-periodic case is clearly more general thanthe periodic case, but is perhaps less important for analysis because it would be difficult to obtainthe parameters fora quasi-periodic modulator. Being a generalization of the periodic case, however, it is the authors opinion that it may be important for signal synthesis. The only difficulty is that in the nonperiodic case, the modulator waveform could not be stored well in sampled form. An inspection of (33) should convince the reader of the possible wealth of spectral components available here. Again, this could be compounded by nesting modulators.

mation procedure can be resorted to which may or may not yield good results depending on how itis applied. Let us suppose that we are given a sampled signal for analysis. Break the signal into small successive pieces of length 2n for some n. Carry out the analysis of Section 111-A on each piece. At some level of modulation, fit the phase curve obtained from the analysis with a straight line segment. Do this for each piece of the signal and store an approximate value for the envelope and one for the modulation frequency obtained from the straight line fit. If the pieces are small enough, the fit will be reasonably good. Recreate the signal using the approximatevalues obtained for each piece of the signal, letting the modulator frequency slide between the successivevalues obtained in the analysis. The result should be an approximation to the original signal. Example 4 shows the analysis for a signal whose modulator has a continuously varying frequency for which this technique is eminently suited. This technique can be combined with methods discussed earlier in this paper. V. CONCLUSION

FM synthesis techniques are enjoying widespread use and

popularity among those interested in the digital synthesis of music. Because of their easily implemented form and compactstructure,they are not likely to be soon supplanted. While intuition and tinkering with the controls are sometimes helpful guides to using these methods, their use has been hampered bythe lack of suitable analytic techniques which will help us to achieve more precisely our desired goals. Inpractice, we may wish to get into the ballpark analytically, and then modify the parameters until the result that pleases us is achieved. In this paper, we haveshown that an easily implemented procedure based on the discrete Hilbert transform leads to an analysis procedure which is compatible with the concept of FM synthesis. It further broadens our field of view beyond the usual Fourier methods, which, while importantand of definite value inmusic generation, can no longer be considered to be our only important basis for synthesis. In this paper we have raised several questions which hopefully will find answers and perhaps light the way to more powerful or versatile procedures for signal analysis. Since this paper was presented, some of the ideas contained herein havealready been finding their way into digital music synthesis. VI. EXAMPLES The accompanying figures show a number of examples which illustrate some of the main ideas in the text. In all examples, thesignal(envdlopewas theone shown in Fig. 1. Where applicable, all modulators were given the envelope shown in Fig. 2 . In allcases envelopes are graphed by 0 and signals are graphed by X. We have connected the graphed points for ease in pattern recognition, but in most cases,sampling of the signalswas too coarse to reveal their true shape, which was generally irrelevant to the example. The few exceptions to this are accurately graphed. Example 2 shows various parts of the analysis of an exponentially decaying Fourier series just to verify the discussion in the text. The envelope only follows the true envelope as dis-

D. Signals with Nested Modulators As a last resort in signal analysis, themethod of Section 111-A can be applied repeatedly to successive modulator terms of a signal until, one hopes, the process can be terminated, the remaining modulator term being neither periodic nor quasiperiodic, but hopefully negligible so that it may be discarded. This type of analysis results in signals of the form (34), (43):
f ( t ) = I ( t ) cos (utt II(t)sin (a1 t + I2 ( t )sin w 2t))

(43)

which represents two nested modulators. The analysis of the signal givenin (3.9, (44),
m

f ( t ) =ICOS

ut t Il
-m

Jn(12) sin ((ol t n o z )t ) ,

,)

(44)

indicates that it is related to the class of signals with quasiperiodic modulators. The advantage here, however, is that we automatically have the parameters for the modulator term in (35) from our analysis which led to (34). Again, the nested modulator technique of signal synthesis, leads to signals of a quasi-periodic nature in compact form (we do not have to worry about storing one period, possibly of great length, of the modulator). Only one period of the cosine aswellas the envelopes need to be stored in this method of synthesis.
E. A Practical Approach to Synthesis Assuming that we wish to employ analytic signal techniques for signal analysis and synthesis, a relatively simple approxi-

678

IEEE TRANSACTIONS ON ACOUSTICS, SPEECH, AND SIGNAL PROCESSING, VOL. ASSP-27, NO. 6 , DECEMBER 1979

Fig. 1. Illustrating envelope on signal in all examples.

cussed, andthe periodic modulator is correctly derived as shown in Example 2(c). Example 3 is a standard FM equation of the form (12). Its phase and modulator are correctly derived as shown in Ex-

ample 3(b) and (c). The modulator envelope is shown in Example 3(d) and its phase in Example 3(e). The remaining error after extracting the linear trend from the modulator is shown in Example 3(e). The error is generally around k0.1 rad.

JUSTICE: ANALYTIC SIGNAL PROCESSING IN MUSIC COMPUTATION

679

..................................... .?............................................................................................................... ....... ......................... ........... . . . U . . ~ .. * . ...................... ..".+. . . . . . . . . . . . , , , ,.,,, ).,... ,.......... ..*,, ...,.,.,., .,. ....... , L . . . ................. ..,.,..,,......
'*a

.\1

.SI I,... .......I..........*...................

.*.

... *

(a)

.................................... . . (....., . . ........... .. . .. .. .. .. .. .... .. . . . . . ... . . . ,?...., .......................... ......... . . . . . . .. .. .. ..> ....... . .. ......... . . . .. .. .. .. .. .. .. . ...
1 1 . . ( .

I , , . e . .

.I,

.....I,

(b)
Example 3. (a) Standard FM signal of the form (12).

(b) Linear phase extracted from (a).

680

IEEETRANSACTIONS ON ACOUSTICS,SPEECH, AND SIGNALPROCESSING, VOL. ASSP-27, NO. 6 , DECEMBER 1979

, r

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

I"

. e
I"

*0

. o

..

. . . . . . .

,
i
A

(dl Example 3. (Continued.) (c) Modulator derived from (a). (d) Envelope derived from ( c ) .

JUSTICE:ANALYTIC SIGNAL PROCESSING IN MUSICCOMPUTATION

681

Il..**l..n.)P.sn

,.. . .-...,.., . ... ..... . ..._.. . . . . ... ................................................ . ........._..... - .-.....-..-...-..-........ .,.,..l.n** ._. . ~.v.,,,.,, . . #",.,,',.,,.,n .,,..-.,.. .,.. "-.,,#.,,?,.,.,.** ..,. ,.,,, .(..."....*... , ." .............. .. ..
~

1.

I 8

..,."1/1..* ( 1 I,,.,~..,,. " . . l . ( ~ , I . I .. ~ . ~ ~ 1 . . . 1 1 1 . . 1 ~ 1 . . . 1 ~ I 1 , ~ , . . * . , 1 . . 1 1 . 1 " , . . . " , . 1 , t l - - l

1,ll.i)

l.lll.

."....C..

.*.",,,"*.I. :.I."\I,.I.1".,...

(f 1

Example 3. (Continued.)(e)

Modulatorlinearphase derived from (c). (f) Modulator errortermafterlinear subtracted from (e). (Error is generally around 0.1 rad.)

trend is

682

IEEE TRANSACTIONS ONACOUSTICS,SPEECH,

AND SIGNAL PROCESSING, VOL. ASSP-27, NO.

6 , DECEMBER 1979

... . ........ ...........-.. ..................

...
V."

. .*

............... _....... ..._.............. ..*..................................................... ......... ........... ,,.. ..,n . ."...... ...................... . ,. ~ . . . . . . .. ..... .. . ................. ....... .,, ................... . .. ..... .. . . .. ..... .. . . .. ................ . . . . . . . . . .~, . .. . . . . .
........I._.

0..

LI.......~.*....I.U.".*.............

.,""I

(a)

..................... ...............
(b)
Example 4. (a) Envelope extracted from F M signal with modulator whose frequency increases linearly in time. (b) Linear phase extracted from (a).

Example 4 isan FM signalwith amodulator whose frequency increases linearly in time. The resulting phase of the modulator should therefore be parabolic. The modulator

frequency eventually overwhelms the sampling interval, leading to some degradation on the right-hand side of these plots. Analysis of the signal correctly derives the modulator in

683

............. ............-. ......................................................................................... ...-......... ............... ........................................................................... ............... .VV.,.? . . . . *. . . ... . . . . . .. . ..*. ...... *, .......... b , . . ...... ... .. ...
a,...-.,
.U?.,:l
..1*.,1..h,

*.0,.

(4
Example 4. (Continued.)(c)Modulatorderivedfrom(a).(Samplinginterval too coarse for increasing modulatorfrequency at right.) (d) Envelope extracted from modulatorin (c).

Example 4(c), and further analysis yields its parabolic phase in Example 4(e). All signalsused in the examples contained 512 points and

were sampled at every fifth point for thepurposes of graphing. The signal envelope chosen is one which is characteristic of bells,gongs, or struck objects which follow this pattern of

684

IEEE TRANSACTIONS ON ACOUSTICS, SPEECH, AND SIGNAL PROCESSING, VOL. ASSP-27, NO. 6 , DECEMBER 1979

~............ ........... ......... ........... .... .. . .........

....,TI .*.,.,.

................. ,.,,, ..,..

. * . , I ,.1

c,*.,.bbL,ur,.

>,>.

. ..... , . .. . .. .. , . .., ... . . . . . . . . * , . . . . ... ... . . . . . ...~. ... ....... ..,.,,, ..,. ,,..,,...,.. ..r, ,>. ,. .,. ..,. ,...., , ..,,.. _... .,,,. , . , : . . , a . . , . ~ .,.,,..,,, ,,? ,..~.,.,..*,, . . , . . . r .,z>,,,,,7.,1r>
1

. . .-

..

..,.,..,.,I

, , I

.,....

.. . . .. . .....
.V

(e) Example 4. (Continued.) (e) Parabolic phase of modulator derived from (c).

decay after being struck. The modulator envelope was chosen more for its characteristic shape than for any other reason. If used in music synthesis, the effect would be that maximum modulation would occur aboutthemidpoint of the sound duration. This is the time when the harmonic structure would be the richest, and perhaps even noisy, returning to a strong sense of distinct pitch on either side. Examples of all the types of signals discussed were played when the paper was presented and sounds ranging from musical, to explosions, to effects like air bubbling through a liquid were demonstrated, but thepossibilities are endless. VII. FORTHE LISTENER One of the great joys of working in this field has been the many opportunities which I have had to come to know a great number of a rare breed of people who are bothartist and technician, composer and programmer-a breed of jack-ofall-trades innovators which I am glad to know have not died out yet. They come from a wide variety of backgrounds and work in many places under all sorts of conditions, generally less than ideal. A few years ago I had the honor of meeting Aaron Copland, a great man and a great composer who was willing to take the time to listen to the work these people are doing and to help us place our work into historical perspective. A recording was made of computer works from around the world selected for

a concert in his honor. The recording is available for those who would like to hear a variety of works representing many approaches to this new medium. The information is contained in [71.

ACKNOWLEDGMENT I would like to thank my associate, B. Vassaur, for his suggestions in revising this manuscript, and the Amoco Production Company for technical assistance.
REFERENCES
[ l ] R. Bracewell, The Fourier Transform and Its Applications, New York: McGraw-Hill, 1965. [2] J. M. Chowning, The synthesis of complex audio spectra by means offrequencymodulation, J. Audio Eng. Soc., vol. 21, no. 7,1973. [3] V. Cizek, Discrete Hilbert transform, IEEE Trans. Audio EZectroacoust., vol. AU-18, no. 4,1970. [4] A. Hund, Frequency Modulation. New York: McGraw-Hill, 1942. [5] J. H. Justice, Analytic signal processing in music computation, aresented at 1st Int.Comaut. MusicConf.. Massachusetts Inst. Tech., Cambridge, 1976. Cambridge, 161 M. Mathews. The Technology -- of ComputerMusic. MA: MIT Press, 1969. [7] NewDirections,2 LP album, Tulsa Studios, Box T Admiral Station, Tulsa, OK 74112. [8] A. Oppenheim and R. Schafer, Digital Signal Processing. Englewood Cliffs, NJ: Prentice-Hall, 1975.
L 1
~