Escolar Documentos
Profissional Documentos
Cultura Documentos
Aims
The aims of this course are to introduce the principles and applications of information theory.
The course will study how information is measured in terms of probability and entropy, and the
relationships among conditional and joint entropies; how these are used to calculate the capacity
of a communication channel, with and without noise; coding schemes, including error correcting
codes; how discrete channels and measures of information generalise to their continuous forms;
the Fourier perspective; and extensions to wavelets, complexity, compression, and efficient coding
of audio-visual information.
Lectures
• Foundations: probability, uncertainty, information. How concepts of randomness,
redundancy, compressibility, noise, bandwidth, and uncertainty are related to information.
Ensembles, random variables, marginal and conditional probabilities. How the metrics of
information are grounded in the rules of probability.
• Entropies defined, and why they are measures of information. Marginal entropy,
joint entropy, conditional entropy, and the Chain Rule for entropy. Mutual information
between ensembles of random variables. Why entropy is the fundamental measure of infor-
mation content.
• Source coding theorem; prefix, variable-, and fixed-length codes. Symbol codes.
The binary symmetric channel. Capacity of a noiseless discrete channel. Error correcting
codes.
• Channel types, properties, noise, and channel capacity. Perfect communication
through a noisy channel. Capacity of a discrete channel as the maximum of its mutual
information over all possible input distributions.
• Continuous information; density; noisy channel coding theorem. Extensions of the
discrete entropies and measures to the continuous case. Signal-to-noise ratio; power spectral
density. Gaussian channels. Relative significance of bandwidth and noise limitations. The
Shannon rate limit and efficiency for noisy continuous channels.
• Fourier series, convergence, orthogonal representation. Generalised signal expan-
sions in vector spaces. Independence. Representation of continuous or discrete data by
complex exponentials. The Fourier basis. Fourier series for periodic functions. Examples.
• Useful Fourier theorems; transform pairs. Sampling; aliasing. The Fourier trans-
form for non-periodic functions. Properties of the transform, and examples. Nyquist’s
Sampling Theorem derived, and the cause (and removal) of aliasing.
• Discrete Fourier transform. Fast Fourier Transform Algorithms. Efficient al-
gorithms for computing Fourier transforms of discrete data. Computational complexity.
Filters, correlation, modulation, demodulation, coherence.
• The quantised degrees-of-freedom in a continuous signal. Why a continuous sig-
nal of finite bandwidth and duration has a fixed number of degrees-of-freedom. Diverse
illustrations of the principle that information, even in such a signal, comes in quantised,
countable, packets.
• Gabor-Heisenberg-Weyl uncertainty relation. Optimal “Logons”. Unification of
the time-domain and the frequency-domain as endpoints of a continuous deformation. The
Uncertainty Principle and its optimal solution by Gabor’s expansion basis of “logons”.
Multi-resolution wavelet codes. Extension to images, for analysis and compression.
• Kolmogorov complexity. Minimal description length. Definition of the algorithmic
complexity of a data sequence, and its relation to the entropy of the distribution from
which the data was drawn. Fractals. Minimal description length, and why this measure of
complexity is not computable.
Objectives
• calculate the information content of a random variable from its probability distribution
• relate the joint, conditional, and marginal entropies of variables in terms of their coupled
probabilities
• define channel capacities and properties using Shannon’s Theorems
• construct efficient codes for data on imperfect communication channels
• generalise the discrete concepts to continuous signals on continuous channels
• understand Fourier Transforms and the main ideas of efficient algorithms for them
• describe the information resolution and compression properties of wavelets
Recommended book
* Cover, T.M. & Thomas, J.A. (1991). Elements of information theory. New York: Wiley.
Information Theory and Coding
3
• Dennis GABOR (1900-1979) crystallized Hartley’s insight by formu-
lating a general Uncertainty Principle for information, expressing
the trade-off for resolution between bandwidth and time. (Signals
that are well specified in frequency content must be poorly localized
in time, and those that are well localized in time must be poorly
specified in frequency content.) He formalized the “Information Di-
agram” to describe this fundamental trade-off, and derived the con-
tinuous family of functions which optimize (minimize) the conjoint
uncertainty relation. In 1974 Gabor won the Nobel Prize in Physics
for his work in Fourier optics, including the invention of holography.
• Claude SHANNON (together with Warren WEAVER) in 1949 wrote
the definitive, classic, work in information theory: Mathematical
Theory of Communication. Divided into separate treatments for
continuous-time and discrete-time signals, systems, and channels,
this book laid out all of the key concepts and relationships that de-
fine the field today. In particular, he proved the famous Source Cod-
ing Theorem and the Noisy Channel Coding Theorem, plus many
other related results about channel capacity.
• S KULLBACK and R A LEIBLER (1951) defined relative entropy
(also called information for discrimination, or K-L Distance.)
• E T JAYNES (since 1957) developed maximum entropy methods
for inference, hypothesis-testing, and decision-making, based on the
physics of statistical mechanics. Others have inquired whether these
principles impose fundamental physical limits to computation itself.
• A N KOLMOGOROV in 1965 proposed that the complexity of a
string of data can be defined by the length of the shortest binary
program for computing the string. Thus the complexity of data
is its minimal description length, and this specifies the ultimate
compressibility of data. The “Kolmogorov complexity” K of a string
is approximately equal to its Shannon entropy H, thereby unifying
the theory of descriptive complexity and information theory.
4
2. Mathematical Foundations; Probability Rules;
Bayes’ Theorem
In either case, it seems sensible that the less probable an event is, the
more information is gained by noting its occurrence. (Surely discovering
that the moon IS made of green cheese would be more “informative”
than merely learning that it is made only of earth-like rocks.)
Probability Rules
Product Rule:
p(A, B) = “joint probability of both A and B”
= p(A|B)p(B)
or equivalently,
= p(B|A)p(A)
Clearly, in case A and B are independent events, they are not condi-
tionalized on each other and so
p(A|B) = p(A)
and p(B|A) = p(B),
in which case their joint probability is simply p(A, B) = p(A)p(B).
6
Sum Rule:
If event A is conditionalized on a number of other events B, then the
total probability of A is the sum of its joint probabilities with all B:
p(A) = p(A, B) = p(A|B)p(B)
X X
B B
From the Product Rule and the symmetry that p(A, B) = p(B, A), it
is clear that p(A|B)p(B) = p(B|A)p(A). Bayes’ Theorem then follows:
Bayes’ Theorem:
p(A|B)p(B)
p(B|A) =
p(A)
The importance of Bayes’ Rule is that it allows us to reverse the condi-
tionalizing of events, and to compute p(B|A) from knowledge of p(A|B),
p(A), and p(B). Often these are expressed as prior and posterior prob-
abilities, or as the conditionalizing of hypotheses upon data.
Worked Example:
Suppose that a dread disease affects 1/1000th of all people. If you ac-
tually have the disease, a test for it is positive 95% of the time, and
negative 5% of the time. If you don’t have the disease, the test is posi-
tive 5% of the time. We wish to know how to interpret test results.
Suppose you test positive for the disease. What is the likelihood that
you actually have it?
From the Sum Rule, we can calculate that the a priori probability
p(D) of testing positive, whatever the truth may actually be, is:
and from Bayes’ Rule, we can conclude that the probability that you
actually have the disease given that you tested positive for it, is much
smaller than you may have thought:
p(D|H)p(H) (.95)(.001)
p(H|D) = = = 0.019 (less than 2%).
p(D) (.051)
This quantity is called the posterior probability because it is computed
after the observation of data; it tells us how likely the hypothesis is,
given what we have observed. (Note: it is an extremely common human
fallacy to confound p(H|D) with p(D|H). In the example given, most
people would react to the positive test result by concluding that the
likelihood that they have the disease is .95, since that is the “hit rate”
of the test. They confound p(D|H) = .95 with p(H|D) = .019, which
is what actually matters.)
A Note on Logarithms:
Let us consider a random variable that takes on only two values, one
with probability p and the other with probability (1 − p). Entropy is a
concave function of this distribution, and equals 0 if p = 0 or p = 1:
11
Example of entropy as average uncertainty:
The various letters of the written English language have the following
relative frequencies (probabilities), in descending order:
E T O A N I R S H D L C ...
.105 .072 .066 .063 .059 .055 .054 .052 .047 .035 .029 .023 ...
If they had been equiprobable, the entropy of the ensemble would
1
have been log2( 26 ) = 4.7 bits. But their non-uniform probabilities im-
ply that, for example, an E is nearly five times more likely than a C;
surely this prior knowledge is a reduction in the uncertainty of this ran-
dom variable. In fact, the distribution of English letters has an entropy
of only 4.0 bits. This means that as few as only four ‘Yes/No’ questions
are needed, in principle, to identify one of the 26 letters of the alphabet;
not five.
Note the important assumption: that the “source statistics” are known!
12
An ensemble is just a random variable X, whose entropy was defined
in Eqt (4). A joint ensemble ‘XY ’ is an ensemble whose outcomes
are ordered pairs x, y with x ∈ {a1, a2, ...aJ } and y ∈ {b1 , b2, ...bK }.
The joint ensemble XY defines a probability distribution p(x, y) over
all possible joint outcomes x, y.
Marginal probability: From the Sum Rule, we can see that the proba-
bility of X taking on a particular value x = ai is the sum of the joint
probabilities of this outcome for X and all possible outcomes for Y :
p(x = ai) = p(x = ai, y)
X
Conditional probability: From the Product Rule, we can easily see that
the conditional probability that x = ai, given that y = bj , is:
p(x = ai, y = bj )
p(x = ai|y = bj ) =
p(y = bj )
p(x, y)
We can simplify this notation to: p(x|y) =
p(y)
p(x, y)
and similarly: p(y|x) =
p(x)
Joint entropy of XY
1
H(X, Y ) = p(x, y) log (5)
X
x,y p(x, y)
(Note that in comparison with Eqt (4), we have replaced the ‘–’ sign in
front by taking the reciprocal of p inside the logarithm).
13
From this definition, it follows that joint entropy is additive if X and Y
are independent random variables:
H(X, Y ) = H(X) + H(Y ) iff p(x, y) = p(x)p(y)
Prove this.
x p(x|y = bj )
If we now consider the above quantity averaged over all possible out-
comes that Y might have, each weighted by its probability p(y), then
we arrive at the...
x,y p(x|y)
This measures the average uncertainty that remains about X, when Y
is known.
14
Chain Rule for Entropy
The joint entropy, conditional entropy, and marginal entropy for two
ensembles X and Y are related by:
H(X, Y ) = H(X) + H(Y |X) = H(Y ) + H(X|Y ) (9)
It should seem natural and intuitive that the joint entropy of a pair of
random variables is the entropy of one plus the conditional entropy of
the other (the uncertainty that it adds once its dependence on the first
one has been discounted by conditionalizing on it). You can derive the
Chain Rule from the earlier definitions of these three entropies.
i=1
Their joint entropy only reaches this upper bound if all of the random
variables are independent.
15
Mutual Information between X and Y
x,y p(x)p(y)
Clearly X says as much about Y as Y says about X. Note that in case
X and Y are independent random variables, then the numerator inside
the logarithm equals the denominator. Then the log term vanishes, and
the mutual information equals zero, as one should expect.
These properties are reflected in three equivalent definitions for the mu-
tual information between X and Y :
I(X; Y ) = H(X) − H(X|Y ) (13)
I(X; Y ) = H(Y ) − H(Y |X) = I(Y ; X) (14)
I(X; Y ) = H(X) + H(Y ) − H(X, Y ) (15)
In a sense the mutual information I(X; Y ) is the intersection between
H(X) and H(Y ), since it represents their statistical dependence. In the
Venn diagram given at the top of page 18, the portion of H(X) that
does not lie within I(X; Y ) is just H(X|Y ). The portion of H(Y ) that
does not lie within I(X; Y ) is just H(Y |X).
16
Distance D(X, Y ) between X and Y
The amount by which the joint entropy of two random variables ex-
ceeds their mutual information is a measure of the “distance” between
them:
D(X, Y ) = H(X, Y ) − I(X; Y ) (16)
Note that this quantity satisfies the standard axioms for a distance:
D(X, Y ) ≥ 0, D(X, X) = 0, D(X, Y ) = D(Y, X), and
D(X, Z) ≤ D(X, Y ) + D(Y, Z).
x q(x)
Note that DKL(pkq) ≥ 0, and in case p(x) = q(x) then their distance
DKL(pkq) = 0, as one might hope. However, this metric is not strictly a
“distance,” since in general it lacks symmetry: DKL(pkq) 6= DKL(qkp).
17
Venn Diagram: Relationship among entropies and mutual information.
Fano’s Inequality
18
!"#
$
%'&)(+*-,.,/(+*-0132547684:9;05*-<&>=?0@4:A=B68&80(+*-21DCE&>CF4:=BGH/054'(/& I,-054J684:9;05*-<&>=K'I=5LM4ON
P =B4;68&80505&80>QSRTA=U!4:A=&8N&>9V2 P =B4:68&80@0XW!I705GCZY[4,T*-0 \&>9;&>=@I25&8<]4:93&>I681D&8<\V&>^*.0
01;4(_9)\:=5I P 1;*.6>IM,.,-G'2@4\&821;&>=`(+*-217I2a(/4052I2@&ZK'I=@L4N P =B4:68&80@0TU!4:=+2b1;&cI, P 1IY&82
d>e Hgf`HghTHi?Hj/kl*m9ng\;A=B&?o>pQ%7&l6>I9q21;&>9054,-N&rUs4:=t21;&T052I2@&T4:686>A P I9;68G A;0@*u9;\
v 4(&8wAI25*-4:9;0ZWx2b1;*.0`&8yIC P ,-&l*-0+2=z*.N;*mI,m^8Q
C, 1/8
A, 1/4
x¢ £"¥¤
¦¥§c
¨ª©]M"¦g
Vg
RU!25&>9«(/&?(+*-017254 &b¬68*.&>9V25,-G=B& P =z&805&>92~21;&05GCEY[4,-0l\&>9;&>=5IM25&8<YVG0@4:CE&?054;A=B68&Q
%'&T01IM,., 684:9;05*-<&>=&>9;684:<*m9;\21;&l0@GCEY4V,.0`I0_Y;*m9I=BG<*-\*-250>Q
o>®
Symbols Source Encoding
encoder
{*-\:A=B&o>¯| iT*-056>=B&825&ECE&>CE4:=zG:,-&8050+0@4:A=B68&lI9;<)&>9;684:<&>=
Ñ÷
%'&T684:9;0@*.<&>=&>I681684;<&T*m9 2bA=59|
oVQ+ø+05*m9;\E21;*-0/&>9;684:<*m9;\gH[(É&n9;<21I2 P =z&805&>92@&8<c(+*-21I~05&8wA;&>9;68&T,.*mL&?o÷÷oH(/&
<49;42`L:9;4((_1;&821;&>=254c<&8684;<&?I0 e f e 4:=ilhTQgÈ+1;*-0lC?IVL&80+0A;681«IE684:<&
A9;0I25*-0BUI68254:=zGQ{A=B21;&>=8H;*m9\V&>9;&>=5I,!H&8N&>9IE684;<&l(`1;&>=B&T0A;681I9ICEY;*-\:A;*-2aG
684:A;,-< Y&l=B&8054V,.NV&8<A9;*-wA;&8,.G«YG,.4;4:LV*m9;\cI2ÉY;*-250tUA=B2b1;&>=ÉIV1;&>I<c*m921;&T0@2=B&>IC
W!I9;< YI68L2b=5I68L*m9;\:^¦*-0A9;0bI25*-0BUI68254:=BGQ
RTY;05&>=BN&`21I2 U!4:=2b1;*.0/684;<&H;21;&_684;<*m9;\?=5I25&HV4:=IN&>=5IM\&_9;ACZY[&>=4UY;*.2@0 P &>=
05GCEY[4,!H*-0_\*-N&>9YVG|
ù Å úbÄ ~ oÔ Ñ+ÔDÑ Ñ
û ü û û ý û
þ
(_1;*-6ã1q*.0,.&80@0_21IV921;&l&>9V2=B4 P GQ
Ñ[Q+È+1;*-0T684:<&E*-0lA9;*-wA;&8,-G'<&8684;<gIY;,-&éUA=B2b1;&>=+21;*-0T684;<&?*-0T*m925&>=z&80525*m9;\*m9«21I2
(É&6>I9<&8684:<&?ÿ 8Ý!Ü Ý!Ü gÚbØ
ªæF2b1I2*-0_9;4?YI68L2b=5I68L*m9;\E*-0t=z&8wA;*m=B&8<
é4:9;68&
(É&?1IN&?2b1;&cY;*-250l4U/21;&?&>9;684;<&8<0@GCEY4V,(/&E6>I9«<&8684;<&c(+*-21;4;A;2l(+I*-25*m9;\
Us4:=`CE4:=z&Ql{A=B21;&>=`21;*-0lI,-054)0I25*-0Bng&80T21; & ÛBÚ
l684:9;<*-25*-4:9H 21IM2_*-0l2b1;&>=B&E*.0
9;4T684:<&`(/4:=B<E(_1;*-6ã1*-0 P =B&bngy Wx*!Q &QV0ICE&+Y;*-2 P I2525&>=@9^ 4UI`,-4:9;\&>=¦684;<&+(/4:=B<
Q
921;*-0`6>I05&T21;&~684:<*m9;\=5I25&`*-0_&8wAI,
25421;&T&>92b=B4 P GQ
õ[Q/% 1;*-,-&E21;*-0_*-0`<&b684:<gIVY;,.&«W!I9;< 684;<*m9;\=@I25&l*-0`&8wAIM,
25421;&l&>9V2=B4 P GI\:I*m9^8H
4:Y;05&>=BN&T2b1I2+*-2_<4;&80T9;42_1IN&l2b1;& P =B&bngy P =B4 P &>=B2BGI9;< *-0T9;42IV9*u9;0@2I9V2I
9;&84:A;0+684;<&Q
ç 1I99;4:9 0/n=B052+2b1;&84:=B&>CS*-0`21;&ãØ:ÛB×bÚa×bØ>ÙVÿcÝ:ÚbØÛBÚ#(`1;*.681*-0>|
{g4:=I'<*-056>=z&825&CE&>CE4;=BG:,-&8050054:A=z68&(*.2b1n9;*-25&7&>92=z4 P GDJéU!4:=cIV9G
W P 40@*.2@*.NV&>
^ `*-2T*-0 P 40@05*mY;,.&254c&>9;684;<&?21;&E05GCEY4,-0TI2TI9«I>NV&>=5I\&l=5IM25&
ÇEH:0bA;6ã1q21I2>|
Ç]SÔ
xW {4:= P =B4:4MU05&8& ç 1I99;4:9 %'&>IN&>=8Q ^EÈ+1;*-0+*-0TI,-054?054:CF&825*mCE&80_6>IM,.,-&8<)21;&Z9;4*-05&8,-&8050
684;<*u9;\E21;&84;=B&>CËI0¦*-2É<&>IM,.0/(+*-21684;<*u9;\E(+*-21;4;A;2684:9;05*-<&>=5IM25*-4:94U9;4*-05& P =B4:68&80@05&80
Wx*!Q &QgY;*-2`684:=5=5A P 2@*.4;9&8256>^8Q
È+1;&/&>9V2=B4 P GTUA9;6825*-4:9E21;&>9=B& P =B&805&>9V250IUA9;<gICE&>9V2I,;,.*mCE*-2/4:9~21;&+9;ACZY[&>=4U
Y;*-250
4:9IN&>=5IM\&=z&8wA;*m=B&8<)254c=B& P =B&80@&>92+2b1;&T05GCEY[4,-0_4U21;&T054;A=B68&Q
.
o
2 Þo
1 0 Ñ
3
Ño
x ' 65
$7 g§c"¦¦87
4
X Channel Y
{*-\:A=B&?o>®[|h/4;<*u9;\I9;<<&8684;<*m9;\4U05GCEY4,-0Us4:=¦2=@I9;0BU!&>=/4N&>=/IE6ã1IV99;&8,sQ
%'&684:9;0@*.<&>=I9J*m9 P A;2I, P 1IY&829 3d :>>5; :8< kI9;<4:A;2 P A;2cI, P 1IY&82>=Ä
d ?V>>>5@ ?,A klI9;<=@I9;<4:CìNOI=z*uIVY;,.&80 Õ I9;<>BË(_1;*-681=5I9;\&`4N&>=2b1;&805&IM, P 1IVY&8250>Q
3
ÑÑ
Ð&8y:2(/&+2IVL&21;&4:A;2 P A;24UI`<*-056>=B&825&`CE&>CE4:=BG;,-&80500@4:A=B68&+I021;&+*m9 P A;2/254TIT681I9V
9;&8,!Q ç 4l(/&_1IN&`I05054;68*mI25&8<?(+*-21F21;&+*m9 P A;2+I, P 1IY&82/4U
21;&+681I99;&8,21;& P =z4:YIY;*-,â
*-2BG?<*-052=B*mYA;25*-4:9q4U4:A;2 P A;2¦U=B4:CSIECE&>CE4:=zG:,-&8050/054:A=z68& d W : ^ l#o Ñ >8 êk;Q%'&
21;&>974:Y;2I*m9«2b1;X & LB4*m92 P =B4:YIY;*-,-*-2BG<*-052=z*uYA;2@*.4;9J4U+21;&c=5I9;<4;CáNOIV=B*mIY;,-&80 Õ I9;<
Bc|
WB ?3FH Õ : ^
]\ F 0 _^ FG0 `
A <
Ñõ
X Y
0.9
0 0
0.1
1-p 1 1
0 0
1.0
2 2
p 1.0
3 3
X Y
p
1 1
1.0 999999
1-p 999999
n 5 ª7_
ëg
¨#
po$
jy;25&>9;<*m9;\21;&_*-<&>I0`4U*m9VU!4:=5C?IM25*-4:9I9;<&>9V2=B4 P GEU!4:=¦21;&T<*-056>=B&825&~054:A=B68&H;(/&T6>I9
9;4( 684;9;05*-<&>=`21;&l*m9VU!4:=5C?I2@*.4;9IVY4:A;2 Õ 4:Y;2I*m9;&8<'YG4;Y;05&>=BN;*u9;\2b1;&ZNI,mA;&E4U21;&
BcQª%'&l<&bn9;&~21;&T&>92b=B4 P GIU!25&>=¦4:Y;05&>=zN:*m9;> \ B ?GF |
<
A <
Ñ
X Source Y X
Decoder Correction
encoder
Observer
Correction data
{*.\;A=B&lÑo|ht4:=5=B&86825*-4:90@G:052@&>C
oVQ qW 9)é_=+^OzD÷Q
È
401;4(#21;*-0>H(/&?9;425&E21I2/W : HJ?GF ^! W ?GF ^+ëW : ;?GF ^TI9;<«0AY;0525*-2A;2@&?21;*-0
*u9q21;&`&8wAI2@*.4;9?IY;4N;&|
W : ;?GF ^
x A <
z ÷ Wx :®^
YGA;05&T4U21;&T*m9;&8wAIM,.*-2BG (/&T&8052bIY;,-*.0b1;&8< P =z&8N:*-4:A;05,-GQ
Ñ¡
x
W ?GFKHT: ^
< A
W : @?GF ^,-4\>}
x 0 F 0 W ?GF ^~
W=Eé;9E^ W!¡÷^
Q+È+1;& P =B&868&8<*m9;\«,-&>I<0l2542b1;&?4:YVN:*-4:A;0~05GCCE&82=B*-6?<&bn9;*-25*-4:9 Us4;=`21;&cCEA;2AI,
*u9VU!4:=5CI25*-4:9*m925&>=@CE0+4Ux 2b1;&l&>9V2=B4 P G?4U21;&T4:A;2 P A;2|
W 9)_é =+^ëXW=^ 'W= H 9?^
q
Ña
I(X;Y)
1
1
I(X;Y)
0.9
0.8
0.5
0.7
0.6
0.5
0
0.4
0.9
0.3 0.8
0.7
0.2 0.6
0.1 0.2 0.5
0.3 0.4 0.4 P(X=0)
0.1
0.5 0.6 0.3
0.2
0 P(transition) 0.7 0.8 0.1
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7
P(transition)
0.8 0.9 1 0.9
§c"¦¦87Tvo¦!8$
§c"¦¦87T
¨ª©
Ñp
054:A=z68&80É(`1;*.681(É&`I=B&`*u9V25&>=B&80@25&8<c*m9&>9;684:<*m9;\Us4;=2=5IV9;0CE*-0505*-4:9;0/1I>NV&Ir05*-\:9;*âng6>I92
ICE4;A92`4Ut=z&8<gA9;<gI9;68GI,m=B&>I<GQEh/4:9;0@*.<&>=r05&>9;<*m9;\I P *-&868&4UÉ05G9V2I6825*-6>I,-,-G«684:=
=B&8682_IV9;<c05&>CI92@*.6>IM,.,-G CE&>IV9;*u9;\MUA;,j¦9;\,-*-014:=t684:C P A;25&>= P =B4\;=5ICì25&8y;2É2b1=B4:A;\:1I
681I99;&8,g(_1;*-681?=5I9;<4;CE,-GZ684;=5=5A P 25&8<?4;9ZIN&>=@I\&To/*m9o>÷`681I=5I6825&>=z0_Wx0A;681?I0CE*-\:1V2
Y&~*m92=z4:<gA;68&8<«YG2=5I9;0bCE*-0505*-4:9 IM6>=B4050+I?=5IM21;&>=/05*-68L,-G È&8,.&8y0@G:052@&>C?^8Q &Q \gQm|
oVQ+fÉ=z*u9;\=B&8*m9VUs4:=z68&>CE&>92@0>Hg(/& =z&_\4V*u9;\254cI<NI9;68&
Ñ[Q+!2 0/&>I0@Gc254=B&8684\:9;*-05&~0 P 8& &8681
oVQ+fÉ=z*j8\=B&8*m9VUs4:=z68&l&>9V250>H(/& =B&`\4*m9;\254?I<NOIV9;68&
Ñ[Q+!2 0/&>I0@GCE4=B&8684\:9;*-05&~0 P &8&8681
RT9;&«4UT21;& 05*mC P ,-&8052)684:<&80*-0 IDÛBÚÚ>Ý!ÿxÝ!ÿxØ]×ØÙÚ>Q]%'&)2IL& IJY;*m9I=BG305GCCE&82=B*-6
681I99;&8,l(+*-21ëIJ2=5I9;05*-25*-4:9 P =B4:YIY;*-,-*-2aG
é~21;*-0 \V*.NV&80IJ681I99;&8,T6>I P I68*-2BG
o/Ô,-4\W-g^ÔWzo_«g^[,.4V\[Wzo_«g^8Q~È+1;&?9I2A=5I,Y;*m9I=BG«&>9;684;<*u9;\qUs4;=`21;*-0T*.0~21;&>9
d ÷ ok¦æZ21;&T=B& P &82@*.2@*.4;9?684;<&_(*.,-,
=B& P &>I221;&80@&_<*-\*-250+I9E4;<<9:ACEY&>=ª4U
25*mCE&80+I9;<
P &>=aUs4:=@C C?GI LB4:=B*-2BG?N4V25*m9;\gQ
Æ&>9;68&+(/&+2=5I9;0bCE*-2 (Ñ Ô«o+Y;*.2@0 P &>= 0@GCEY4V,sH;I9;<?(+*-,-,g4;Y;2I*m9?I9~&>=5=B4;= * ZU Ô«o
4:=+CF4:=B&TY;*.2@0I=z&l=B&868&8*-N&8< *m9&>=@=B4:=8H;21I2*.0|
¢¡
s
Ñ #Ô]o Wzo+g^ % ¡ l
\ 1 0 ¢¡
Z
£ t
ѯ
1 0.1
0.97
0.96
Capacity
1e-05
0.95
1e-06
0.94
1e-07
0.93
1e-08
0.92
0.91 1e-09
0.9 1e-10
1e-08 1e-07 1e-06 1e-05 0.0001 0.001 0.01 0.1 0.01 0.1 1
P(transition) Code rate
¤ §c"¦¦87~§c
¨ª©À£"
g
Ñ®
úú úþúü ú©bú i
ú
úª ú
m
úþúü ú©bú ú
ú úª
i
úþúü ú©bú i
ú m
o P
o o o U
p+Ô ¯gW -, 4\ ^
p R
¯ ¯ Ã W
o ¯ o o
} p+ÔDÃW ,-4\ ^
p ¯ ¯ Ã~
p
È+1;&«6>I P I68*-2BGD4MUl21;&«6ã1I99;&8,T*-0 :öVp'*m9VU!4:=5C?I2@*.4;9DY;*-250 P &>=Y;*m9I=BGD<*-\*-2)4Ul21;&
681I99;&8,684;<*m9;\gQ?h+I9«(/&~n9;<ICE&8681I9;*-0Cè254&>9;684:<& *u9VU!4:=5CI25*-4:9'Y;*-250~*u97p
681I99;&8,Y;*-250`0AIY LB&8682+254?21;&T&>=@=B4:= P =B4 P &>=B2BGc<&8056>=z*uY[&8<'IY4N& ¨
È+1;&W!p :^Æ_ICCE*m9;\h/4;<& P =B4N:*-<&80T I ,ãÝ!,Ú EÜÝ!ÿ ×`684;<&l254 P &>=aUs4:=@C 21;*-0æqI05G;0B
25&>C?IM25*-6+684:<&`*-0/4:9;&+*m9?(`1;*.68121;&+4;YN;*-4:A;0ÉY;*m9I=BG&>9;684:<*m9;\?4U
21;&054:A=B68&`05GCZY[4,-0
*-0 P =B&805&>9V2/*m921;&_681I99;&8,&>9;684;<&8<U!4:=5CQ{g4:=¦4:A=¦054:A=z68&_(_1;*-6>1&>CE*-250+I2¦&>I68125*mCE&
*m92@&>=BNOI,/o~4Ut,o aE05GCZY[4,-0>H(/&T2IL&T2b1;&ZY;*m9I=BG«=B& P =B&805&>9V2I25*-4:9q4U21;*-0lI9;< 684 P G*-2
254Y;*-250lú þ H
ú ©H
ú i IV9;<ú m 4MUÉ21;&~&>9;684:<&8<Y;,-4;6ãLé
2b1;&F=z&>C?I*m9;*m9;\ Y;*.2@0ZI=B&~\*-N&>9«YG
ú ü ú ú HgIV9;¬< GgÙVÛaØ ZwÚ tYVG Å ü Å bÅ |
ú ü ú ©eú i 3ú m IV9;<
H
Å ü ú ü ú ©Z3ú i 3ú m
ú ú þ ú i 3ú m IV9;<
H
Å ú ú þ 3ú i 3ú m
ú ú þ ú ©Z3ú m IV9;<
H
ÅM ú ú þ 3ú ©Z3ú m
RT9=B&868& P 25*-4:9* U 21;&_Y;*m9I=BGc9;ACEY&>= Å ü Å8Å ë÷T21;&>9E2b1;&>=B&_*-0/9;4Z&>=@=B4:=8H&8,-05&lú®¯®°_®%±
*-0_2b1;&lY;*.2`*m9&>=5=z4:=8Q
È+1;*-0 Æ`IC?CE*m9;\J684:<&'A;05&80 õJY;*-250c2@4684:=@=B&8682p]W! Ñ þ #o>^E&>=5=z4:= P IM2525&>=59;0 I9;<
2=5IV9;0BUs&>=ª ZA;05&bUA;,
Y;*-250>Q9\&>9;&>=5 I,IEÆ_ICCE*m9;\l684;<&A;05&8 0 óY;*-250254~684:=5=B&8682/Ñ «o
õ÷
È+1;&lÆ`IC?CE*m9;\?684;<&80+&8y;*-052/U!4:=/I,-, P I*m=B0ZW!Ñ Jo Ñ l ^ÉIV9;<<&825&8682+4:9;&lY;*-2+&>=5=z4:=B0>Q
e ,.0@421;&«}T4,mIG684;<& *-0 IDW!Ñõ o>Ñ^~Y;,-4:68L684;<&«(_1;*-6ã1¥684:=5=B&868250A P 2@4'21=B&8&«Y;*.2
&>=5=B4;=B0>HI9A99ICE&8<)684:<&~&8y:*-052@0_I2ZW!®V÷ p¯V^(_1;*-681684:=5=z&868250+A P 254?2B(/4ZY;*-2`&>=5=B4:=z0>Q
1
Residual error rate
1e-10
1e-20
1e-08 1e-07 1e-06 1e-05 0.0001 0.001 0.01 0.1
Mean error rate
{*-\:A=B&EÑ g|+W!pM :^/Æ_ICCE*m9;\?684:<&E=B&805*-<gAI, &>=5=B4:=/=5IM25&
%'&~6>I921;&>9684;9;05*-<&>=`21;&ZCE4;=B&T\&>9;&>=5I,6>I05&H(_1;&>=z&l(/&l1IN&l=5I9;<4:CóY;*-2_&>=@=B4:=B0
A9;*âUs4;=5CE,-G?<*-052=B*mYA;25&8<'Wx*!Q &Q(/&_I,-,-4(D21;& P 4050@*uY;*-,-*-2BG?4U2B(/4T4:=CE4;=B&+Y;*.2¦&>=5=B4:=z0 P &>=
plY;*-2+Y;,-4:68L;^8Q e \:I*m9E(/&_4;Y;2I*m9E21;& P =B4:YIY;*-,-*-2BG?4U=B&805*-<gAI,&>=5=B4;=I0¦21;&_=B&>C?IM*u9;<&>=
4U21;&lY;*m9;4:CE*mI, 05&>=B*-&80>|
s p £ WBo+g^{m@l
m
\ 1 0 t
´ µ¶¸·¹vº@·#»X¶¸»¸¼º·¸½,¶¾]¿ÁÀ¹vº_¶¸·
õo
ø05*m9;\c4:A=ÉUs4:=@CZA;,mI~U!4:=¦<*-056>=B&825&~&>92b=B4 P GI9;<2bIL*m9;\?2b1;&T,.*mCE*-2>|
o
qW 9^ ,-*uC k W :/F _^ Ä : ,-4\ } :/F :
Æ Ç F W _^ Ä ~
o
,-*uC k#È W :/F ^[,-4\ } :/F Ä : ',-4\ WqÄ : ^ W : ^_Ä :IÉ
Æ Ç F
W ^~ F
o
ÊÌË W : ^,-4\ } : : }~,.*mC k ,-4\ qW Ä : ^ û Ê²Ë W : ^ :
l
W ^ , ~DÍ Æ Ç ~
l
Í
È+1;*-0l*-0l=5IM21;&>=`(/4:=5=BG;*m9;\I0T2b1;&Z,mI2@25&>=`,-*uCF*.2~<4:&80l9;4V2T&8y:*-052>QEÆ4O(/&8N&>=ÏHI0T(/&EI=B&
4U!25&>9«*m9V25&>=B&80525&8<7*u9721;&<*Ï&>=z&>9;68&80FY[&82B(É&8&>9 &>92=z4 P *-&80qWs*sQ &Q
*m9'2b1;&?684:9;05*-<&>=5I2@*.4;9
4UÉCEA;2AI,&>92=z4 P G4;=+6>I P IM68*.2BG^8H(É&~<&bn9;&?21;& P =B4:Y;,-&>CáI(tIGYG«A;05*m9;\21;&Tn=z052
25&>=5C 4:9;,-GI0`I?CE&>I0A=z&T4UtÙVÿ Ð_ÚÛBÚ gÝ!ÿxÜ Ú Ý!ÛB@Ø v |
o
ÎqW 9^Ñ
Ê²Ë W : ^,-4\ } : : W!¡õ^
l
W ^ ~>Í
Ë
%'&E6>I9«&8y;25&>9;<«21;*-0l2@4 I?684:9V25*m9;A;4:A;0Z=5IV9;<4:CóN&868254:=4U/<*uCF&>9;05*-4:Ò 9 (7684:9;68&>I,-*m9;\
21;X
& ([!U!4,-< *m92@&8\:=5I,Y[&>1;*m9;<NV&868254:=+9;42bI25*-4:9I9;<Y[4,-< 2BG P &|
o
ÎqW Ó^ Ê²Ë W ¶ ^,-4\ } W!¡ :^
l
W ¶ ^ ~ Í ¶
Ë
%'&~6>I921;&>9 IM,.0@4?<&bn9;&E21;&La4V*u9V2TI9;<684:9;<*-25*-4:9I,&>9V2=B4 P *-&80/U!4:=/684;925*m9;A;4:A;0`<*.0z
2=B*mYA;25*-4:9;0|
o
Îq W Ó {Ô ^à ÊÕÊ3W ¶ c ^,-4\ }
W ¶ c G^ ~ Í ¶ Í c
W c ^
Î WqÓ H Ô ^à ÊÕÊ3W ¶ c ^,-4\ }
W ¶ c G^ ~DÍ ¶ Í c
W ¶ ^
Î W ÔÖH Óc^à W ¶ c ^,-4\ }
W ¶ c G^ ~ Í ¶ Í c
ÊÕÊ
(+*-21|
W : ^à DW ¶ c ^
Ê c
Í
õÑ
×
n ?V
po+gMM g
Ø
(`1;&>=B&|eÚ Ê : W : ^ :
Í
W!¡¡^
È 1;*-0É4 P 25*mCE*
j>I25*-4:9 P =B4:Y;,-&>Cì*-0/054,-N&8< A;05*m9;\)Ü
I\:=@I9;\&_CEA;,-25* P ,-*.&>=z0I9;<C?IyV
+
*uCF*j8*m9;\g|
ÊÁÝ> W : ^[,.4V\
W : ^Ô-Þ W : ^>W : ßÚ^
Ô-Þ W : ^%à :
Í
(_1;*-6ã1q*.04:Y;2I*m9;&8<YVG054,-N:*m9;\g|
To+',-4\ W : ^Ô-Þ W : ¬Ú^ ÔÞ ÷
054E21I2+(+*-21<gA;&E=B&8\:I=z<Us4:=t21;&T684:9;052=@I*m92504:921;&T05G;0525&>C|
W : ^ á o æ°
°;å %
l
l/ä
Ñ âpÛã
1;&>9;68&|
o
ÎW Õ ^ ,.4V\W!Ñ â Û ^ W!¡a^
Ñ ã
õõ
n E*é7 g
è
921;&+6>IM05&_4MU684;925*m9;A;4:A;00@*.\;9I,-0tIV9;<c681I99;&8,-0/(É&TI=B&*u9V25&>=B&80@25&8<?*m9UA9;682@*.4;9;0rWs4U
0IGE25*mCE&>^8HV6ã1;40@&>9EU=B4:C#Ir05&824U P 40505*mY;,-&`UA9;682@*.4;9;0>HI0*m9 P A;2/254l4;A=681I99;&8,!H(+*-21
054:CF& P &>=B2A=5Y[&8<NV&>=B05*-4:9«4U/21;&EUA9;6825*-4:9;0?Y&8*m9;\ <&82@&86825&8<XIM2T21;&?4:A;2 P A;2Q?È+1;&805&
UA9;6825*-4:9;0+21;&>92IVL&_2b1;& P ,mI68&~4U21;&T*m9 P A;2_I9;<4;A;2 P A;2+I, P 1IY&82504U2b1;&_<*-056>=B&82@&
6>I05&Q
%'&CEA;052ZI,-054«21;&>97*m9;68,mA;<& 21;& P =B4:YIY;*-,-*-2BG4U_I)\*-N&>9 UA9;682@*.4;9Y&8*m9;\'*m9,La&8682@&8<
*m92@4Z2b1;&T6ã1I99;&8,(_1;*-6ã1Y=B*m9;\0`A;0É2@4Z2b1;&T*.<&>IF4UÉÚêãÚ,)ë-Ú æ?21;*-0+\&>9;&>=5IM,IV=B&>IT4U
(/4:=5LE*-0TL:9;4(_9IMX 0 EÚGÜ ;ÛaÚEÝ :ÚbØ@Û Q
{g4;=/&8yIVC P ,-&H684:9;05*-<&>=21;&l&>9;05&>CEY;,-&80>|
oVQ ìGí
Wîa^]0@*u9 WîÔï^
xU?(/&I,-054D684;9;05*-<&>=)UA9;682@*.4;9;0,-*mCE*-25&8<#254]25*mCE&7*u9V25&>=BNI,òlH~21;&>9(/&'4;YV
2I*m9J4:9;,-GXÑròð 9;4;9V'j8&>=B4'684;&b¬c68*-&>9250I9;<D(/&6>I9J684:9;05*-<&>=21;&&>9;05&>CEY;,-&
254?Y&Z=z& P =B&805&>9V25&8<YVGIV9(<*mCE&>9;0@*.4;9I,_qW (GÑ òð^ P =B4:YIVY;*.,-*-2BG<*-052=B*mYA;25*-4:9
W :8;:>>5;:[ ^ÏQ
Q/K4;=B&?0 P &868* ng6>IM,.,-G;H+*âU_(/&684:9;05*-<&>=E21;&&>9;05&>CEY;,-& 4U`,.*mCE*-25&8< P 4(/&>=W!YG ^8H
YI9;<;,-*mCE*-25&8<DWx254 Å ð ^I9;<c2@*uCF&b,.*mCE*-25&8<)05*-\:9I,-0ZW!9;4;9V' j8&>=B4?4:9;,-G*u9*m9V25&>=BNI,
W!÷ ò`^5^8H¦(É&)n9;<]21I22b1;&&>9;0@&>CZY;,-&'*-0«=B& P =B&805&>9V25&8<YG]I 9 ([<*uCF&>9;05*-4:9I,
=P B4:YIY;*-,-*-2aG3<*-052=B*mYA;25*-4:9ë(_1;*-6ã13*-
0 j8&>=B44:A;2@05*-<&2b1;ó
& ([0 P 1;&>=z& =@I<*mA;Ì
0 ô
á Ñð Q
õ
f¦G684;9;05*-<&>=B*m9;\21;&,mI2525&>=~2BG P &80?4U`&>9;05&>CEY;,.&80H(/&6>I9 ng2?21;&>Cå*m92@42b1;&n9;*.2@&
<*mCE&>9;05*-4:9I, 684:9V25*m9:A;4;A;0_<*Ï
&>=B&>9V25*mI,
&>9V2=B4 P Gc<&bn9;*-25*-4:9;0T\V*.NV&>9*m905&8682@*.4;9¡Q
x¢
n §c"¦¦87~§cvo¦!8$
%'&+684;9;05*-<&>=/6ã1IV99;&8,.0/*m9(_1;*-681c9;4*-05&`*-0/*u9,LB&86825&8<*m9;<& P &>9;<&>9V25,-G4U
21;&+0@*.\;9I,!é21;&
P I=B2@*.6>A;,mI=¦9;4*-05&+054;A=B68&+4U*u9V25&>=B&80@2*-021;&054
6>I,-,-&8<ÜÙVÙÿ Ý!ÿõ>Ú#ö];ÿ Ý!Úy÷+Ü@8ÿ Ü>gØVÿ18Ú>Q
{A=z21;&>=/(/&l=B&8052=z*.682684:9;05*-<&>=5I25*-4:9;0`254?2b1;&`n9I,
68,mI050`4U&>9;05&>CEY;,-&Q
%'&'1IN&I705*-\:9I,T(*.2b1ÀIN&>=@I\& P 4(/&>= H+2@*uCF&bF,-*mCE*-25&8<2@ 4 ò I9;<YI9;<(*.<2b1
,-*mCE*-25&8<«25D 4 ðìQ
%'&)21;&>93684:9;05*-<&>=21;Ì & (#Ñ ðñòó0IC P ,-&80W D Õ F I9;- < B F ^E21IM2c6>I93Y&'A;05&8<]254
681I=5I6825&>=z*.0@&lY4212b1;&T*u9 P A;2TI9;<4:A;2 P A;2É0@*.\;9I,-0>QÉÈ1;&>921;&l=B&8,mI25*-4:9;01;* P Y&82B(/&8&>9
21;&T*m9 P A;2TI9;< 4:A;2 P A;2+*-0_\*-N&>9YVG|
B F Õ F ÔÃ F H/Â?#o Ñ >8 (
D
Æ&>9;68&|
£W Õ é B^Ñ
{ Î W B?^6 ÎWÃ^
È+1;&T6>I P IM68*.2BG4U21;&T681I99;&8,
*-0+2b1;&>94:Y;2I*m9;&8< YVGC?IMy:*mCE
* j8*m9;\c2b1;*.04ONV&>=ÉIM,.,*u9 P A;2
<*-052=B*mYA;25*-4:9;0~æ 21;*-0E*-0ZI681;*-&8N&8<YVGC?IMy:*mCE
* j8*m9;\'(+*-21'=z&80 P &8682~25421;&E<*-052=B*mYA;25*-4:9
4U Õ 0AIY LB&8682+25421;&lIN&>=5I\& P 4(/&>=/,-*mCE*-2I25*-4:9|
ø MÕ F
N
e 0_(/&Z1IN&T0@&8&>9 2b1;*.0`*-0IM6ã1;*-&8N&8<«(_1;&>9)(/&l1IN&ZI?}~IA;0505*mI9q<*-052=B*mYA;25*-4:9H 1;&>9;68&
(/&Tn9;<'Y4V21 Õ I9;² < B CEA;052_1IN&Z}~IA;0505*mI9q<*-052=B*mYA;25*-4:9;0>QT 9 P IV=B25*-6>A;,mI= Õ 1I0
NOIV=B*mI9;68& I9; < B 1I0+NOIV=B*mI9;68& ÔÃ k ðìQ
%'&T6>I9q21;&>9&8NOIM,uAIM25&l21;&T681I99;&8, 6>I P I68*-2BGHgI0>|
o
Î W B?^à ,-4\/Ñ â W ÔÃ k ð ^
Ñ ã
o
ÎW!ë^à ,-4\/Ñ â W!à k ð^
Ñ ã
o
,-4\>}goÔ k W!¡p^
Ñ Ã ð ~
õ¡
È+1;*-0/6>I P I68*-2BG?*-0*m9Y;*-250 P &>=ª6ã1I99;&8,05GCEY4,!H;054~(É&6>I9E4:Y;2IM*u9Il=@I25& P &>=0@&8684:9;<
H
YVGCEA;,-25* P ,-*.6>IM25*-4:9'YVGu(öGòlH[*sQ &Q;U=z4:CÃ(ÑðñòlHgCEA;,-25* P ,.*-6>I2@*.4;9YVG Ñðì|
à k í \
õa
1. Foundations: Probability, Uncertainty, and Information
2. Entropies Defined, and Why they are Measures of Information
3. Source Coding Theorem; Prefix, Variable-, and Fixed-Length Codes
4. Channel Types, Properties, Noise, and Channel Capacity
5. Continuous Information; Density; Noisy Channel Coding Theorem
6. Fourier Series, Convergence, Orthogonal Representation
7. Useful Fourier Theorems; Transform Pairs; Sampling; Aliasing
8. Discrete Fourier Transform. Fast Fourier Transform Algorithms
9. The Quantized Degrees-of-Freedom in a Continuous Signal
10. Gabor-Heisenberg-Weyl Uncertainty Relation. Optimal “Logons”
11. Kolmogorov Complexity and Minimal Description Length.
37
Introduction to Fourier Analysis, Synthesis, and Transforms
It has been said that the most remarkable and far-reaching relationship in all of mathemat-
ics is the simple Euler Relation,
eiπ + 1 = 0 (1)
which contains the five most important mathematical constants, as well as harmonic analysis.
This simple equation unifies the four main branches of mathematics: {0,1} represent arithmetic,
π represents geometry, i represents algebra, and e = 2.718... represents analysis, since one way
to define e is to compute the limit of (1 + n1 )n as n → ∞.
Fourier analysis is about the representation of functions (or of data, signals, systems, ...) in
terms of such complex exponentials. (Almost) any function f (x) can be represented perfectly as
a linear combination of basis functions:
X
f (x) = ck Ψk (x) (2)
k
where many possible choices are available for the expansion basis functions Ψk (x). In the case
of Fourier expansions in one dimension, the basis functions are the complex exponentials:
Ψk (x) = exp(iµk x) (3)
√
where the complex constant i = −1. A complex exponential contains both a real part and an
imaginary part, both of which are simple (real-valued) harmonic functions:
exp(iθ) = cos(θ) + i sin(θ) (4)
which you can easily confirm by using the power-series definitions for the transcendental functions
exp, cos, and sin:
θ θ2 θ3 θn
exp(θ) = 1 + + + + ··· + + ···, (5)
1! 2! 3! n!
θ2 θ4 θ6
cos(θ) = 1 − + − + ···, (6)
2! 4! 6!
θ3 θ5 θ7
sin(θ) = θ − + − + ···, (7)
3! 5! 7!
Fourier Analysis computes the complex coefficients ck that yield an expansion of some function
f (x) in terms of complex exponentials:
X
n
f (x) = ck exp(iµk x) (8)
k=−n
where the parameter µk corresponds to frequency and n specifies the number of terms (which
may be finite or infinite) used in the expansion.
Each Fourier coefficient ck in f (x) is computed as the (“inner product”) projection of the function
f (x) onto one complex exponential exp(−iµk x) associated with that coefficient:
1 Z +T /2
ck = f (x) exp(−iµk x)dx (9)
T −T /2
where the integral is taken over one period (T ) of the function if it is periodic, or from −∞ to
+∞ if it is aperiodic. (An aperiodic function is regarded as a periodic one whose period is ∞).
For periodic functions the frequencies µk used are just all multiples of the repetition frequency;
for aperiodic functions, all frequencies must be used. Note that the computed Fourier coefficients
ck are complex-valued.
38
ÿ
"!$#&%('*),+-"-.+/10,23/4065&+7)98,+-:'*!&),'<;=>5,#&;7?('*!&#&%A@
BDCFE7G4'DHI+JHK=D!$-K% !$LM+ONP/J#&)PQRCG,@
BDC"S9N&E
T
@
BDCUEVHWX'<?ZY&!$5&?0*![% %O!1=]\J+#&+-(/40<'*?.^`_"+?Z/Ja4+NTcbJd
H
We+!$f&%(+-.2J+?ZY&+O!&-.?ZY&!J\J!&#,/40<'*?.^g8,-:!$8,+-.?('*+7%!4=% '6#B>hCUE"/J#&);7!J%3B>iRCFE
=j!$-
'6#J? +7\[+-.%Oh
/J#&)9iKk
lnm:o ;7!J%3BDiCUEq;7!J%BDhCUE(r$C T s b[d 't=
huTciTcv
p x
d z
w |
y { !J?ZY&+-:_}'<% +
l m:o % '6#BDiCUEq% '6#BDhCUE(r$C T s v 't=
huTciTcv
p x
d w |
y { !J?ZY&+-:_}'<% +
l m:o % '6#BDiCUEq;7!J%BDhCUE(r$C T v QRh~Zi B.E
p
We+5&% +?ZY&+-.!$#&+7;7a4+-}w=5,#&;7? '<!$#?(!gLP+/J#k
wzy|{PT s v!Jh
?ZY&Tc+-:i _}'<% +
Y&+#?ZY&+U!$5,-:'*+-&+-.'<+7%=j!$-"@
BDCFE"'<%k
N p S B>NJ{;7!J%3BDiCUExSz{|% '6#B>iRCFE E BDbJE
b {3x
_OY&+-.+?zY&+U!&5,-.'<+-}"!&+zg;7'<+#J? %/J-.+Jk
N[{T d l m.o @
BDCUEq;7!J%BDiCUE r&C iv BDJE
p
z{T d l9m:o @
BDCUEq% '6#BDiCUE r&C ic B$E
p
We+PY&!$8,+?ZY,/4?}?ZY&+F!$5,-.'<+-&+-.'<+7%8,-.!42$'<),+7%/J#`+78,/[#&% '<!$#!4=]?zY&+=5,#&;7? '<!$#9@
BDCFE"'6#
? +- LM%}!4=;7!J%('#&+P/J#&)9%('#&+? +- LP%H
J¸
_OY&+-.+¹N { /J#&)º { /J-.+P?ZY&+PU!$5,-:'*+-;7!&+zg;7'<+#[? %H¹"!$#&%('*),+-?ZY&+P'6#J? +7\&- /40
\J'<2&'#&\?ZY&+
),'<% ;-.+8,/J#&;7^`f,+7?._"+7+#¹@
BDCUE/J#&)g² { ³ B>CUE}BD/4%(%Z5,LP'6#&\@
BDCFEK'*%_"+70*0f,+Y,/2J+7)¹+#&!&5&\$YP=D!$-
?ZY&+'6#[? +7\$- /40x? !P+7$'<% ?ZEVk
l m.o @
BDCFE
¼e² { ³ BDCUE½ m r$C
p »
_OY&'<;7Y% '6L¹8&0<'t¾U+7%? !Fk
l m.o @
BDCFEzÀ m r$C
p ¿
´ ¶
¼ b[dN pm ¼Ád BDN {m S {m E
{4x
´¶
S b[d|BDN p ¼eN p E m Sd BDN ³{ ¼ÁN[{$E m ScBD ³{ ¼ez{$E m4Ã
³ BDÄJE
{3x"Â
Å !J? +P?ZY,/4??ZY&+¹? +-(LP%O'6#[2J!J0<2&'#&\eN[³]/[#&)Æz³
/J-.+P/40<0OÇv/J#&)23/J#&'<%ZYÈ_OY&+#eN[{³ TÉNJ{
/J#&)e ³{ TÊz{FH9ËKÌ$ÍÏÎÐJÑ$ÒzÓÍÒÔRÍÒZÓÍ7ÕPÓ6Õ¹ÖÌ$Í× ÍVÕØÖÙzÚJÚRÒ.Ð4Û4ÓÝÜPÙJÖDÓÐJÞ,ßÓÝÞeÖDÍÒZÜÕ¹Ð.à¹ÜPÍZÙ[Þ
Õ7áÑ&ÙJÒ.ÍZâ`ÍÒzÒ.ÐJÒ ß"ÖDÐ@ãÖÌ$ÙJÖäZÙ[ÞÆ×(ÍPÙJä7Ì&ÓÝÍåÍzâgÑ[ÕØÓÞJæÖÌ$ÍVÕØÍPäÓÝÒ.äÑ&ç<ÙJÒKà7Ñ$ÞFäÖDÓÐJÞ,Õè
$v
1.5
1 fseries( 1,x)
fseries( 7,x)
fseries(63,x)
rect(x)
0.5
-0.5
-1
-1.5
-1 0 1 2 3 4 5 6 7
1.5
1 fseries( 63,x)
fseries(163,x)
rect(x)
0.5
-0.5
-1
-1.5
-0.2 -0.15 -0.1 -0.05 0 0.05 0.1 0.15 0.2
'<\$5,-.+b,kú'6f,f&%O8,Y&+#&!$LP+#&!&#
U
î!3_"+72J+-VG$'6#-.+/400<'t=j++7/[L¹8&0<+7%O+#&;7!$5,#[? +-.+7)'6#% '<\$#,/40
8,-.!&;7+7% % '6#&\`?ZY&'6#&\J%O/J-.+%('Lû
8&0<+-O/1%k
[H}ü=|@]B>CUE"'<%f,!$5,#&),+7)'#BDvq~ZbJdE
/[#&)n8&'<+7;7+7_}'<% +P;7!$#[? '6#$5&!$5&%G,?ZY&+F!$5,-.'<+-"% +-:'*+7%
;7!$#J2[+-.\J+7%? ! @
BDC ¶ ExS@]B>CUýKEzÀþJb}/4?'6#[? +-.'<!$-K8,!J'6#[? %]/[#&) @
BDvJýKERSã@
BDbJd ¶ EzÀþJb
/4?}v¹/J#&)9bJd
H ¿ ¿
bqH}ü=@
BDCUE
'*%"/40<% !;7!&#J? '6#&5&!$5&%"!$#9BDvq~ZbJdEVG?ZY&+%Z5,L!4=?ZY&+U!$5,-.'<+-&+-.'<+7%}'<%|+7ï,5,/40
? !¹@]B>CUE}/4?}/40<0
8,!J'6#[? %H
qH}NJ{g/[#&)nz{¹? +#&)?(!¹ÿ7+-.!g/4?0*+/1% ?O/4%"=/4% ?}/4%þJiKH
g£¦ì¡|¬q¤ëZ£¢J¦
{4x
T { {
B JE
¶
_}'<?ZYk
N p Jþ b p T
i v { T BDN { ¼ { E þ[b
B JE
¶ { T BDNJ{S z {$E þ[b
!$f&% +-.2J+Jk ¶ { T {
¡|¡¢J£,¤|¥¦c§F¨J¥£©cªx«¥©x¨1F¢J¡}£¬§F¨1¥£©
"!$#&%('*),+-Ï?ZY&+24/4065&+n!1=/e8q+-.'<!$),'<;=5,#&;7? '<!$#@
BDCFE¹/4? ),'<% ;-.+7? ++7ï5,/10*0<^%Z8,/1;7+7)
23/105&+7%!4=|Ck
C T B T bJd G Tcv,~J~ .~ u¼E
"!$# #
%! && & '
{3 p
+,.-0/213'145/-76*/8(9:45;<451=>45=?4:- 18@/AB$CEDFAHG$DI8@DKJML*N(O(P>QR9:45;TSVUXWMJ?L*N>Y[ZEP$3-0AHJ?L*N]\^P%QR9:45;_SUXW^J?L*NM`
ZEPIa
$b
b 5&0<? '68&0*^fJ^ ¶ .y J/ #&)%Z5,Lu_}'<?ZY-.+7%Z8q+7;7?O?(! &k
*)
c!
p p {3 p
f,5&?Of[^g?zY&+%Z5,Lµ!1=
/P\[+7!$LP+7?Z-.'<;% +-:'*+7%/J#&)9f&06/4?Z/[#J?O/1% % +-.? '<!$#k
´ ¶ <{ ¶ y
*)
p l
iTch
ll
% !Uk ln
´¶
y T @
B RE ¶ .y !$# B.JE *)
p
p&+-.;7'<% +Ï=j!&-?ZY&+-.+/4),+-7k%ZY&!4_ñ'6#&% +-:? '6#&\e+7ï5,/1? '<!$#[P'#[? !+7ï,5,/4? '<!$#ãv%Z/1? '<%.¾U+7%
?ZY&+!$-:'*\['#,/10+7ï,5,/4? '<!$#&%H
3 saw(x)
’trig.pts.8’
’trig.pts.64’
2
-1
-2
-3
-4
-5
0 2 4 6 8 10 12
_OY&+-.+gBÝ=j!$-vPóió ¹þJb[E7k
´¶
N p T @
B b[d E X!
: p
$
NJ{ T b ´ ¶
@ B b[d qE ;7!J%B bJd 4i E
X! X!
: p
dy g£®4¥©|º§R©
<z|{ ]¥©| ]U¢[¥U®
}{
J
Y&+#&;7+/1%O@
BDCUE [t vÏ/4%OC st v,k
dm T |
S S Sc÷÷÷
Ä bm m
?ZY&+;7!$-(-.+7%Z8,!&#&),'#&\`% '6#&+P% +-.'<+7%}'<%k
CBDd`¼ºCUE
T d ö % '6#B>CUES % '6# BD[CUE S % '6#K· BD·JCUE SX÷÷÷ ø
$
/J#&)9/4?CgTcdxþJb,k
d Tc[b ö ¼ S · ÷÷÷ ø
$ $
£°¢J¥F¢¨J¢J§©|®VëZ£¢J¦
8,+-:'*!&)ebJd
H
{ T bJ d l o @
BDCFE ¶ { r$C e
¶o
d
T bJd l B &E ¶ { o r
e r (
¶
B ,E T bJd l xB $E ¶ r
B.7$E d e f ?
¶
@
BDCFE T { { (
{
B &E T B ,E d
e *
T
B ,E
e * $ w B.·JE
l B ,E rB ,E
e f $.t w e * (
¶
î+#&;7+_"+!$f&?Z/4'6#`?ZY&+U!$5,-:'*+- - /[#&%.=j!&- Lµ8,/4'6-7k
BDCUE T b[ d l B ,E r
e *
? B.ÄJE
¶
B ,E T l BDCFE ¶ r&C
e *
B. JE
¶
$·
p ï,5,/4? '<!$#ÏÄ"+7,8,-.+7% % +7%
/=5,#&;7? '<!$#P/4%/"%z8,+7;7?Z- 5,Lì!4=&=>-:+7ï5&+#&;7^;7!$L¹8,!&#&+#J? %H /Ja4+#
? !J\[+7?ZY&+-B_}'<?ZY)U5&+;7!&#&% '<),+- /4? '<!$#=D!$-
=>-.+7+24/J-.'6/Jf&0<+7%ZE_"+!$f&?Z/1'#k
@
BDCFE
T bJ d l l @]B $E ¶ r ,r B. JE d e *
g
h
? (
¶ ¶
U!&5,-.'<+- I%}ü#[? +7\$- /10 Y&+7!$-:+L'<%O/%(?Z/4 ? +LP+ #[?|!1= _Y&+#`?ZY&'<%|=D!$- LP5&06/Y&![0*),% ,'t=]@
BDCUE
'*%
+-.? '<+7%k
[H}&^L¹LM+7?Z-.^g/J#&)-.+/40<'<?.^
' = BDCUE"-.+/40x?ZY&+# B.¼ ,E
T B ,E
¡ 7
' = BDCUE"-.+/40K/J#&)+72J+#k
¡
p
' = BDCUE"-.+/40K/J#&)!$),)
¡
?ZY&+#
N ,BDCUES m BDCUE N VB ,ExS m B ,E
$ [
FH"R-.+7ï,5&+#&;7^%ZY&'t=j?('#&\ BDCUE B ¼ UE
R e¦*§
¨
BDCFE T bJ d l ¹B ,E B ,E r
® e f (
$Ä
R-:!$LÉ+7ï5,/4?('*!&#e ,G&_"+?ZY&+#!$f&?Z/4'6#`?ZY&+¹äzÐJÞUåÐJç<Ñ$ÖDÓÐJÞÏ!4=
@/J#&) k
s
!$-7k
l @
B $E xBDCP¼ &E r E ¹B ,E B ,E ? d
¶
&'6LP'<06/J-.0<^Rk
@
B &E xB $E l ¹B UE B ¼ UE r
I d X I¨ ¨ ¨
¶
ù%O/P%Z8q+7;7'/10;/4%(+;7!$#J2[!J0<2J+?ZY&+-:+/40F=>5,#& ;7? '<!$#&%@
BDCUE/J#&)9@
B.¼}CUE7Gq?ZY&+#k
¹B:¼ ,ET
7 B ,E d
B ,E T ¹B ,E B ,E
° d e
T ¹B ,E m BDb,E
¹
BDbJbJE d ?
º^»
¶
Y&+`=5,#&;7? '<!$# º^»'6# ï,5,/4? '<!$#b[b'<%¹?ZY& +ÁÙ[Ñ$ÖDÐäZÐJÒzÒ.Íç<ÙJÖDÓÐJÞ=5,#&;7? '<!$#ÇB!4= @E7G
_Y&'*0<+
p
: ¹B ,E m '6#+7ï5,/4?('*!	b,'<%O?zY&+PÕÚ,ÍZäÖDÒ Ù[çRâ[ÍÞ,Õ7ÓÖ 3H f&% +-.2J+
/J-:% +723/40 I%?ZY&+7!$-.+Lk
¼ <~ ½
BDv[ET l @
B $E m r T bJ d l ¹B ,E m r EÀ ? ?
º
¶ ¶
¿¾
£¦c
Á { _ ¡§¥¢J®
&!$LP++7,/JL¹8&0<+ - /[#&%.=j!&- Lµ8,/4'6-.%k
[H}&'LÏ8&0*+P+7,/JL¹8&0<+Jk
@]B>CUE
Tñs C ãv G ¹B ,ET
¶ ¥
v Cóãv N S %
@
¹B ,E T l ¶ ¶ r&C
§Ã
à *
T ¶ l ¶ B ý E r&C
Ä Ã
ÅFÆ Ã §
ÈÇ
Ä
à Æ
Ã
T ÅFÄ Æ
Ã
Ã
¶ l ý Ç
à Æ
Ä
¶ r
¶ ý
MÉ Ã ?Ê
¨ Ç
Ä
T Ë
d ¶
Ä Ã
à Æ
BDbJJE
ÅFÆ Ã
¨
(
1 1
0.8 0.8
0.6 0.6
0.4 0.4
0.2 0.2
0 0
-2 -1.5 -1 -0.5 0 0.5 1 1.5 2 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2
q H +7;7?Z/J#&\$5&06/J-8,5&0*%(+=5,#&;7? '<!$#k
N G ¹B ,E
T % '6#B ,N$E
,BDCUE
TÉs m v CC &ó
!
N ¥
¡
,N
¡?
DB b4$E
%Í
l ,BDCUE(r$CPT s Î
Õ
BDbJ·JE
¶ v ¹óc¼
!
7Õ
~ #?zY&+_}'<0*)e/4%(%Z5,L¹8&? '<!$#`?ZY,/1? '<?+7$'<% ?(%k
0<'6L B>CUE
Tcw,BDCFE
p
!
ØÚÙ
&!$LP+8,-:!$8,+-.?('*+7%k
(
[H";7!$#&;7+8&?Z5,/40<0<^
wB>CUE
T s v µCgTcv
ÜÛ
v C Tcv
o
bqH"?Z/Ja['#&\Ï?ZY&+0<'LM'*?!1=+7ï5,/1? '<!$#9bJ·qk
l w,BDCUE r&C¹Tñs ãv
k
¶ v góãv
qH}/4% %Z5,LP'6#&\e@
BDCUE'<%¹;7!$#[? '6#$5&!$5&%P'6#º?ZY&+`'6#J? +-:23/40B ¼ 7~ S EP/J#&)5&% '6#&\e?ZY&+
Õ "Õ
),'*%z8&0/1;7+7)Æ8,5&0<% +=>5,#&;7?('*!&# BDC¹¼ E f[^`?ZY&+'6#J? +-(LP+7),'6/4? +24/4065&+P?ZY&+7!$-.+LÉ=D!$-
u! ¦
w BDCUE
T w,BDCϼei eE
{
%
_}'<?ZYw
H
Þ
§¦c¡|¬¥©
{ [ß
sÐ|F£¢JF¦
~ 5,-;7!$L¹8&0<+7/J#&)g;7![% '6#&+ 4% '6#&+% +-:'*+7%"\$/2J+5&%A/),'<% ;-.+7?(+% +7?|!4=KU!$5,-.'<+-;7!&+zg;7'<+#J? %
¤
=D!$-P/e8,+-.'<!&),'<;9=5,#&;7? '<!$# "!$-0<!&!$aJ'6#&\/4?P'<?g/J#&!J?ZY&+-M_A/^=D!$-P/=>5,#&;7?('*!&#_OY&'<;7Yº'*%
à
$¸
"!$#&%('*),+-K%z/JL¹8&0<'6#&\ /% '<\$#,/40 xB>CUE/4?K'#[? +-.24/40<% µ? !}!$f&?Z/1'#?ZY&+|),'<% ;-:+7? +"? '6LP+|% '<\$#,/10jk
á [
T xB>CUE w BDCFE
T
B ,E w,B ¼ºbJdh`E
y
¸ ?
T B P¼ bJdh E BDb JE
y
]
!4= B ,E_}'<?ZY8,+-.'<!&)nbJdxþ H
$
B ,EGB.¼ ó ó E
]ã
7ã
ã
!$f&?Z/1'#k
B ,E
T b bJd xB bJb di E ¶ { m.o m GB.¼ ó `ó ÇE BDb JE * r eä
{
e 7ã
ã ]
]ã ]ã
BDCUE T b[ d l B ,E r
d e * ?
¶
T b[d b bJd xB id E ¶ { o
l ä
r * r ä *
{
e ?
¶ ä
]ã ã
T b B b id E l ¶{o r ä
*
g¶
r ä
h
{
?
áã ¶ ]ã
ä
T xB E ÇC¹¼º¼ºiRdidE
i d
% 6
'
# B Ç ¹
C BDbJ¸JE
ã
{
ã ã
% +7;Jk
B E T xB @ i E % '6#Kd|BDd|BD@ BD@ K¼º¼eiEiRE(E $éFê
{
ê
$é $éFê
·Jv
ñ ó
ìí
ëì íî
ë î
ñõ]ó
û
@ï
v ï
ð
ñ ó
° ñ ó
ñò ó
ð
û
@ï
ô
ï
"# û%$'&
(*)+-,/.- õ û]ý õ% øù þ^õ
_ö^õ]þ?ÿ ø.ø û ÿ ö10 ã ñ2 û- 43 óRò õ]þ
ö û ú þ ø65^ú û 0ÿ û
û- c
ø þ û ÿ"ö107'õ
ø þ ù õ õ - õ û 8:9 é<;>=áã " # û .ø þ ø?.ú 'õ
ø þ ù - õ û
9 éA@B]= ã ø !# ûDCFEG
H IJ¦êO- õ û "# û 'õ
ø þ ùK!# û -Fû æø L M û eø û _ò õ86 û ÿ !# û
PRQSPT UFVSWXY
WZ[ß
@þ -
Fû õ þ
[ ø ö û 4 ö ú?ø ÿ !# û õ]þ^õ
]ù(ú û û-]\ ?
8 ø 0 ø ø q # ø ò # \ ( ú ÿ # õO^ û _# û
û-
û ò -
Fû !
þ û-Fû ^ 5 ú?ø - û ÿ 4 õ$ò #?ø û ^ û !# ûa` 0 5^ú?ø - õ û ñ -Fû !
þ û # \ þL ù(ú -Fû×ü ñò óeó
÷[øù(ú Fû - cb
d þ - õ û !# û
- ö û æøef_# û 'õ
ø þ ù - õ û ø [M \ - õ ùáø ^ û þ
ÿ û
û- ïû # õ$ÿõ
4 ú û ÿ õK øù þ^õ
7ö^õ]þ?ÿ ø.ø û ÿ 4 ã õ]þ?ÿ<'õ
û ÿ
õ ` 0 5Mú?ø
- õ ûg=]ãBh ö úa!# û ø¶ù þ^õ
iñ - õÎö^õ$ÿ0i û-Fû ÿK^ û- ø þ
!# û øù þ^õ
*ó # õ
Tþ þ1 3 û-
-Fû 5Mú û þ?ò0ò M
þ û þ õ L -Fû 5^ú û þ?ò ø û #?ø¶ù# û- !# õ]þ ï \ #?ø ò #%!# û
û- ø ÿ ø ò ø6 0
!# û - õ]þ - kj ñ^ l óò õ ú û 4 ö û õ$ÿ^ÿ û ÿ ø þ7@þm MMnáø þ ùo þ60 ø þ !# û ø þ û- ^$õ
p
ü
1
y = sinc(x)
0.8
0.6
0.4
0.2
-0.2
-0.4
-8 -6 -4 -2 0 2 4 6 8
r6s1t6uOv*w
÷[ø¶ù?ú Fû - gq ^ý "# û ø þ?ò dú 6
þ?ò ø þ h
é
uOv
1.2
0.8
0.6
0.4
0.2
0
-W 0 W
÷[øù(ú - :b
û ^ý ]x ø õ
ø þ ù _y
û Xû ò û z õ
û
=
ü
|{ ã<}'ã ó ø õ
û
ñ õ - !# õ g !# û õ ø # Oõ ^ û ö ûû þ ~ ÿ û ÿ ^ û- õ]ö ?ú @ { ã õ]þ?ÿ
" @ ã \ ø6_##?øùM# -
û -Fû ^5 ú û þ?ò
ø û Tö û ø þ ù - _
Fû %û ò û ÿ õ8 H\ û- -Fû M
5 ú û þ?ò ø û *
" õO^ ]ø ÿ !#?ø c
- ö û h ø.ø þ - õ
4 õ ø?k4 2õ
ûD\Hû iõáö ^ û !# û` 0 5Mú?ø
- õ û ~ -Hûz õ
û õáþ?ÿ%õ - ÿ q
M ö
Ag ú û ÿ ø þ ! # û
# þ û 4 0 û õ ø?4
ö^õ]þ?ÿ ø.øA!# û õáþ^õ
]ù(ú û ø þ
ú ø¶ù þ^õ
4 õáö (úa
n 2 3 õ]þ?ÿ2õ
û õ an 2 3
z õ
û ×õ86 õ - ø û ø þ ?
ø õ ù û ÿ ø¶ùáø ø 6 63 õ e ø þ \ # -
û û !
^õ e ø õ
-Fû 5^ú û þ?ò ø û #?øùM# -
û !# õ]þ
g4]RgA [iR¡¢g£*¤]¥
ïû # Oõ ^ û
# \¦\Hû ò õ]þRÿ û e ò ûû þ - ø ö û õg
û- ø ÿ ø òd øù þ^õ
Xö10õ_ÿ ø eò - û û û
â÷§(ú - -
ø û
ò û_¨ ò ø û þ _õ]þ?ÿò þ1^ û- û 60 # \ õÿ ò -Fû û û 8
]ø þ Èò õ]þö
ø e û ú û ÿ 4 ÿ û e ò - ø ö û
õÎö^õ]þ?ÿ ø.ø û ÿ ø¶ù þ^õ
T¹QSP º¼»ZW½W¾RZY
Ñ
þ ø ÿ û- õ¿ÿ%õ õi û M 5 ú û þ?ò F s Ài@¿ Á } Â }OÃ*ÃOÃ ÄaÅÂ À §
û ¿ ÷ -_ûz õ
û !# û û ò (
ú ÿ
-Fû
-Fû û þ _# û ^0õ
ú û A' õ
û ÿ - M õ]þõ]þ^õ
]ù(ú û øù þ^õ
J ñ ê ó \ ø!
# sg@ÆJ ñÇ§È é ó
"# û ø e ò -
Fû û ÷ ?ú - - " -
ø û - ø ý õ]þ
ÄÅÂ
Å ÍÎ_Ï É s ñ l
jdÉ @ Ê sMÌ Ð h @"ô}Op}*ÃOÃÃ|}!C { p ó ñ p ó
s
Ë Á
õ]þ?ÿ ø þ^
ø -
û û]ý
ÄÅÂ
p Ê Í Î_Ï É s ñ Ç
sd@ j:É Ì Ð h @"ô}Op}*ÃOÃó}_C { p ó ñ = ó
C ÉË Á
_þ û õ
Ñ -
^õ ø þ
! # û ò þ eø þ ú(ú § ÷ (ú - ø û- " - - õ]þ?ÿc û- ø û qþ \ ÿ ø 'õ
û õ - h
õ]þ
!# û-Fû ø 7þ :5^ú û eø þ 8
4 ø ö û ò þ^ û- ù û þ?ò û
- ö û \ ø!# ø.ø ×õ8 _ # û û ú
õ -Fû õ86 £þ ø6 û
"# û
-
-
û eø û
!# û ¸
÷ "Æ.ø.ø ò _# û ò ú (ú ^
þ ø þ -
û ø þ ý
p 0 c û -
0 õ]þ?ÿ -
Fû õ
ø 0
ü
Õ øÖ] s - û õ
! # û þ j ¼Å É @ jd×É
Õ õ
s ø a
û- ø6 ÿ ø ò h j t Ù
Ä Ø[Ú w Å É @ j:×t ÄÙØ4Ú wÜÛ É
= Ý ø þ û õ - 0
ø Þ s'ß7àá¼s # 8õ ¸
÷ " Þ jdÉ ß©àâ ÉÈø þ !# û ö^ ø(ú õ]þ^þ û-
? # øÖeø þ ù ö û- ^ û % #?øe~sø þ ! # û s 0^ õ
ú û ø - û õ
60õ
õ -
õ eø þ ÷ - !# û
- õ û ÿD û ^5 ú û þ?ò û s Å sOã !# û ÷¸"Ýø j:É Ì Å Ú *u r É s*ã ØÄ
"# -
û Fû 86 c!# û
^õ - õ
û 8
ø ×õ ò 1þ ^ úeø
þ "# c
û «O+ܬ|«O)M䲬A«_¯´ åO¯ä)M.+°¯´
û M
5 ú û þ?ò û
s õ]þ?ÿ ás ø ¢ÿ û £
þ û ÿö10 ý
ÄÅÂ
EOsc@ Ê ás Å æ h ñ Ç @"ô}Op}*ÃOÃÃ~C { p ó
Ë Á æ
æ
"# û ¸
÷ "¦8 E s ø !# û þ ý
ÄÅèÂ
Ê Å ÍÎ_Ï É s
çÉ @ EOsMÌ Ð
s
Ë Á
ÄÅè ÄÅèÂ
Ê Ê Å Í Î!Ï É s
@ æ ás Å æ Ì Ð
s
Ë Á Ë
ÄÅè æ Á ÄÅèÂ
Ê Å ÍÎ_Ï É æ Ê Å ÍÎ_Ï É ts Å æ w
@ æ Ì Ð á s Å Ì Ð
Ë Ë æ
æ Á æ Á
@ j É â É ñ ó
T¹Q°é êXYO½ëê¾RìÓWº§ÓíÓXZ¸Yïî!¾RÓð
x ø
ø eø ò ø
û û þ õ eø þ 8!# û ÷¸" \ (ú ÿ -Fû 5^ú?ø -FûiC Ú ò M
ûz .ú eø
ø
ò õ eø þ_õ]þ?ÿ C ñ C { p óHò M
ûz õ$ÿ^ÿ øeø þ*2 \Hû ^ û- !# û 40 c û - 0 8¸!# û ÷¸"
ò õ]þö ûÎûz
áø6 û ÿ
õ]þ?ÿõsÿ ø ^ ø ÿ û õ]þ?ÿò þ 5Mú û- - õ û ù 0i û õ$ÿ 4i!# û ÷ õ
_÷§(ú - ø û-
Ú|ø§ÅÂ
Ê s É
jdÉ @ sò h l @ }*p}OÃ*ÃÃ=÷ { p
ô ñ
ó
s
Ë Á
ø§Å Ú|ø§ÅÂ
@ Ê s
ò s É
ß Ê sò
s É
sNË Á sNË ø
ø§ÅÂ
Ê ñ saß É4ø És
@ s
Û ø ò ó ò
sNË Á
ø§ÅÂ
Ê ñ saß É És
@ s
Û ø |ñ { p ó ó ò ñ ü ó
sNË Á
ü
Á ð î ð ß ð ð jLÁ
î ë ú
ë
î
 î
ë
ß
³
]ø þ jLü
ð î ë ð ð ð
î
î
ë ë ú ¸
÷ "
î ë
î ë
î ë
MÚ ð î
î ë
î ë ð ß ð ð jdÚ
î
î ë
î ë ë ú
ëî ë
î ë î
ë î ë
û ð ëî
ë
î
î
ë ð ß ð ð jLþ
î ë î
î
ë î î ë ë ú
ëî ëî
ë
î ë
ë î î ù ò Á
ë î ë î ë
ü ð ëî ë î ð ð ð jñÂ
ë î
ë î î ë
ë î
ë
ë
î ë î î ù ò Â
Mý ë
ë
î ë î
³
]ø þ jdý
ð ð ð ð
ë î î
ë
ë î î
¸
÷ "
ò Ú
ë
ë ë î î ù
þ ð ë î
ð ð ð jLû
ë î
î
ù ò û
ë
î
ë
Mÿ ð ð ð ð jdÿ
÷[øù(ú Fû - ý ø ^ ø ø þ
ó ³
]ø þ ×ø þ 4c \
³
]ø þ ¸
÷ "
õ
h¼ò ø @ { p h # û þ?ò û dò [É ø @ ñ|{ p ó É ïû ò õáþ _# û þ ?ø
60F û ^
õ - õ û (ú û ^ û þõ]þ?ÿ
ÿ^ÿ û- 8Ù
jdÉ \ û ö õ øþ ý
õ]þ?ÿ i
ø§ÅÂ
jdÚ @ Ê ñ saß sNÛ ø ó ñ ò Ú ó s h @"ô}*p1}*Ã*ÃÃ[}!÷ { p
sNË Á
ø§ÅÂ
j Ú Û Â @ Ê ñeñ s { sNÛ ø ó ò s óñ ò Ú ó s h@"ô¼}*p}OÃ*ÃÃ|}!÷ { p
sNË Á
` eø ò û ! # õ d !# û û õ Fû - \ ÷ ³
]ø þ ¸
÷ "
A!# û û ^
5 ú û þ?ò û ¿ s:ß sNÛ ø À õ]þ?ÿ
¿ s
ñ {
s Û ø ó ò s À
C ø_ õd
H\ û-
=h !# û þ \Hû ò õ]þsÿ û ò
û ø þ !#?ø õ]þ^þ û - áù Ú C eø û ú þ eø
- õáþ ~ - *
\Hû ö õ ø þ C ø þ ù û
]ø þ
"# û ÿ ø õ ù -
õ |ø þ ù(ú Fû - # \ !# û ÿ ø ^ ø ø þ
õáþ ³
]ø þ ÷¸"Ýø þ 4d \
³
]ø þ
÷¸" * û ò ú - ø ^ û ÿ ø ^ ø ø6 þ ø þ ! #?ø õ]þ^þ û -}- Fû ú ø þ _ # û ÷ õ
H÷§(ú - ø û- " - õ]þ -
ñ ÷[÷¸" óï
ü]ü
Stage 1 Stage 2 Stage 3
÷[øù(ú Fû
^ý - x þ z õ
û
û )..*¬f&
^õ 4 -
û þõ û õ$ò #i
!# û _# -
Fûû õ ù û ø
# \ þ ø þ
ö ¶ÿ
p qõ$ò # ö ú4 û-³ 0 - û 5Mú?ø -Fû þ û ò
ûz .ú ø
ø ò õ eø þÝõ]þ?ÿ \ õ$ÿ^ÿ øeø þ
# û þ?ò û ÷[÷¸" -Fû 5^ú?ø -Fû ñ C
= ó¼ ]ù Ú C ò M
ûz .ú eø
ø ò õ eø þ õáþ?ÿ C MÚ C
ò M
ûz õ$ÿ^ÿ øeø þ
\Hû # õO^ ûÁ-Fû ÿ ú ò û ÿ !# û ñ C Ú ó ÷¸"
- ò û 4 4 õ]þ
ñ C ] ù Ú C ó þ û
= x û õò #ø û- õ eø þ \ ø!#?ø þ û õò # õ ù ûh\Hû þ ûû ÿ"ò þ ø ÿ û- þ0 \ ø þ
ú
ò û_¨ ò ø û þ \ #?ø ò #ù û þ û- õ û \ i(ú
ú ò û_¨ ò ø û þ \Hû\Hû-Fû 4 4 -Fû
øe
!# û ò û_¨ ò ø û þ ø þÎõ]þõ - - õO0 h !# û \ ?ú
ú [ò õ]þ òò ú
0 !# û 'õ û ò õ eø þ
õ
!# û \ ø þ
ú .ñ !#?ø
ò (ú - û ÿ û - 0 !# û ø þ
ú7ø þ !# û
- ò û 4'ó¸ó^ û þ
øÖ H\ û 4 -Fû !# û (ú
ú_ø þõ]þ !# û- õ - - õ*0 _#?ø õ8 ù1 - ø!#ø A eø 6 þ60 ñ C ó
þø
õ$ò û û- *
!^
"R £þ?ÿ ! # û ò õ ø6 þ 8 ¸j:Éø þ _ # û ÷[÷¸"¦(ú
ú õ -(- õO0 h õ n û lõ
×õ]þö ø þ^õ -0
þ ú ö û-
T¹Q OZ¹º§ÓY
ºÑ%êAí êAêAí
"# û Iþ^ -
û û ¸
÷ "Ýø ùáø ^ û þuö 10 ý
ÄÅÂ
p Ê Í Î_Ï É s ñ Ç
sd@ j:É Ì Ð h @"ô}Op}*ÃOÃó}_C { p ó
C ÉË Á
"#?ø Èò õ]þö P- \O- 4
û Fû ø û þõ 8 ý
ÄÅÂ
C s× @ Ê É s h ñ Ç @"ô}Op}*ÃOÃÃ|}!C { p
j ×É ò ó
ÉË Á
" #?ø ø ¤
ûû þ L ¸
4 ÷ "¦
¹_# û ò M
ö û
ûz ò þOÑ ú?ù õ û 8 ¹!# û ÷ ?
õ ú - ø û- ò û_¨ ò ø û þ
!#?ú !# û ÷[÷¸" ò õ]þö û ú û ÿõ8 õ]þ ø þ1^ û- û ÷¸" õ û- õ
-
- ø õ û õ84'õ ù]ø þ ùi
q
ü
T¹Q ¾Óº"!Wðƺ§Z¸Y
W¾Z¸Y
)iÅÂ ÄÅÂ
p Ê Ê Å Ú u*r°t.0/ Û3254 w
*¹É,+ @ 9 % +s Ì 1 Ð
- C % Ë Á sNË Á
- p ô # \ a
÷[øù(ú Fû û
¸_# û ò ø þ û ö^õ
ø @ ú þ?ò eø þ* ÷ - c û M
õ 5 ú û þ?ò û
qø õ ù û h
\Hû .øùM#1 ò þ ø ÿ û- ÿ ø û þ ø þ
ñ \ !
^õò ûh þ û ø? û ó
1 1 1
0 0 0
6 6 6
5 5 5
4 4 4
0 3 0 3 0 3
1 1 1
2 2 2 2 2 2
3 3 3
4 1 4 1 4 1
5 0 5 0 5 0
6 6 6
"# û ÷[÷¸"
õ ù - ø!# õ1
ø û û 5Mú õ860 \Hû ø þ #?øùM# û- ÿ ø û þ ø þ - ÿ ø ? û þ ø þ76 h
\Hû ö õ ø þõ]þ ñ 7 C 8 áù C óHõ
ù - ø6_# - õ ! # û- !# õ]þ ! # û ñ 7 C 8 Û Â ó
ïû # õO^ û .øù - õ û ÿ - Mæ÷§(ú - ø û- û- ø û Îõ]þ?ÿ " - õ]þ -
ò þ eø þ ú(ú g_
^õ$ò û -
eø û @ ú þ?ò eø þ \ #?ø ò # ûz
-Fû 4 øù þ^õ
ø þ -Fû 5^ú û þ?ò0_
^õ$ò û 4 ÿ ø e ò -Fû û - õ]þ -
ÿ ø ò -Fû û dú þ?ò eø þO¸2 \Hû ^ û-h
\Hû ò õ]þ - û ò õ
!# û ÷¸"ø þÎõ d -Fû ù û þ û- õ
- õ û
\ - n ï - náø þ ùÎø þ \ ÿ ø û þ ø6 þ h \HûA\O- ø6 û?> 9A@R4 -Fû
- û û þ ¤!# û õ - ø zàý
BC
C 9*Á,+ Á 9OÁE+ Â FGFHF 9OÁE+ ÄÅÂ JLK
K
C 9IÂ + Á 91Â + Â FGFHF 91Â + ÄÅÂ K
C K
9 @
>A @
D M
9 i
) ÅÂ+ Á 9 i
) ÅÂ+ Â FGFHFB9 )iÅÂ+ ÄÅÂ
ö1^ ø(ú õ - ø z - ÿ ú ò ý
@ >QP )
+ )
@ > R
> *N@ O 9 @ >QP 3
Ä + Ä@
Å ÅèÂ
>A @ >QP )S+ )
9 @ O @ Ä +Ä @
> *N@ >QP 3
b
ü
\ # -
û Fû
>TP=U + U @Xø !# û;VW7V õ - ø
z \ ø !# û û û þ ñX } Çàó ù]ø ^ û þö 0 ý
p Å ÍÎ_Ï ¸
% s
Ì Y h X } Ç @"ô}*p1}*Ã*ÃOÃ|}ZV { p
V
@þ ù û þ û -
õ !# û þ - þ þ1 ø þ ù(ú * õ - õ - ø ò û >Q[ @ õ]þ?ÿ >]\ @ h\Hû ò õ]þò
þ ø ÿ û - !# û
ù û þ û - 8
õ ø û ÿÿ ò -Fû û - õáþ ~ - ^
ø e
õ ø- ù]ø ^ û þuö10 ý
@ >Q[ @ > A
> *N@ ^ 9 @ >]\ @
Åè ÅèÂ
>A @ >Q[ @
9 @ O > *N@ >]\ @
"# û g ""ø þ \Æ\ ø ÿ û 0 ú û ÿ ø þ ø õ ù û ò M
-Fû 4 ø þñced¸gf õ]þ?ÿ7hid(f kjA|@ó
õ]þ ø õ ù û ø Î ò õ - ^ û ÿ ú
ø þ 4 õ86 5^ú õ -Fû eø û
ClWoC
ø z?û õ]þ?ÿ !# û g "
õ
ø û ÿ 4 û õ$ò #eø û ñõ û- ú.ø þ^õ]þ?ò û #ò # -
.ø þ^õ]þ?ò û û
^õ - õ ø6 þ^ó å 0õ
-
- ø õ û
5^ú õ]þ ø63 õ e ø þõ]þ?ÿ û þ?ò ÿ ø þ ùK
_# û ò û_¨ ò ø û þ õ #?øùM# 0ò M
-Fû [ û ÿ ~ -
!# û
ø õ ù û ò õáþuö û ù û þ û- õ û ÿñþ û !# õ _#?ø ø × þ ×ú ú õ
60õL 4 -Fûû
- ò û 4'óï
0.5
0.45
0.4 DCT
DFT
Original
0.35
0.3
0.25
0.2
0.15
0.1
0.05
0
0 1 2 3 4 5 6 7
ü
-
û Tö û ø þ ùù û þ û -
10 ! # û ÷¸" "¤# û ø
- õáþ
-
û- 0
!#?ø ø !# õ õ
õ û ÿö
!# ûcCmW C
ø z?û eø ûÎû ÿ ù û h _# û g " û þ ú -Fû _# õ '9 ñ Å ó @ 9 ñ Û óï è ø þ ùK!# û
÷¸" þ !# û - øù]ø þ^õ
eø û û õ$ÿ [ ä¯O«EnN+°´po²¬_.+ µ²«*.°,h\ # û-Fû õ û- ø þ1^ û- ø6 þñ øe \Hû
# õO^ û õ
- z ø? õ û ÿ !# û ? # ø¶ù#F -Fû 5^ú û þ?ò0 ò
þ û þ 'ó h !# û
ø z?û eø ûsû ÿ ù û 'ú - þ
(úd4 # õO^ û !# û û õ]þ<$^ õ
ú û
a_# û
ø z?û Öõ :_# ûû þ?ÿ
û õ$ò # - \ - ò ú þ
÷[øù(ú -Fûcpp # \ !# û_û_X y û ò
õ Fû¢û ø -
6_# û-({dp õáþ?ÿ \ # û-Fû _# ûO- \ ×ñõ]þ?ÿ # û þ?ò û þ û ò û 42õ - 0
ø Îò ú þ'óHõ -Fû . ú'ú õ
0
- _#] ù þ^õ
÷ -Hûz õ
û]ý
BC
C p p p p JLK
K
C p p { p { p K
p { p { p p
D M
p { p p { p
éQ°é |Ó
½]и¾Z¸¾RÓðXèVî[ìZ(`½W¾Z¸Y
@þ ù û þ û - õ
H\ û õ -Fû Mnáø þ ùg - õ û
- !# - þ -
õ dú þ?ò eø þ_ñ \Hû # õ*^ û ú û ÿ: ø þ ûh
ò ø þ ûh ò
ûzÎûz
û þ eø õ
h õáþ?ÿc£þ^õ
0i2×õ$ÿ%õ õ - ÿ%ó \ ø!# \ ? # ø ò #4 -Fû
-Fû û þ õ
@ú þ?ò eø þ
"# û.ûz õ
û ù]ø ^ û þK ñ õ - õ -Fû õ86qõ
-Fû eò - ø ö û ÿ û
@ú þ?ò e ø þ ø þ?ÿ û
û þ?ÿ%õ]þ g
!# û dú þ?ò ø6 þ \ #?ø ò # \Hû õ - û - 0 ø þ ù[ - û
-Fû û þ # \Hû ^ û-h ø ø Îõ
4 ø ö û 4
û û ò - _
# þ - õ8(ö^õ
ø @ú þ?ò eø þ \ #?ø ò # õ - û
eø.ø û ÿ ~ - õa^
õ - eø ò ú fõ - d ú þ?ò eø þ h
- · -Fû ø þ û-Fû eø þ ù 0 õ õ .ø 0
R@ú þ?ò eø þ*
"# ûA õ - #?ú þ û þ1³Ý û } ^ û _# û -Fû ÿ û eò - ø ö û ú ò # õ û ò # þ ø65^ú û ø þ \ #?ø ò # O+Qo*´§å*_«*.¯¬[,
~ ¶:²1.¬!+°«!ï,k
¸!# û õ ú[ ò -(-Fû *õ eø þ @ú þ?ò eø þõ -Fû !# û ö^õ
ø *A ò õ]þö û # \ þ !# õ
!?# ø . ÿ û ò M
øeø þ ø !# û ö û õ$ò #?ø û ^0õ]ö û 2 \iû ^ û- !# û ò
ú õ eø þ
!# û û
û øù û þ1^ û ò 4 - ø ûz
û þ ø ^ ûh ñ C
ü ó - õ = ø õ ù ûh õ
!#(ú?ù#K õ8 û- ^ û- ø þ.ÿ
ûz ø ö^õ
û ÿ þ ÷[÷¸" _# û ÿ û- ø ^$õ eø þ 8 \ #?ø ò #ø þ L - !# û 5Mú û õ cø # ñ° ûû
û þ û ÿõ]þ?ÿ õ n h øù]ø õ
d ø ò 'ú -Fû d - ò û 4 ø þ ù ó
éQ bD¾RZ¹¾RVSì½W¾Z
üp
G&
þ ø ÿ û - !#
-
8 û ÿ ù û ÿ û û ò eø þ ø þõ = ø õ ù ûh\ # û-Fû !# ûÎû ÿ ù û ø .
û ö û þ
þ û ò û 4'õ - 60L^ -
-
ø # - ø3 þ õ
û " #?ø ø þ1^ ^ û [õHò þ ø ÿ û- õ ø6 þ
!# û ù - õÿ ø û þ R
eø ò õ
!# @ ú eø 9
û ?
þ ò þ
ñ } _ E ¡
ó
õ ù
þ
^ õ
!# ! #
þ û -
ú $
õ ò
û õ
!# [ø ! # û ^ û ò -
4
¹ ù ø'ú ÿ ûh
õ þ
ñ 9 %ó Ú ß ñ 9 E ó Ú h õáþ?ÿuÿ ø -Fû ò eø þ õ]þ ÅÂ ñeñ 9 E ó ñ 9 %óFó
200
function
gradient
150
100
50
-50
-100
0 50 100 150 200 250 300
÷ - õÿ ø¶ùáø 6
õ -
-
û Fû û þ õ e ø þ !#?ø ø þ?ÿ ø ò õ y û-Fû
õcÿ ø X
û T 5
þ?ò û.û ^ú õ eø þ ý
` û !#?ø ø 7
!# õ ¸ õ_ò þ ^ úø6 þ
¹9sø þ !# û sõ]þ?ÿ E -
ÿ ø Fû ò eø þ \ ø !# ¿ À ù]ø ^ û þRö0 ý
¿ { p}Op}'ô}2ô}ÃÃÃ À
pr
ïû -
õ Fû ø þ -
û Fû û ÿ ø þ= -
Fû ò þ -
ú ò eø þ ù õ ? ø \ #?ø ò #F# õ$ÿ ø þ øeø
õ
ø z?û ^0õ8 ú û
õ ù û ñ
9 ñ }_E óó h - M ö ú - -Fû ÿ ø? õ ù û ö û- ^ û ÿ M û eø û f õ û- !# û ö ú - -Fû ÿ ø õ ù û õ
ø ûê ø ñ }!E§}eê ó ïû .øùM#id ÿ û _#
e û ö ú - ø þ ù7
õ]þ ø? õ ù û õ8Öõÿ ø y ú ø6 þ
- ò û 4 ^ û- ø? û øÖ \Hû þ ûû ÿ 4 ò þ ø ÿ û- !# û úeø þ 4 !# ûK\iû n þ \
^õ - ø õ8Xÿ ø yKû- û þ eø õ
û 5Mú õ ø6 þ ø þ^ ^ ø þ ù!# û ÝXõ
* õ$ò ø õ]þ ý
Ú
Ú p
ß @
è Ú E Ú l ê
q]ô
5 9000
’P0903A.DAT’ ’P0903B.DAT’
4 8000
7000
3
6000
2
5000
1
4000
0
3000
-1
2000
-2
1000
-3
0
-4 -1000
-5 -2000
0 50 100 150 200 250 300 0 50 100 150 200 250 300
61
9 Quantized Degrees-of-Freedom in a Continuous Signal
We have now encountered several theorems expressing the idea that even though a signal
is continuous and dense in time (i.e. the value of the signal is defined at each real-valued
moment in time), nevertheless a finite and countable set of discrete numbers suffices to
describe it completely, and thus to reconstruct it, provided that its frequency bandwidth is
limited.
Such theorems may seem counter-intuitive at first: How could a finite sequence of num-
bers, at discrete intervals, capture exhaustively the continuous and uncountable stream of
numbers that represent all the values taken by a signal over some interval of time?
In general terms, the reason is that bandlimited continuous functions are not as free to
vary as they might at first seem. Consequently, specifying their values at only certain
points, suffices to determine their values at all other points.
Thus we see that any signal that is limited in its bandwidth to W , during some
duration T has at most 2W T degrees-of-freedom. It can be completely specified by
just 2W T real numbers (Nyquist, 1911; R V Hartley, 1928).
and
F (|k| < kmin ) = 0 (6)
and if it is also true that the signal f (x) contains no complex zeroes in common with
its Hilbert Transform (too complicated to explain here, but this constraint serves to
62
exclude families of signals which are merely amplitude-modulated versions of each
other), then the original signal f (x) can be perfectly recovered (up to an amplitude
scale constant) merely from knowledge of the set {xi } of zero-crossings of f (x) alone:
Comments:
(1) This is a very complicated, surprising, and recent result (W F Logan, 1977).
(2) Only an existence theorem has been proven. There is so far no stable constructive
algorithm for actually making this work – i.e. no known procedure that can actually
recover f (x) in all cases, within a scale factor, from the mere knowledge of its zero-
crossings f (x) = 0; only the existence of such algorithms is proven.
(3) The “Hilbert Transform” constraint (where the Hilbert Transform of a signal
is obtained by convolving it with a hyperbola, h(x) = 1/x, or equivalently by shifting
the phase of the positive frequency components of the signal f (x) by +π/2 and shifting
the phase of its negative frequency components by −π/2), serves to exclude ensem-
bles of signals such as a(x) sin(ωx) where a(x) is a purely positive function a(x) > 0.
Clearly a(x) modulates the amplitudes of such signals, but it could not change any
of their zero-crossings, which would always still occur at x = 0, ωπ , 2π 3π
ω , ω , ..., and so
such signals could not be uniquely represented by their zero-crossings.
(5) Logan’s Theorem has been proposed as a significant part of a “brain theory”
by David Marr and Tomaso Poggio, for how the brain’s visual cortex processes and
interprets retinal image information. The zero-crossings of bandpass-filtered retinal
images constitute edge information within the image.
63
10 Gabor-Heisenberg-Weyl Uncertainty Relation. “Logons.”
10.1 The Uncertainty Principle
If we define the “effective support” of a function f (x) by its normalized variance, or the
normalized second-moment:
Z +∞
f (x)f ∗ (x)(x − µ)2 dx
(∆x)2 = −∞
Z +∞ (10)
f (x)f ∗ (x)dx
−∞
and if we similarly define the effective support of the Fourier Transform F (k) of the function
by its normalized variance in the Fourier domain:
Z +∞
F (k)F ∗ (k)(k − k0 )2 dk
2 −∞
(∆k) = Z +∞ (12)
F (k)F ∗ (k)dk
−∞
where k0 is the mean value, or normalized first-moment, of the Fourier transform F (k):
Z +∞
kF (k)F ∗ (k)dk
−∞
k0 = Z +∞ (13)
∗
F (k)F (k)dk
−∞
then it can be proven (by Schwartz Inequality arguments) that there exists a fundamental
lower bound on the product of these two “spreads,” regardless of the function f (x):
1
(∆x)(∆k) ≥ 4π (14)
64
The unique family of signals that actually achieve the lower bound in the Gabor-Heisenberg-
Weyl Uncertainty Relation are the complex exponentials multiplied by Gaussians. These
are sometimes referred to as “Gabor wavelets:”
2 /a2
f (x) = e−(x−x0 ) e−ik0 (x−x0 ) (15)
Note that in the case of a wavelet (or wave-packet) centered on x 0 = 0, its Fourier Trans-
form is simply a Gaussian centered at the modulation frequency k0 , and whose size is 1/a,
the reciprocal of the wavelet’s space constant.
Because of the optimality of such wavelets under the Uncertainty Principle, Gabor (1946)
proposed using them as an expansion basis to represent signals. In particular, he wanted
them to be used in broadcast telecommunications for encoding continuous-time informa-
tion. He called them the “elementary functions” for a signal. Unfortunately, because such
functions are mutually non-orthogonal, it is very difficult to obtain the actual coefficients
needed as weights on the elementary functions in order to expand a given signal in this
basis. The first constructive method for finding such “Gabor coefficients” was developed
in 1981 by the Dutch physicist Martin Bastiaans, using a dual basis and a complicated
non-local infinite series.
The following diagrams show the behaviour of Gabor elementary functions both as complex
wavelets, their separate real and imaginary parts, and their Fourier transforms. When a
family of such functions are parameterized to be self-similar, i.e. they are dilates and trans-
lates of each other so that they all have a common template (“mother” and “daughter”),
then they constitute a (non-orthogonal) wavelet basis. Today it is known that an infinite
class of wavelets exist which can be used as the expansion basis for signals. Because of
the self-similarity property, this amounts to representing or analyzing a signal at different
scales. This general field of investigation is called multi-resolution analysis.
65
Gabor wavelets are complex-valued functions, so for each value of x we have a phasor in
the complex plane (top row). Its phase evolves as a function of x while its magnitude grows
and decays according to a Gaussian envelope. Thus a Gabor wavelet is a kind of localised
helix. The difference between the three columns is that the wavelet has been multiplied
by a complex constant, which amounts to a phase rotation. The second row shows the
projection of its real and imaginary parts (solid and dotted curves). The third row shows
its Fourier transform for each of these phase rotations. The fourth row shows its Fourier
power spectrum which is simply a Gaussian centred at the wavelet’s frequency and with
width reciprocal to that of the wavelet’s envelope.
66
The first three rows show the real part of various Gabor wavelets. In the first column,
these all have the same Gaussian envelope, but different frequencies. In the second column,
the frequencies correspond to those of the first column but the width of each of the Gaussian
envelopes is inversely proportional to the wavelet’s frequency, so this set of wavelets form a
self-similar set (i.e. all are simple dilations of each other). The bottom row shows Fourier
power spectra of the corresponding complex wavelets.
67
2D Gabor Wavelet: Real Part 2D Fourier Transform
1
1
0.5
0.5
Z
Z
0
0
0.5 10
0.5 10
Sp )
Po ati
al PD
sit es Fre 0 0 cy (C
ion 0 0 e gre qu n
in D en ue
De in cy req
gre ion (C - lF
es -0. sit PD 10 -10 atia
5 -0.5 Po ) Sp
Figure 1: The real part of a 2-D Gabor wavelet, and its 2-D Fourier transform.
These properties are particularly useful for texture analysis because of the 2-D spectral
specificity of texture as well as its variation with 2-D spatial position. A rapid method
for obtaining the required coefficients on these elementary expansion functions for the pur-
pose of representing any image completely by its “2-D Gabor Transform,” despite the non-
orthogonality of the expansion basis, is possible through the use of a relaxation neural net-
work. A large and growing literature now exists on the efficient use of this non-orthogonal
expansion basis and its applications.
Two-dimensional Gabor filters over the image domain (x, y) have the functional form
68
F (u, v) of a 2-D Gabor filter has exactly the same functional form, with parameters just
interchanged or inverted:
Hence the Gabor expansion basis “contains” both domains at once. It allows us to make
a continuous deformation that selects a representation lying anywhere on a one-parameter
continuum between two domains that were hitherto distinct and mutually unapproachable.
A new Entente Cordiale, indeed.
69
Reconstruction of Lena: 25, 100, 500, and 10,000 Two-Dimensional Gabor Wavelets
70
11 Kolmogorov Complexity and Minimal Description Length
An idea of fundamental importance is the measure known as Kolmogorov complexity: the
complexity of a string of data is defined as the length of the shortest binary program for
computing the string. Thus the complexity is the data’s “minimal description length.”
It is an amazing fact that the Kolmogorov complexity K of a string is approximately
equal to the entropy H of the distribution from which the string is a randomly drawn
sequence. Thus Kolmogorov descriptive complexity is intimately connected with informa-
tion theory, and indeed K defines the ultimate data compression. Reducing the data to a
program that generates it exactly is obviously a way of compressing it; and running that
program is a way of decompressing it. Any set of data can be generated by a computer
program, even if (in the worst case) that program simply consists of data statements. The
length of such a program defines its algorithmic complexity.
It is important to draw a clear distinction between the notions of computational com-
plexity (measured by program execution time), and algorithmic complexity (measured by
program length). Kolmogorov complexity is concerned with finding descriptions which
minimize the latter. Little is known about how (in analogy with the optimal properties
of Gabor’s elementary logons in the 2D Information Plane) one might try to minimize
simultaneously along both of these orthogonal axes that form a “Complexity Plane.”
Most sequences of length n (where “most” considers all possible permutations of n
bits) have Kolmogorov complexity K close to n. The complexity of a truly random binary
sequence is as long as the sequence itself. However, it is not clear how to be certain of
discovering that a given string has a much lower complexity than its length. It might be
clear that the string
0101010101010101010101010101010101010101010101010101010101010101
has a complexity much less than 32 bits; indeed, its complexity is the length of the program:
Print 32 "01"s. But consider the string
0110101000001001111001100110011111110011101111001100100100001000
which looks random and passes most tests for randomness. How could you √ discover that
this sequence is in fact just the binary expansion for the irrational number 2 − 1, and that
therefore it can be specified extremely concisely?
Fractals are examples of entities that look very complex but in fact are generated by
very simple programs (i.e. iterations of a mapping). Therefore, the Kolmogorov complexity
of fractals is nearly zero.
A sequence x1 , x2 , x3 , ..., xn of length n is said to be algorithmically random if its Kol-
mogorov complexity is at least n (i.e. the shortest possible program that can generate the
sequence is a listing of the sequence itself):
71
tests for randomness. (Otherwise, identifying the statistical test for randomness that the
string failed would reduce the descriptive complexity of the string, which contradicts its
incompressibility.) Therefore the algorithmic test for randomness is the ultimate test, since
it includes within it all other computable tests for randomness.
12 A Short Bibliography
Cover, T M and Thomas, J A (1991) Elements of Information Theory. New York: Wiley.
Blahut, R E (1987) Principles and Practice of Information Theory. New York: Addison-
Wesley.
Bracewell, R (1995) The Fourier Transform and its Applications. New York: McGraw-
Hill.
Fourier analysis: besides the above, an excellent on-line tutorial is available at:
http://users.ox.ac.uk/˜ball0597/Fourier/
72