densityfunctionforthesumisgivenbytheconvolution:

ZZr?x1xnp?y1yn?q?x1?y1xn?yn?dy1dyn?

Physicallythiscorrespondstoaddingthenoisesorsignalsrepresentedbythe
originalensemblesoffunctions.
ThefollowingresultisderivedinAppendix6.Theorem15:Lettheaveragepowerof
twoensemblesbeN1andN2andlettheirentropypowersbeN1
andN2.Thentheentropypowerofthesum,N3,isboundedbyN1?N2?N3?N1?N2?
WhiteGaussiannoisehasthepeculiarpropertythatitcanabsorbanyothernoiseor
signalensemblewhichmaybeaddedtoitwitharesultantentropypower
approximatelyequaltothesumofthewhitenoisepowerandthesignalpower
(measuredfromtheaveragesignalvalue,whichisnormallyzero),providedthe
signalpowerissmall,inacertainsense,comparedtonoise.
Considerthefunctionspaceassociatedwiththeseensembleshavingndimensions.
ThewhitenoisecorrespondstothesphericalGaussiandistributioninthisspace.The
signalensemblecorrespondstoanotherprobabilitydistribution,notnecessarily
Gaussianorspherical.Letthesecondmomentsofthisdistributionaboutitscenterof
gravitybeaij.Thatis,ifp?x1xn?isthedensitydistributionfunction
ZZaijp?xii??xj j?dx1dxn

wherethe?iarethecoordinatesofthecenterofgravity.Nowaijisapositivedefinite
quadraticform,andwecanrotateourcoordinatesystemtoalignitwiththeprincipal
directionsofthisform.aijisthenreducedtodiagonalformbii.Werequirethateach
biibesmallcomparedtoN,thesquaredradiusofthesphericaldistribution.
Inthiscasetheconvolutionofthenoiseandsignalproduceapproximatelya
Gaussiandistributionwhosecorrespondingquadraticformis

Theentropypowerofthisdistributionis
orapproximately
h?
i1?n?

h??

N?bii?
N
?

b
ii

i1?nThelasttermisthesignalpower,whilethefirstisthenoisepower.

PARTIV:THECONTINUOUSCHANNEL24.THECAPACITYOFA
CONTINUOUSCHANNEL

Inacontinuouschanneltheinputortransmittedsignalswillbecontinuousfunctions
oftimef?t?belongingtoacertainset,andtheoutputorreceivedsignalswillbe
perturbedversionsofthese.Wewillconsideronlythecasewherebothtransmitted
andreceivedsignalsarelimitedtoacertainbandW.Theycanthenbespecified,for
atimeT,by2TWnumbers,andtheirstatisticalstructurebyfinitedimensional
distributionfunctions.Thusthestatisticsofthetransmittedsignalwillbedetermined
by
P?x1xn??P?x?41
n

ii

?
1

N?N?b?
??

N
?1

iin

?
?

andthoseofthenoisebytheconditionalprobabilitydistributionPx1?????xn?y1yn??
Px?y??
Therateoftransmissionofinformationforacontinuouschannelisdefinedinaway
analogoustothatforadiscretechannel,namely
R?H?x Hy?x?
whereH?x?istheentropyoftheinputandHy?x?theequivocation.Thechannel
capacityCisdefinedasthemaximumofRwhenwevarytheinputoverallpossible
ensembles.ThismeansthatinafinitedimensionalapproximationwemustvaryP?x?
P?x1xn?andmaximize
Thiscanbewritten
ZZZ
?ZZ

P?x?y?
yP?y?
P
x
logP
x
dx
P
x
log
dxdy
?
?

?
?
?
?
?
?
?

P?x?y?logZZZ
dxdy
PP?x?P?y?
?

x
y
?
?

usingthefactthatfollows:
P?x?y?logP?x?dxdy?P?x?logP?x?dx.Thechannelcapacityisthusexpressedas
ZZ

1PC?LimMaxP?x?y?logdxdy?
?T?P ?x?TP?x?P?y?

x
y
?

ItisobviousinthisformthatRandCareindependentofthecoordinatesystem
sincethenumeratorP?x?y?
anddenominatorinlog
willbemultipliedbythesamefactorswhenxandyaretransformedin
P?x?P?y?anyonetooneway.ThisintegralexpressionforCismoregeneralthanH?x
Hy?x?.Properlyinterpreted(seeAppendix7)itwillalwaysexistwhileH?x Hy?x?may
assumeanindeterminateform? insomecases.Thisoccurs,forexample,ifxis
limitedtoasurfaceoffewerdimensionsthanninitsndimensionalapproximation.
IfthelogarithmicbaseusedincomputingH?x?andHy?x?istwothenCisthe
maximumnumberofbinarydigitsthatcanbesentpersecondoverthechannelwith
arbitrarilysmallequivocation,justasinthediscretecase.Thiscanbeseen
physicallybydividingthespaceofsignalsintoalargenumberofsmallcells,
sufficientlysmallsothattheprobabilitydensityPx?y?ofsignalxbeingperturbedto
pointyissubstantiallyconstantoveracell(eitherofxory).Ifthecellsare
consideredasdistinctpointsthesituationisessentiallythesameasadiscrete
channelandtheproofsusedtherewillapply.Butitisclearphysicallythatthis
quantizingofthevolumeintoindividualpointscannotinanypracticalsituationalter
thefinalanswersignificantly,providedtheregionsaresufficientlysmall.Thusthe
capacitywillbethelimitofthecapacitiesforthediscretesubdivisionsandthisis
justthecontinuouscapacitydefinedabove.
Onthemathematicalsideitcanbeshownfirst(seeAppendix7)thatifuisthe
message,xisthesignal,yisthereceivedsignal(perturbedbynoise)andvisthe
recoveredmessagethen
H?x Hy?x H?u Hv?u?
regardlessofwhatoperationsareperformedonutoobtainxoronytoobtainv.
Thusnomatterhowweencodethebinarydigitstoobtainthesignal,orhowwe
decodethereceivedsignaltorecoverthemessage,thediscreterateforthebinary
digitsdoesnotexceedthechannelcapacitywehavedefined.Ontheotherhand,itis
possibleunderverygeneralconditionstofindacodingsystemfortransmitting
binarydigitsattherateCwithassmallanequivocationorfrequencyoferrorsas
desired.Thisistrue,forexample,if,whenwetakeafinitedimensional
approximatingspaceforthesignalfunctions,P?x?y?iscontinuousinbothxandy

exceptatasetofpointsofprobabilityzero.
Animportantspecialcaseoccurswhenthenoiseisaddedtothesignalandis
independentofit(intheprobabilitysense).ThenPx?y?isafunctiononlyofthe
differencen??y?x?,
Px?y??Q?y?x?42
andwecanassignadefiniteentropytothenoise(independentofthestatisticsofthe
signal),namelytheentropyofthedistributionQ?n?.Thisentropywillbedenotedby
H?n?.
Theorem16:
Ifthesignalandnoiseareindependentandthereceivedsignalisthesumofthetransmitted
signalandthenoisethentherateoftransmissionis
R?H?y H?n??
i.e.,theentropyofthereceivedsignallesstheentropyofthenoise.Thechannel
capacityis
C?MaxH?y H?n??P?x?
Wehave,sincey?x?n:
H?x?y??H?x?n??Expandingtheleftsideandusingthefactthatxandnare
independent
Hence
H?y Hy?x??H?x H?n??
R?H?x Hy?x??H?y H?n??
SinceH?n?isindependentofP?x?,maximizingRrequiresmaximizingH?y?,the
entropyofthereceivedsignal.Iftherearecertainconstraintsontheensembleof
transmittedsignals,theentropyofthereceivedsignalmustbemaximizedsubjectto
theseconstraints.
25.CHANNELCAPACITYWITHANAVERAGEPOWERLIMITATION
AsimpleapplicationofTheorem16isthecasewhenthenoiseisawhitethermal
noiseandthetransmittedsignalsarelimitedtoacertainaveragepowerP.Thenthe
receivedsignalshaveanaveragepowerP?NwhereNistheaveragenoisepower.

Themaximumentropyforthereceivedsignalsoccurswhentheyalsoformawhite
noiseensemblesincethisisthegreatestpossibleentropyforapowerP?Nandcan
beobtainedbyasuitablechoiceoftransmittedsignals,namelyiftheyformawhite
noiseensembleofpowerP.Theentropy(persecond)ofthereceivedensembleis
then
H?y??Wlog2?e?P?N??
H?n??Wlog2?eN?
Summarizingwehavethefollowing:Theorem17:Thecapacityofachannelofband
WperturbedbywhitethermalnoisepowerNwhenthe
averagetransmitterpowerislimitedtoPisgivenbyC?Wlog
Thismeansthatbysufficientlyinvolvedencodingsystemswecantransmitbinary
digitsattherate
P?NN
higherratebyanyencodingsystemwithoutadefinitepositivefrequencyoferrors.
Toapproximatethislimitingrateoftransmissionthetransmittedsignalsmust
approximate,instatisticalproperties,awhitenoise.6Asystemwhichapproachesthe
idealratemaybedescribedasfollows:Let
6

Thisandotherpropertiesofthewhitenoisecasearediscussedfromthegeometricalpointofview
inCommunicationinthePresenceofNoise,loc.cit.

andthenoiseentropyis
Thechannelcapacityis
CHy Hn

P?N

?????Wlog?N

P?NN
?

Wlog2

bitspersecond,witharbitrarilysmallfrequencyoferrors.Itisnotpossibleto
transmitata
43
M?2ssamplesofwhitenoisebeconstructedeachofdurationT.Theseareassigned
binarynumbersfrom0toM?1.Atthetransmitterthemessagesequencesarebroken
upintogroupsofsandforeachgroupthecorrespondingnoisesampleistransmitted
asthesignal.AtthereceivertheMsamplesareknownandtheactualreceivedsignal
(perturbedbynoise)iscomparedwitheachofthem.Thesamplewhichhastheleast
R.M.S.discrepancyfromthereceivedsignalischosenasthetransmittedsignaland
thecorrespondingbinarynumberreconstructed.Thisprocessamountstochoosing
themostprobable(aposteriori)signal.ThenumberMofnoisesamplesusedwill
dependonthetolerablefrequency?oferrors,butforalmostallselectionsofsamples
wehave
?Wlog

forthewhitenoisecasehavebeendevelopedindependently
byseveralotherwriters,althoughwithsomewhatdifferentinterpretations.Wemay
mentiontheworkofN.Wiener,7W.G.Tuller,8andH.Sullivaninthisconnection.
Inthecaseofanarbitraryperturbingnoise(notnecessarilywhitethermalnoise)it
doesnotappearthatthemaximizingprobleminvolvedindeterminingthechannel
capacityCcanbesolvedexplicitly.However,upperandlowerboundscanbesetfor
CintermsoftheaveragenoisepowerNthenoiseentropypowerN1.Thesebounds
aresufficientlyclosetogetherinmostpracticalcasestofurnishasatisfactory
solutiontotheproblem.
Theorem18:ThecapacityofachannelofbandWperturbedbyanarbitrarynoiseis
boundedbytheinequalities
logM???T0T?T

P?NN
LimLimsothatnomatterhowsmall?ischosen,wecan,bytakingTsufficiently
large,transmitasnearaswewish
?

P?NNFormulassimilartoC?Wlog
toTWlog
binarydigitsinthetimeT.P?N

N
P?N1 WlogN1
?

P?NN1
CWlog

where
P?averagetransmitterpowerN?averagenoisepower
N1?entropypowerofthenoise.
HereagaintheaveragepoweroftheperturbedsignalswillbeP?N.Themaximum
entropyforthispowerwouldoccurifthereceivedsignalwerewhitenoiseand
wouldbeWlog2?e?P?N?.Itmaynotbepossibletoachievethis;i.e.,theremaynotbe
anyensembleoftransmittedsignalswhich,addedtotheperturbingnoise,producea
whitethermalnoiseatthereceiver,butatleastthissetsanupperboundtoH?y?.We
have,therefore
C?MaxH?y H?n??Wlog2?e?P?N Wlog2?eN1?
Thisistheupperlimitgiveninthetheorem.Thelowerlimitcanbeobtainedby
consideringtherateifwemakethetransmittedsignalawhitenoise,ofpowerP.In
thiscasetheentropypowerofthereceivedsignalmustbeatleastasgreatasthatof
awhitenoiseofpowerP?N1sincewehaveshownininaprevioustheoremthatthe
entropypowerofthesumoftwoensemblesisgreaterthanorequaltothesumofthe
individualentropypowers.Hence
MaxH?y Wlog2?e?P?N1?
7

Cybernetics,loc.cit.

TheoreticalLimitationsontheRateofTransmissionofInformation,ProceedingsoftheInstitute
ofRadioEngineers,v.37,No.5,May,1949,pp.46878.

44
and
Ifthenoiseisitselfwhite,N?N1andtheresultreducestotheformulaproved
previously:? ?
P
C?Wlog2?e?P?N1 Wlog2?eN1
P?N1?Wlog?N1
AsPincreases,theupperandlowerboundsapproacheachother,sowehaveasan
asymptoticrate
P?NN1
Wlog
?

C?Wlog1?IfthenoiseisGaussianbutwithaspectrumwhichisnotnecessarilyflat,
N1isthegeometricmeanof
thenoisepoweroverthevariousfrequenciesinthebandW.Thus1Z
N1?explogN?f?dfWW
whereN?f?isthenoisepoweratfrequencyf.Theorem19:Ifwesetthecapacityfor
agiventransmitterpowerPequalto
P?N??C?WlogN1
then?ismonotonicdecreasingasPincreasesandapproaches0asalimit.Suppose
thatforagivenpowerP1thechannelcapacityis
P1?N1N1
Thismeansthatthebestsignaldistribution,sayp?x?,whenaddedtothenoise
distributionq?x?,givesareceiveddistributionr?y?whoseentropypoweris?P1?N1?.

LetusincreasethepowertoP1Pbyaddingawhitenoiseofpower?Ptothesignal.
Theentropyofthereceivedsignalisnowatleast
H?y??Wlog2?e?P1?N 1 P?
byapplicationofthetheoremontheminimumentropypowerofasum.Hence,
sincewecanattaintheHindicated,theentropyofthemaximizingdistributionmust
beatleastasgreatand?mustbemonotonicdecreasing.Toshowthat 0asP?
c onsiderasignalwhichiswhitenoisewithalargeP.Whatevertheperturbing
noise,thereceivedsignalwillbeapproximatelyawhitenoise,ifPissufficiently
large,inthesenseofhavinganentropypowerapproachingP?N.
26.THECHANNELCAPACITYWITHAPEAKPOWERLIMITATION
Insomeapplicationsthetransmitterislimitednotbytheaveragepoweroutputbut
bythepeakinstantaneouspower.Theproblemofcalculatingthechannelcapacityis
thenthatofmaximizing(byvariationoftheensembleoftransmittedsymbols)
H?y H?n?subjecttotheconstraintthatallthefunctionsf?t?intheensemblebeless
thanorequaltoS,say,forall
t.Aconstraintofthistypedoesnotworkoutaswellmathematicallyastheaverage
powerlimitation.TheS
?

Wlog
?
p

mostwehaveobtainedforthiscaseisalowerboundvalidforall
SSforlarge)andanasymptoticvalueofCforsmall.
,anasymptoticupperbound(valid

NN
45
Theorem20:ThechannelcapacityCforabandWperturbedbywhitethermalnoise
ofpowerNisboundedby
samplesareindependentandhaveadistributionfunctionwhichisconstantfromcan
becalculatedas
Wlog4S?Thereceivedsignalwillthenhaveanentropylessthan
Sto
?

S.Theentropy
2S

C?

Wlog e3N

SwhereSisthepeakallowedtransmitterpower.Forsufficientlylarge
N
??

SN
wemaximizetheentropyofthetransmittedensemble.Theasymptoticupperbound
isobtainedbyrelaxingtheconditionsontheensemble.Letussupposethat
thepowerislimitedtoSnotateveryinstantoftime,butonlyatthesamplepoints.
Themaximumentropyof
thetransmittedensembleundertheseweakenedconditionsiscertainlygreaterthan
orequaltothatunderthe
originalconditions.Thisalteredproblemcanbesolvedeasily.Themaximum

entropyoccursifthedifferent
S?NC?Wlog?e?1

where?isarbitrarilysmall.As

N
?

??

S
N ?0(andprovidedthebandWstartsat0)
Wewishtomaximizetheentropyofthereceivedsignal.If
islargethiswilloccurverynearlywhen
CWlog1N
1
?

S
Wlog?4S?2?eN??1with ? a ndthechannelcapacityisobtainedbysubtracting
theentropyofthewhitenoise,
0as
NWlog2?eN:
?
pp
2

S?N

Wlog?4S?2?eN??1Wlog?2?eN Wlog?e
?1

Thisisthedesiredupperboundtothechannelcapacity.Toobtainalowerbound
considerthesameensembleoffunctions.Letthesefunctionsbepassedthrough
anidealfilterwithatriangulartransfercharacteristic.Thegainistobeunityat
frequency0anddeclinelinearlydowntogain0atfrequencyW.Wefirstshowthat
theoutputfunctionsofthefilterhaveapeak
powerlimitationSatalltimes(notjustthesamplepoints).Firstwenotethatapulse
thefilterproduces
1sin2?Wt2??Wt?2
sin2?Wt2?Wt
goinginto
N

intheoutput.Thisfunctionisnevernegative.Theinputfunction(inthegeneral
case)canbethoughtofasthesumofaseriesofshiftedfunctions
sin2?Wta
2?Wtp
wherea,theamplitudeofthesample,isnotgreaterthanofthenonnegativeform
abovewiththesamecoefficients.Thesefunctionsbeingnonnegative,thegrea ptest
positivevalueforanytisobtainedwhenallthecoefficientspahavetheirmaximum
positivevalues,i.e.,S.InthiscasetheinputfunctionwasaconstantofamplitudeS
andsincethefilterhasunitgainforD.C.,theoutputisthesame.Hencetheoutput
ensemblehasapeakpowerS.

S.Hencetheoutputisthesumofshiftedfunctions

46
Theentropyoftheoutputensemblecanbecalculatedfromthatoftheinput
ensemblebyusingthetheoremdealingwithsuchasituation.Theoutputentropyis
equaltotheinputentropyplusthegeometrical

meangainofthefilter:
Hencetheoutputentropyis
Z
ZW?

W?f?2

logG2df00W
??
?

logdf
2W
?

Wlog4S
?Wlog

e22S
?

4S

?2W

andthechannelcapacityisgreaterthan
SN
Wlog

Wenowwishtoshowthat,forsmallcapacityisapproximately
?e3N(peaksignalpoweroveraveragewhitenoisepower),thechannel

??

C?Wlog1? S S
?
??

SN
MorepreciselyC Wlog1NN
StothepeakS,itfollowsthatforall
N? ?C?Wlog1 Wlog1

PS

NN
? ?Therefore,ifwecanfindanensembleoffunctionssuchthattheycorrespondtoa

S
ratenearlyWlog1?
N

andarelimitedtobandWandpeakStheresultwillbeproved.Considerthe
ensembleoffunctionsofthepp
??

1as
?

0.SincetheaveragesignalpowerPislessthanorequal

followingtype.Aseriesoftsampleshavethesamevalue,eitherthesamevalue,etc.
Thevalueforaseriesischosenatrandom,probability1for
?

Sor
S,thenthenexttsampleshavepp
??

Sandthisensemblebepassedthroughafilterwithtriangulargaincharacteristic(unit
gainatD.C.),theoutputispeaklimitedto?S.Furthermoretheaveragepoweris
nearlySandcanbemadetoapproachthisbytakingtsufficientlylarge.Theentropy
ofthesumofthisandthethermalnoisecanbefoundbyapplyingthetheorem
122

S.If
for

onthesumofanoiseandasmallsignal.Thistheoremwillapplyif
pS

tN
SN

smallenough(aftertischosen).TheentropypowerwillbeS?Ntoasclosean
approximationasdesired,andhencetherateoftransmissionasnearaswewish
issufficientlysmall.Thiscanbeensuredbytaking
to?Wlog

PARTV:THERATEFORACONTINUOUSSOURCE
27.FIDELITYEVALUATIONFUNCTIONS
Inthecaseofadiscretesourceofinformationwewereabletodetermineadefinite
rateofgeneratinginformation,namelytheentropyoftheunderlyingstochastic
process.Withacontinuoussourcethesituationisconsiderablymoreinvolved.In
thefirstplaceacontinuouslyvariablequantitycanassumeaninfinitenumberof
valuesandrequires,therefore,aninfinitenumberofbinarydigitsforexact
specification.Thismeansthattotransmittheoutputofacontinuoussourcewith
exactrecoveryatthereceivingpointrequires,
SN

N
?

47
ingeneral,achannelofinfinitecapacity(inbitspersecond).Since,ordinarily,
channelshaveacertainamountofnoise,andthereforeafinitecapacity,exact
transmissionisimpossible.
This,however,evadestherealissue.Practically,wearenotinterestedinexact
transmissionwhenwehaveacontinuoussource,butonlyintransmissiontowithina
certaintolerance.Thequestionis,canweassignadefiniteratetoacontinuous
sourcewhenwerequireonlyacertainfidelityofrecovery,measuredinasuitable
way.Ofcourse,asthefidelityrequirementsareincreasedtheratewillincrease.It
willbeshownthatwecan,inverygeneralcases,definesucharate,havingthe
propertythatitispossible,byproperlyencodingtheinformation,totransmititover
achannelwhosecapacityisequaltotherateinquestion,andsatisfythefidelity
requirements.Achannelofsmallercapacityisinsufficient.
Itisfirstnecessarytogiveageneralmathematicalformulationoftheideaoffidelity
oftransmission.Considerthesetofmessagesofalongduration,sayTseconds.The
sourceisdescribedbygivingtheprobabilitydensity,intheassociatedspace,thatthe
sourcewillselectthemessageinquestionP?x?.Agivencommunicationsystemis
described(fromtheexternalpointofview)bygivingtheconditionalprobability
Px?y?thatifmessagexisproducedbythesourcetherecoveredmessageatthe
receivingpointwillbey.Thesystemasawhole(includingsourceandtransmission
system)isdescribedbytheprobabilityfunctionP?x?y?ofhavingmessagexandfinal
outputy.Ifthisfunctionisknown,thecompletecharacteristicsofthesystemfrom
thepointofviewoffidelityareknown.Anyevaluationoffidelitymustcorrespond
mathematicallytoanoperationappliedtoP?x?y?.Thisoperationmustatleasthave
thepropertiesofasimpleorderingofsystems;i.e.,itmustbepossibletosayoftwo
systemsrepresentedbyP1?x?y?andP2?x?y?that,accordingtoourfidelitycriterion,
either(1)thefirsthashigherfidelity,(2)thesecondhashigherfidelity,or(3)they
haveequalfidelity.Thismeansthatacriterionoffidelitycanberepresentedbya
numericallyvaluedfunction:
??vP?x?y?
??writteninaseeminglymuchmorespecializedform,namelyasanaverageofa

function??x?y?overtheset
whoseargumentrangesoverpossibleprobabilityfunctionsP?x?y?.Wewillnow
showthatunderverygeneralandreasonableassumptionsthefunctionvP?x?y?can
be
ofpossiblevaluesofxandy:
??vP?x?y
ZZ

P?x?y???x?y?dxdy?
Toobtainthisweneedonlyassume(1)thatthesourceandsystemareergodicso
thataverylongsamplewillbe,withprobabilitynearly1,typicaloftheensemble,
and(2)thattheevaluationisreasonableinthesensethatitispossible,by
observingatypicalinputandoutputx1andy1,toformatentativeevaluationonthe
basisofthesesamples;andifthesesamplesareincreasedindurationthetentative
evaluationwill,withprobability1,approachtheexactevaluationbasedonafull
knowledgeofP?x?y?.Letthetentativeevaluationbe??x?y?.Thenthefunction??x?y?
approaches(asT?)aconstantforalmostall?x?y?whichareinthehighprobability
regioncorrespondingtothesystem:
andwemayalsowrite
since
??
??x?y P?x?y???x?y?dxdyZZ

P?x?y?dxdy?1?
??

y
x
??

vP
x

y
?ZZ
?
?
?

Thisestablishesthedesiredresult.Thefunction??x?y?
hasthegeneralnatureofadistancebetweenxandy.9Itmeasureshowundesirable
itis(accordingtoourfidelitycriterion)toreceiveywhenxistransmitted.The
generalresultgivenabovecanberestatedasfollows:Anyreasonableevaluationcan
berepresentedasanaverageofadistancefunctionoverthesetofmessagesand
recoveredmessagesxandyweightedaccordingtotheprobabilityP?x?y?ofgetting
thepairinquestion,providedthedurationTofthemessagesbetakensufficiently
large.
Thefollowingaresimpleexamplesofevaluationfunctions:9Itisnotametricinthe
strictsense,however,sinceingeneralitdoesnotsatisfyeither ??x?y????y?x?or??x?y????y?z????x?
z?.

48
functionspace.
ZT?

?2
??
?
??

x
t
?
?

??
?
??

x
t
?
?

x
y
tT0
??

y
dt
x
x
y
1
f
t
dt
??
?
??
?

?
?
?
?

v
?

x
t
??

y
t
??
??
2

1.R.M.S.criterion.
Inthisverycommonlyusedmeasureoffidelitythedistancefunction??x?y?is(apart
fromaconstantfactor)thesquareoftheordinaryEuclideandistancebetweenthe
pointsxandyintheassociated
?
?

2.FrequencyweightedR.M.S.criterion.Moregenerallyonecanapplydifferent
weightstothedifferentfrequencycomponentsbeforeusinganR.M.S.measureof
fidelity.Thisisequivalenttopassingthedifferencex?t y?t?throughashapingfilter
andthendeterminingtheaveragepowerintheoutput.Thuslet
and
then

3.Absoluteerrorcriterion.
e?t x?t y?t?Z
f?t
ek?td?
?
?
?Z
T
2T0

ZT?
?

1
y

tT0
??

y
dt
4.

Thestructureoftheearandbraindetermineimplicitlyanevaluation,orratheranumberofeva
luations,appropriateinthecaseofspeechormusictransmission.Thereis,for
example,anintelligibilitycriterioninwhich??x?y?isequaltotherelative
frequencyofincorrectlyinterpretedwordswhenmessagex?t?isreceivedasy?t?.
Althoughwecannotgiveanexplicitrepresentationof??x?y?inthesecasesitcould,
inprinciple,bedeterminedbysufficientexperimentation.Someofitsproperties
followfromwellknownexperimentalresultsinhearing,e.g.,theearisrelatively
insensitivetophaseandthesensitivitytoamplitudeandfrequencyisroughly
logarithmic.

5.

Thediscretecasecanbeconsideredasaspecializationinwhichwehavetacitlyassumedanev
aluationbasedonthefrequencyoferrors.Thefunction??x?y?isthendefinedasthe
numberofsymbolsinthesequenceydifferingfromthecorrespondingsymbolsinx

dividedbythetotalnumberofsymbolsinx.28.THERATEFORASOURCE
RELATIVETOAFIDELITYEVALUATION
Wearenowinapositiontodefinearateofgeneratinginformationforacontinuous
source.WearegivenP?x?forthesourceandanevaluationvdeterminedbya
distancefunction??x?y?whichwillbeassumedcontinuousinbothxandy.Witha
particularsystemP?x?y?thequalityismeasuredby
ZZFurthermoretherateofflowofbinarydigitscorrespondingtoP?x?y?is
ZZ

R?WedefinetherateR1ofgeneratinginformationforagivenqualityv1of
reproductiontobetheminimumof
Rwhenwekeepvfixedatv1andvaryPx?y?.Thatis:ZZ
P?x?y?P?x?P?y?
R1?MinPx?y?
P?x?y?log
dxdy
v?
??x?y?P?x?y?dxdy?

P?x?y?P?x?y?logdxdy?P?x?P?y?
?
?

49