Chapter 3

Cramer-Rao Lower Bound

Abbreviated: CRLB or sometimes just CRB
CRLB is a lower bound on the variance oI any !"#$%&'(
estimator:
The CRLB tells us the best we can ever expect to be able to do
(w/ an unbiased estimator)
then , oI estimator unbiased an is
`
II ! !
) ( ) ( ) ( ) (
` ` `
2
`
! ! " ! ! "
! ! !
!
!"#$ !"#$ # $ #
What is the Cramer-Rao Lower Bound
1. Feasibility studies ( e.g. Sensor useIulness, etc.)
Can we meet our speciIications?
2. Judgment oI proposed estimators
Estimators that don`t achieve CRLB are looked
down upon in the technical literature
3. Can sometimes provide Iorm Ior MVU est.
4. Demonstrates importance oI physical and/or signal
parameters to the estimation problem
e.g. We`ll see that a signal`s BW determines delay est. accuracy
$ Radars should use wide BW signals
Some Uses of the CRLB
Q: What determines how well you can estimate ! ?
Recall: Data vector is x
3.3 Est. Accuracy Consideration
samples Irom a random
process that depends on an !
$ the PDF describes that
dependence: %(x;! )
Clearly iI %(x;! ) depends strongly/weakly on !
.we should be able to estimate ! well/poorly.
See surIace plots vs. x & ! Ior 2 cases:
1. Strong dependence on !
2. Weak dependence on !
$ Should look at %(x;! ) as a Iunction oI ! Ior
Iixed value oI observed data x
Surface Plot Examples of )(x;! )
% &
'
'
(
)
*
*
+
,
-
- .
2
2
2
2
) | 0 | (
exp
2
1
|; 0 |
"
/"
& '
& ' %
'|0| & (|0|
Ex. 3.1: PDF Dependence for DC Level in Noise
(|0| ~ )(0,"
2
)
Then the parameter-dependent PDF oI the data point '|0| is:
&
'|0|
& 3
%('|0|3;! )
Say we observe *0] 3.
So ~Slice at *0] 3
The LF the PDF %(x;! )
.but as a Iunction oI parameter ! w/ the data vector x Iixed
Define: Likelihood Function (LF)
We will also oIten need the Log Likelihood Function (LLF):
LLF lnLF} ln %(x;! )}
LF Characteristics that Affect Accuracy
Intuitively: 'sharpness oI the LF sets accuracy. But How???
Sharpness is measured using curvature:
% &
value true
data given
2
2
; ln
.
. 0
0 -
!
!
!
x
x %
Curvature ! $ PDF concentration ! $ Accuracy !
But this is Ior a particular set oI data. we want 'in general:
So.Average over random vector to give the average curvature:
% &
value true
2
2
; ln
.
1
2
1
3
4
1
5
1
6
7
0
0
-
!
!
! x %
*
'Expected sharpness
oI LF
*} is w.r.t %(x;! )
Theorem 3.1 CRLB for Scalar Parameter
Assume 'regularity condition is met: !
!
!
8 .
2
3
4
5
6
7
0
0
0
) ; ( ln ' %
*
*} is w.r.t %(x;! )
Then
% &
value true
2
2
2
`
; ln
1
.
1
2
1
3
4
1
5
1
6
7
0
0
-
#
!
!
!
!
"
x %
*
3.4 Cramer-Rao Lower Bound
% & % &
+' ' %
% %
* ) ; (
; ln ; ln
2
2
2
2
!
!
!
!
!
9
0
0
.
1
2
1
3
4
1
5
1
6
7
0
0 x x
Right-Hand
Side is
CRLB
1. Write log 1ikelihood Iunction as a Iunction oI !:
ln %(x;! )
2. Fix x and take 2
nd
partial oI LLF:
0
2
ln %(x;! )/0!
2
3. II result still depends on x:
Fix ! and take expected value w.r.t. x
Otherwise skip this step
4. Result may still depend on !:
Evaluate at each speciIic value oI ! desired.
5. Negate and Iorm reciprocal
Steps to Find the CRLB
Need likelihood Iunction:
% &
: ; % &
% &
: ; % &
'
'
'
'
(
)
*
*
*
*
+
,
- < -
.
'
'
(
)
*
*
+
,
- -
.
-
.
-
.
=
2
2
1
0
2
2
1
0
2
2
2
2
exp
2
1
2
exp
2
1
;
"
/"
"
/"
& , '
& , '
& %
)
,
)
)
,
x
Example 3.3 CRLB for DC in AWGN
'|,| & (|,|, , 0, 1, . , ) 1
(|,| ~ )(0,"
2
)
& white
Due to
whiteness
Property
oI exp
% & : ; % &
! ! ! " ! ! ! # $
! ! " ! ! # $
? (~~)
1
0
2
2
0 (~~)
2
2
2
1
2 ln ) ; ( ln
.
0
0
-
.
.
0
0
<
- -
'
'
(
)
*
*
+
,
- .
&
)
,
&
)
& , ' & %
"
/" x
Now take ln to get LLF:
: ; % & % & & '
)
& , ' & %
&
)
,
- . - .
0
0
<
-
.
2
1
0
2
1
) ; ( ln
" "
x
Now take Iirst partial w.r.t. &:
sample
mean
(!)
Now take partial again:
2 2
2
) ; ( ln
"
)
& %
&
- .
0
0
x
Doesn`t depend
on x so we don`t
need to do E}
Since the result doesn`t depend on x or & all we do is negate
and Iorm reciprocal to get CRLB:
% &
)
%
*
!"#$
2
value true
2
2
; ln
1 "
!
!
!
.
1
2
1
3
4
1
5
1
6
7
0
0
-
.
.
x
)
&
2
}
`
var
"
#
Doesn`t depend on +
Increases linearly with "
2
Decreases inversely with ,
CRLB
For Iixed )
"
2
&
CRLB
For Iixed ) & "
2
CRLB Doubling Data
Halves CRLB!
)
For Iixed "
2
Continuation of Theorem 3.1 on CRLB
There exists an unbiased estimator that attains the CRLB iff:
: ; ! !
!
!
- .
0
0
) ( ) (
) ; ( ln
x
x
. /
%
Ior some Iunctions /(! ) and .(x)
Furthermore, the estimator that achieves the CRLB is then given
by:
) (
`
x . . !
(!)
Since no unbiased estimator can do better. this
is the MVU estimate!!
This gives a possible way to Iind the MVU:
Compute 0ln %(x;! )/0! (need to anyway)
Check to see iI it can be put in
Iorm like (!)
II so. then .(x) is the MVU esimator
with
!"#$
/
. .
) (
1
}
`
var
!
!
Revisit Example 3.3 to Find MVU Estimate
% & & '
)
& %
&
- .
0
0
2
) ; ( ln
"
x
For DC Level in AWGN we Iound in (!) that:
Has Iorm oI
/(&)|g(x) &|
: ;
<
-
.
. . .
1
0
1
) (
`
)
,
, '
)
' . x !
!"#$
)
&
)
& / . . $ .
2
2
}
`
var ) (
"
"
So. for the DC Level in AWGN:
the sample mean is the MVUE!!
Definition: Efficient Estimator
An estimator that is:
unbiased and
attains the CRLB
is said to be an 'EIIicient Estimator
Notes:
Not all estimators are eIIicient (see next example: Phase Est.)
Not even all MVU estimators are eIIicient
So. there are times when our
~1
st
partial test won`t work!!!!
Example 3.4: CRLB for Phase Estimation
This is related to the DSB carrier estimation problem we used
Ior motivation in the notes Ior Ch. 1
Except here. we have a pure sinusoid and we only wish to
estimate only its phase
Signal Model:
| | ) 2 cos( | |
| ; |
, ( , 0 & , '
1
, 2
1 1
> > .
! ! ! " ! ! ! # $
?
? /
AWGN w/ zero
mean & "
2
Assumptions:
1. 0 0
1
( 0
1
is in cycles/sample)
34 & and 0
1
are known (we`ll remove this assumption later)
Signal-to-Noise Ratio:
Signal Power &
2
/2
Noise Power "
2
2
2
2"
&
5)" .
Problem: Find the CRLB Ior estimating the phase.
We need the PDF:
% &
% &
: ; % &
'
'
'
'
(
)
*
*
*
*
+
,
> - < -
.
-
.
2
2
1
0
2
2
2
) 2 cos(
exp
2
1
;
"
? /
/"
?
, 0 & , '
%
1
)
,
)
x
Exploit
Whiteness
and Exp.
Form
Now taking the log gets rid oI the exponential, then taking
partial derivative gives (see book Ior details):
% &
: ;
2
1
0
2
) 2 4 sin(
2
) 2 sin(
; ln
@
A
B
C
D
E
> - > <
-
.
0
0
-
.
? / ? /
"
?
?
, 0
&
, 0 , '
& %
1 1
)
,
x
Taking partial derivative again:
% &
: ; % & ) 2 4 cos( ) 2 cos(
; ln
1
0
2 2
2
? / ? /
" ?
?
> - > <
-
.
0
0
-
.
, 0 & , 0 , '
& %
1 1
)
,
x
Still depends on random vector x. so need E}
Taking the expected value:
% &
: ; % &
: ; F G % & ) 2 4 cos( ) 2 cos(
) 2 4 cos( ) 2 cos(
; ln
1
0
2
1
0
2 2
2
? / ? /
"
? / ? /
" ?
?
> - > < .
2
3
4
5
6
7
> - > < .
1
2
1
3
4
1
5
1
6
7
0
0
-
-
.
-
.
, 0 & , 0 , ' *
&
, 0 & , 0 , '
&
*
%
*
1 1
)
,
1 1
)
,
x
*'|,|} A cos(2/ 0
1
, 6 ? )
So. plug that in, get a cos
2
term, use trig identity, and get
% &
5)" )
)&
, 0
& %
*
)
,
1
)
,
H . I
'
'
(
)
*
*
+
,
> - .
1
2
1
3
4
1
5
1
6
7
0
0
-
< <
-
-
-
.
2
2 1
0
1
0
2
2
2
2
2
) 2 4 cos( 1
2
; ln
"
? /
" ?
? x
) ) iI
0
1
not near 0 or
,
)-1
Now. invert to get CRLB:
5)" ) H
#
1
}
`
var?
CRLB Doubling Data
Halves CRLB!
)
For Iixed 5)"
Non-dB
CRLB Doubling SNR
Halves CRLB!
5)" (non-dB)
For Iixed )
Halve CRLB
Ior every 3B
in SNR
Does an eIIicient estimator exist Ior this problem? The CRLB
theorem says there is only iI
: ; ! !
!
!
- .
0
0
) ( ) (
) ; ( ln
x
x
. /
%
% &
: ;
2
1
0
2
) 2 4 sin(
2
) 2 sin(
; ln
@
A
B
C
D
E
> - > <
-
.
0
0
-
.
? / ? /
"
?
?
, 0
&
, 0 , '
& %
1 1
)
,
x
Our earlier result was:
EIIicient Estimator does NOT exist!!!
We`ll see later though, an estimator Ior which !"#$ J }
`
var?
as ) J K or as 5)" J K
Such an estimator is called an 'asymptotically eIIicient estimator
(We`ll see such a phase estimator in Ch. 7 on MLE)
CRB
)
}
`
var?