5 cramer-rao lower bound
solohermelin
3,418 views
46 slides
Jan 11, 2015
Slide 1 of 46
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
About This Presentation
The Cramér-Rao Lower Bound on the Variance of the Estimator of continuous SISO, MIMO and Digital Systems.
Probability and Estimation are prerequisite.
For comments please connect me at [email protected].
For more presentations on different subjects visit my website at http://solohermelin.com.
Slide Content
1
Cramér – Rao Lower Bound
SOLO HERMELIN
25.09.09http://www.solohermelin.com
Cramér – Rao Lower Bound
SOLO HERMELIN
25.09.09http://www.solohermelin.com
2
Cramér-Rao Lower Bound (CRLB)SOLO
Table of Content
The Cramér-Rao Lower Bound on the Variance of the Estimator
The Cramér-Rao Lower Bound on the Variance of the Estimator –Scalar Case
The Cramér-Rao Lower Bound on the Variance of the Estimator – Multivariable Case
Gradient
Matrix Inversion Relations
Helpfully Relations
Nonrandom Parameters
Random Parameters
Nonrandom and Random Parameters Cramér – Rao Bounds
Discrete Time Nonlinear Estimation
Discrete Time Nonlinear Estimation – Recursive Cramér–Rao Lower Bound
Discrete Time Nonlinear Estimation –Special Cases
Probability Density Function of is Gaussian
0x
Additive Gaussian Noises
References
Linear System with Zero System Noise
Linear/Gaussian Systems
Cramér-Rao Lower Bound (CRLB)SOLO
Table of Content
The Cramér-Rao Lower Bound on the Variance of the Estimator
The Cramér-Rao Lower Bound on the Variance of the Estimator –Scalar Case
The Cramér-Rao Lower Bound on the Variance of the Estimator – Multivariable Case
Gradient
Matrix Inversion Relations
Helpfully Relations
Nonrandom Parameters
Random Parameters
Nonrandom and Random Parameters Cramér – Rao Bounds
Discrete Time Nonlinear Estimation
Discrete Time Nonlinear Estimation – Recursive Cramér–Rao Lower Bound
Discrete Time Nonlinear Estimation –Special Cases
Probability Density Function of is Gaussian
0x
Additive Gaussian Noises
References
Linear System with Zero System Noise
Linear/Gaussian Systems
3
Cramér-Rao Lower Bound (CRLB)
v
( )vxh,
z
x
Estimator
x
SOLO
The Cramér-Rao Lower Bound (CRLB) on the Variance of the Estimator
{}xE
- estimated mean vector
[]( ) []( ){ } { } {}{}
TTT
x xExExxExExxExE
-=--=
2
s - estimated variance matrix
For a good estimator we want
{}xxE=
- unbiased estimator vector
{ } {}{}
TT
x xExExxE
-=
2
s - minimum estimation covariance
() (){ }
T
k
kzzZ 1:
:1
= - the observation matrix after k observations
( ) () (){ }xkzzLxZL
k ,,,1,
:1 = - the Likelihood or the joint density function of Z
1:k
We have:
( )
T
pzzzz ,,,
21= ( )
T
n
xxxx ,,,
21
= ( )
T
pvvvv ,,,
21=
The estimation of , using the measurements
of a system corrupted by noise is a random variable with
xˆx z
v
( ) ( ) ( ) ()ò
== dvvpxvZpxZpxZL
vkvzkxzk ;||,
:1|:1|:1
() ()[ ]{ } () ()[ ] () ()[ ] () ()
[ ][ ] ()xbxZdxZLZx
kzdzdxkzzLkzzxkzzxE
kkk
+==
=
ò
ò
:1:1:1
,ˆ
1,,,1,,1ˆ,,1ˆ
- estimator bias()xb
therefore:
Cramér-Rao Lower Bound (CRLB)
v
( )vxh,
z
x
Estimator
x
SOLO
The Cramér-Rao Lower Bound (CRLB) on the Variance of the Estimator
{}xE
- estimated mean vector
[]( ) []( ){ } { } {}{}
TTT
x xExExxExExxExE
-=--=
2
s - estimated variance matrix
For a good estimator we want
{}xxE=
- unbiased estimator vector
{ } {}{}
TT
x xExExxE
-=
2
s - minimum estimation covariance
() (){ }
T
k
kzzZ 1:
:1
= - the observation matrix after k observations
( ) () (){ }xkzzLxZL
k ,,,1,
:1 = - the Likelihood or the joint density function of Z
1:k
We have:
( )
T
pzzzz ,,,
21= ( )
T
n
xxxx ,,,
21
= ( )
T
pvvvv ,,,
21=
The estimation of , using the measurements
of a system corrupted by noise is a random variable with
xˆx z
v
( ) ( ) ( ) ()ò
== dvvpxvZpxZpxZL
vkvzkxzk ;||,
:1|:1|:1
() ()[ ]{ } () ()[ ] () ()[ ] () ()
[ ][ ] ()xbxZdxZLZx
kzdzdxkzzLkzzxkzzxE
kkk
+==
=
ò
ò
:1:1:1
,ˆ
1,,,1,,1ˆ,,1ˆ
- estimator bias()xb
therefore:
4
Cramér-Rao Lower Bound (CRLB)
v
( )vxh,
z
x
Estimator
x
SOLO
The Cramér-Rao Lower Bound on the Variance of the Estimator –Scalar Case
[ ]{ } [ ][ ] ()xbxZdxZLZxZxE
kkkk
+==ò :1:1:1:1
,ˆˆ
We have:
[ ]{ }
[ ]
[ ] ()
x
xb
Zd
x
xZL
Zx
x
ZxE
k
k
k
k
¶
¶
+=
¶
¶
=
¶
¶
ò
1
,
ˆ
ˆ
:1
:1
:1
:1
Since L [Z
1:k
,x] is a joint density function, we have:
[ ] 1,
:1:1
=ò kk
ZdxZL
[ ] [ ] [ ]
[]0
,,
0
,
:1
:1
:1
:1
:1
:1
=
¶
¶
=
¶
¶
®=
¶
¶
òòò k
k
k
k
k
k
Zd
x
xZL
xZd
x
xZL
xZd
x
xZL
[ ]( )
[ ] ()
x
xb
Zd
x
xZL
xZx
k
k
k
¶
¶
+=
¶
¶
-ò
1
,
ˆ
:1
:1
:1
Using the fact that:
[ ]
[ ]
[ ]
x
xZL
xZL
x
xZL
k
k
k
¶
¶
=
¶
¶ ,ln
,
,
:1
:1
:1
[ ]( )[ ]
[ ] ()
x
xb
Zd
x
xZL
xZLxZx
k
k
kk
¶
¶
+=
¶
¶
-ò
1
,ln
,ˆ
:1
:1
:1:1
Cramér-Rao Lower Bound (CRLB)
v
( )vxh,
z
x
Estimator
x
SOLO
The Cramér-Rao Lower Bound on the Variance of the Estimator –Scalar Case
[ ]{ } [ ][ ] ()xbxZdxZLZxZxE
kkkk
+==ò :1:1:1:1
,ˆˆ
We have:
[ ]{ }
[ ]
[ ] ()
x
xb
Zd
x
xZL
Zx
x
ZxE
k
k
k
k
¶
¶
+=
¶
¶
=
¶
¶
ò
1
,
ˆ
ˆ
:1
:1
:1
:1
Since L [Z
1:k
,x] is a joint density function, we have:
[ ] 1,
:1:1
=ò kk
ZdxZL
[ ] [ ] [ ]
[]0
,,
0
,
:1
:1
:1
:1
:1
:1
=
¶
¶
=
¶
¶
®=
¶
¶
òòò k
k
k
k
k
k
Zd
x
xZL
xZd
x
xZL
xZd
x
xZL
[ ]( )
[ ] ()
x
xb
Zd
x
xZL
xZx
k
k
k
¶
¶
+=
¶
¶
-ò
1
,
ˆ
:1
:1
:1
Using the fact that:
[ ]
[ ]
[ ]
x
xZL
xZL
x
xZL
k
k
k
¶
¶
=
¶
¶ ,ln
,
,
:1
:1
:1
[ ]( )[ ]
[ ] ()
x
xb
Zd
x
xZL
xZLxZx
k
k
kk
¶
¶
+=
¶
¶
-ò
1
,ln
,ˆ
:1
:1
:1:1
5
Cramér-Rao Lower Bound (CRLB)
SOLO
The Cramér-Rao Lower Bound on the Variance of the Estimator – Scalar Case (continue – 1)
[ ]( )[ ]
[ ] ()
x
xb
Zd
x
xZL
xZLxZx
k
k
kk
¶
¶
+=
¶
¶
-ò
1
,ln
,ˆ
:1
:1
:1:1
Hermann Amandus
Schwarz
1843 - 1921
Let use Schwarz Inequality:
()() () ()
òòò
£ dttgdttfdttgtf
22
2
The equality occurs if and only if f (t) = k g (t)
[ ]( ) [ ]
[ ]
[ ]xZL
x
xZL
gxZLxZxf
k
k
kk
,
,ln
:&,ˆ:
:1
:1
:1:1
¶
¶
=-=choose:
[ ]( )[ ]
[ ]
()
[ ]( ) [ ]( ) [ ]
[ ]
÷
÷
ø
ö
ç
ç
è
æ
÷
÷
ø
ö
ç
ç
è
æ
¶
¶
-£
ú
û
ù
ê
ë
é
¶
¶
+=
ú
û
ù
ê
ë
é
¶
¶
-
òò
ò
k
k
kkkk
k
k
kk
Zd
x
xZL
xZLZdxZLxZx
x
xb
Zd
x
xZL
xZLxZx
:1
2
:1
:1:1:1
2
:1
2
2
:1
:1
:1:1
,ln
,,ˆ1
,ln
,ˆ
[ ]( ) [ ]
()
[ ]
[ ]
ò
ò
÷
÷
ø
ö
ç
ç
è
æ
¶
¶
ú
û
ù
ê
ë
é
¶
¶
+
³-
k
k
k
kkk
Zd
x
xZL
xZL
x
xb
ZdxZLxZx
:1
2
:1
:1
2
:1:1
2
:1
,ln
,
1
,ˆ
Cramér-Rao Lower Bound (CRLB)
SOLO
The Cramér-Rao Lower Bound on the Variance of the Estimator – Scalar Case (continue – 1)
[ ]( )[ ]
[ ] ()
x
xb
Zd
x
xZL
xZLxZx
k
k
kk
¶
¶
+=
¶
¶
-ò
1
,ln
,ˆ
:1
:1
:1:1
Hermann Amandus
Schwarz
1843 - 1921
Let use Schwarz Inequality:
()() () ()
òòò
£ dttgdttfdttgtf
22
2
The equality occurs if and only if f (t) = k g (t)
[ ]( ) [ ]
[ ]
[ ]xZL
x
xZL
gxZLxZxf
k
k
kk
,
,ln
:&,ˆ:
:1
:1
:1:1
¶
¶
=-=choose:
[ ]( )[ ]
[ ]
()
[ ]( ) [ ]( ) [ ]
[ ]
÷
÷
ø
ö
ç
ç
è
æ
÷
÷
ø
ö
ç
ç
è
æ
¶
¶
-£
ú
û
ù
ê
ë
é
¶
¶
+=
ú
û
ù
ê
ë
é
¶
¶
-
òò
ò
k
k
kkkk
k
k
kk
Zd
x
xZL
xZLZdxZLxZx
x
xb
Zd
x
xZL
xZLxZx
:1
2
:1
:1:1:1
2
:1
2
2
:1
:1
:1:1
,ln
,,ˆ1
,ln
,ˆ
[ ]( ) [ ]
()
[ ]
[ ]
ò
ò
÷
÷
ø
ö
ç
ç
è
æ
¶
¶
ú
û
ù
ê
ë
é
¶
¶
+
³-
k
k
k
kkk
Zd
x
xZL
xZL
x
xb
ZdxZLxZx
:1
2
:1
:1
2
:1:1
2
:1
,ln
,
1
,ˆ
6
Cramér-Rao Lower Bound (CRLB)SOLO
The Cramér-Rao Lower Bound on the Variance of the Estimator – Scalar Case (continue – 2)
[ ]( ) [ ]
()
[ ]
[ ]
ò
ò
÷
÷
ø
ö
ç
ç
è
æ
¶
¶
ú
û
ù
ê
ë
é
¶
¶
+
³-
k
k
k
kkk
Zd
x
xZL
xZL
x
xb
ZdxZLxZx
:1
2
:1
:1
2
:1:1
2
:1
,ln
,
1
,ˆ
This is the Cramér-Rao bound for a biased estimator
Harald Cramér
1893 – 1985
Cayampudi Radhakrishna
Rao
1920 -
[ ]{ } () [ ] 1,&ˆ
:1:1:1 =+= ò kkk ZdxZLxbxZxE
[ ]( ) [ ] [ ] [ ]{ } ()( ) [ ]
[ ] [ ]{ }( ) [ ] () [ ] [ ]{ }( )[ ]
() [ ]
1
:1:1
2
0
:1:1:1:1:1:1
2
:1:1
:1:1
2
:1:1:1
2
:1
,
,ˆˆ2,ˆˆ
,ˆˆ,ˆ
ò
òò
òò
+
-+-=
+-=-
kk
kkkkkkkk
kkkk
k
kk
ZdxZLxb
ZdxZLZxEZxxbZdxZLZxEZx
ZdxZLxbZxEZxZdxZLxZx
[ ] [ ]{ }( ) [ ]
()
[ ]
[ ]
()xb
Zd
x
xZL
xZL
x
xb
ZdxZLZxEZx
k
k
k
kkkkx
2
:1
2
:1
:1
2
:1:1
2
:1:1
2
ˆ
,ln
,
1
,ˆˆ -
÷
÷
ø
ö
ç
ç
è
æ
¶
¶
ú
û
ù
ê
ë
é
¶
¶
+
³-=
ò
ò
s
Cramér-Rao Lower Bound (CRLB)SOLO
The Cramér-Rao Lower Bound on the Variance of the Estimator – Scalar Case (continue – 2)
[ ]( ) [ ]
()
[ ]
[ ]
ò
ò
÷
÷
ø
ö
ç
ç
è
æ
¶
¶
ú
û
ù
ê
ë
é
¶
¶
+
³-
k
k
k
kkk
Zd
x
xZL
xZL
x
xb
ZdxZLxZx
:1
2
:1
:1
2
:1:1
2
:1
,ln
,
1
,ˆ
This is the Cramér-Rao bound for a biased estimator
Harald Cramér
1893 – 1985
Cayampudi Radhakrishna
Rao
1920 -
[ ]{ } () [ ] 1,&ˆ
:1:1:1 =+= ò kkk ZdxZLxbxZxE
[ ]( ) [ ] [ ] [ ]{ } ()( ) [ ]
[ ] [ ]{ }( ) [ ] () [ ] [ ]{ }( )[ ]
() [ ]
1
:1:1
2
0
:1:1:1:1:1:1
2
:1:1
:1:1
2
:1:1:1
2
:1
,
,ˆˆ2,ˆˆ
,ˆˆ,ˆ
ò
òò
òò
+
-+-=
+-=-
kk
kkkkkkkk
kkkk
k
kk
ZdxZLxb
ZdxZLZxEZxxbZdxZLZxEZx
ZdxZLxbZxEZxZdxZLxZx
[ ] [ ]{ }( ) [ ]
()
[ ]
[ ]
()xb
Zd
x
xZL
xZL
x
xb
ZdxZLZxEZx
k
k
k
kkkkx
2
:1
2
:1
:1
2
:1:1
2
:1:1
2
ˆ
,ln
,
1
,ˆˆ -
÷
÷
ø
ö
ç
ç
è
æ
¶
¶
ú
û
ù
ê
ë
é
¶
¶
+
³-=
ò
ò
s
7
Cramér-Rao Lower Bound (CRLB)
SOLO
The Cramér-Rao Lower Bound on the Variance of the Estimator – Scalar Case (continue – 3)
[ ] [ ]{ }( ) [ ]
()
[ ]
[ ]
()xb
Zd
x
xZL
xZL
x
xb
ZdxZLZxEZx
k
k
k
kkkkx
2
:1
2
:1
:1
2
:1:1
2
:1:1
2
ˆ
,ln
,
1
,ˆˆ -
÷
÷
ø
ö
ç
ç
è
æ
¶
¶
ú
û
ù
ê
ë
é
¶
¶
+
³-=
ò
ò
s
[ ]
[ ]
[ ]
[ ]
[ ]
[ ]
[ ] 0,
,ln
0
,
1,
:1:1
:1
,
,
,ln
:1
:1
:1:1
:1
:1
:1
=
¶
¶
®=
¶
¶
®= òòò
¶
¶
=
¶
¶
kk
k
xZL
x
xZL
x
xZL
k
k
kk ZdxZL
x
xZL
Zd
x
xZL
ZdxZL
k
k
k
[ ]
[ ]
[ ] [ ]
[ ]
[ ]
0,
,ln,ln
,
,ln
:1
,
:1
:1:1
:1:12
:1
2
:1
=
¶
¶
¶
¶
+
¶
¶
® òò
¶
¶
¶
¶
k
x
xZL
k
kk
kk
k
x
ZdxZL
x
xZL
x
xZL
ZdxZL
x
xZL
k
[ ] [ ]
0
,ln,ln
2
:1
2
:1
2
=
ï
þ
ï
ý
ü
ï
î
ï
í
ì
÷
÷
ø
ö
ç
ç
è
æ
¶
¶
+
þ
ý
ü
î
í
ì
¶
¶
®
¶
¶
x
xZL
E
x
xZL
E
kk
x
()
[ ]
()
()
[ ]
()xb
x
xZL
E
x
xb
xb
x
xZL
E
x
xb
k
k
x
2
2
:1
2
2
2
2
:1
2
2
,ln
1
,ln
1
-
þ
ý
ü
î
í
ì
¶
¶
ú
û
ù
ê
ë
é
¶
¶
+
-=-
ï
þ
ï
ý
ü
ï
î
ï
í
ì
÷
÷
ø
ö
ç
ç
è
æ
¶
¶
ú
û
ù
ê
ë
é
¶
¶
+
³s
Cramér-Rao Lower Bound (CRLB)
SOLO
The Cramér-Rao Lower Bound on the Variance of the Estimator – Scalar Case (continue – 3)
[ ] [ ]{ }( ) [ ]
()
[ ]
[ ]
()xb
Zd
x
xZL
xZL
x
xb
ZdxZLZxEZx
k
k
k
kkkkx
2
:1
2
:1
:1
2
:1:1
2
:1:1
2
ˆ
,ln
,
1
,ˆˆ -
÷
÷
ø
ö
ç
ç
è
æ
¶
¶
ú
û
ù
ê
ë
é
¶
¶
+
³-=
ò
ò
s
[ ]
[ ]
[ ]
[ ]
[ ]
[ ]
[ ] 0,
,ln
0
,
1,
:1:1
:1
,
,
,ln
:1
:1
:1:1
:1
:1
:1
=
¶
¶
®=
¶
¶
®= òòò
¶
¶
=
¶
¶
kk
k
xZL
x
xZL
x
xZL
k
k
kk ZdxZL
x
xZL
Zd
x
xZL
ZdxZL
k
k
k
[ ]
[ ]
[ ] [ ]
[ ]
[ ]
0,
,ln,ln
,
,ln
:1
,
:1
:1:1
:1:12
:1
2
:1
=
¶
¶
¶
¶
+
¶
¶
® òò
¶
¶
¶
¶
k
x
xZL
k
kk
kk
k
x
ZdxZL
x
xZL
x
xZL
ZdxZL
x
xZL
k
[ ] [ ]
0
,ln,ln
2
:1
2
:1
2
=
ï
þ
ï
ý
ü
ï
î
ï
í
ì
÷
÷
ø
ö
ç
ç
è
æ
¶
¶
+
þ
ý
ü
î
í
ì
¶
¶
®
¶
¶
x
xZL
E
x
xZL
E
kk
x
()
[ ]
()
()
[ ]
()xb
x
xZL
E
x
xb
xb
x
xZL
E
x
xb
k
k
x
2
2
:1
2
2
2
2
:1
2
2
,ln
1
,ln
1
-
þ
ý
ü
î
í
ì
¶
¶
ú
û
ù
ê
ë
é
¶
¶
+
-=-
ï
þ
ï
ý
ü
ï
î
ï
í
ì
÷
÷
ø
ö
ç
ç
è
æ
¶
¶
ú
û
ù
ê
ë
é
¶
¶
+
³s
8http://www.york.ac.uk/depts/maths/histstat/people/cramer.gif
Cramér-Rao Lower Bound (CRLB)
[ ]( ) [ ]
()
[ ]
()
[ ]
þ
ý
ü
î
í
ì
¶
¶
ú
û
ù
ê
ë
é
¶
¶
+
-=
ï
þ
ï
ý
ü
ï
î
ï
í
ì
÷
÷
ø
ö
ç
ç
è
æ
¶
¶
ú
û
ù
ê
ë
é
¶
¶
+
³-ò
2
:1
2
2
2
:1
2
:1:1
2
:1
,ln
1
,ln
1
,
x
xZL
E
x
xb
x
xZL
E
x
xb
ZdxZLxZx
k
k
kkk
SOLO
The Cramér-Rao Lower Bound on the Variance of the Estimator – Scalar Case (continue – 4)
()
[ ]
()
()
[ ]
()xb
x
xZL
E
x
xb
xb
x
xZL
E
x
xb
k
k
x
2
2
:1
2
2
2
2
:1
2
2
,ln
1
,ln
1
-
þ
ý
ü
î
í
ì
¶
¶
ú
û
ù
ê
ë
é
¶
¶
+
-=-
ï
þ
ï
ý
ü
ï
î
ï
í
ì
÷
÷
ø
ö
ç
ç
è
æ
¶
¶
ú
û
ù
ê
ë
é
¶
¶
+
³s
For an unbiased estimator (b (x) = 0), we have:
[ ] [ ]
þ
ý
ü
î
í
ì
¶
¶
-=
ï
þ
ï
ý
ü
ï
î
ï
í
ì
÷
÷
ø
ö
ç
ç
è
æ
¶
¶
³
2
:1
22
:1
2
,ln
1
,ln
1
x
xZL
E
x
xZL
E
k
k
x
s
Return to Table of Content
Cramér-Rao Lower Bound (CRLB)
[ ]( ) [ ]
()
[ ]
()
[ ]
þ
ý
ü
î
í
ì
¶
¶
ú
û
ù
ê
ë
é
¶
¶
+
-=
ï
þ
ï
ý
ü
ï
î
ï
í
ì
÷
÷
ø
ö
ç
ç
è
æ
¶
¶
ú
û
ù
ê
ë
é
¶
¶
+
³-ò
2
:1
2
2
2
:1
2
:1:1
2
:1
,ln
1
,ln
1
,
x
xZL
E
x
xb
x
xZL
E
x
xb
ZdxZLxZx
k
k
kkk
SOLO
The Cramér-Rao Lower Bound on the Variance of the Estimator – Scalar Case (continue – 4)
()
[ ]
()
()
[ ]
()xb
x
xZL
E
x
xb
xb
x
xZL
E
x
xb
k
k
x
2
2
:1
2
2
2
2
:1
2
2
,ln
1
,ln
1
-
þ
ý
ü
î
í
ì
¶
¶
ú
û
ù
ê
ë
é
¶
¶
+
-=-
ï
þ
ï
ý
ü
ï
î
ï
í
ì
÷
÷
ø
ö
ç
ç
è
æ
¶
¶
ú
û
ù
ê
ë
é
¶
¶
+
³s
For an unbiased estimator (b (x) = 0), we have:
[ ] [ ]
þ
ý
ü
î
í
ì
¶
¶
-=
ï
þ
ï
ý
ü
ï
î
ï
í
ì
÷
÷
ø
ö
ç
ç
è
æ
¶
¶
³
2
:1
22
:1
2
,ln
1
,ln
1
x
xZL
E
x
xZL
E
k
k
x
s
Return to Table of Content
9
Cramér-Rao Lower Bound (CRLB)
SOLO
Gradient
Gradient of a Scalar() ( )
nT
n
xxxxxL RR Î=Î ,,,
21
1
()
n
nn
x
x
L
x
L
x
L
L
x
x
x
xL RÎ
ú
ú
ú
ú
ú
ú
ú
ú
ú
û
ù
ê
ê
ê
ê
ê
ê
ê
ê
ê
ë
é
¶
¶
¶
¶
¶
¶
=
ú
ú
ú
ú
ú
ú
ú
ú
ú
û
ù
ê
ê
ê
ê
ê
ê
ê
ê
ê
ë
é
¶
¶
¶
¶
¶
¶
=Ñ
2
1
2
1
()
nxn
nnn
n
n
n
n
T
xx
x
L
xx
L
xx
L
xx
L
x
L
xx
L
xx
L
xx
L
x
L
L
xxx
x
x
x
xL RÎ
ú
ú
ú
ú
ú
ú
ú
ú
ú
û
ù
ê
ê
ê
ê
ê
ê
ê
ê
ê
ë
é
¶
¶
¶¶
¶
¶¶
¶
¶¶
¶
¶
¶
¶¶
¶
¶¶
¶
¶¶
¶
¶
¶
=
ú
û
ù
ê
ë
é
¶
¶
¶
¶
¶
¶
ú
ú
ú
ú
ú
ú
ú
ú
ú
û
ù
ê
ê
ê
ê
ê
ê
ê
ê
ê
ë
é
¶
¶
¶
¶
¶
¶
=ÑÑ
2
2
2
2
1
2
2
2
2
2
2
12
2
1
2
21
2
2
1
2
21
2
1
The Cramér-Rao Lower Bound on the Variance of the Estimator – Multivariable Case
Cramér-Rao Lower Bound (CRLB)
SOLO
Gradient
Gradient of a Scalar() ( )
nT
n
xxxxxL RR Î=Î ,,,
21
1
()
n
nn
x
x
L
x
L
x
L
L
x
x
x
xL RÎ
ú
ú
ú
ú
ú
ú
ú
ú
ú
û
ù
ê
ê
ê
ê
ê
ê
ê
ê
ê
ë
é
¶
¶
¶
¶
¶
¶
=
ú
ú
ú
ú
ú
ú
ú
ú
ú
û
ù
ê
ê
ê
ê
ê
ê
ê
ê
ê
ë
é
¶
¶
¶
¶
¶
¶
=Ñ
2
1
2
1
()
nxn
nnn
n
n
n
n
T
xx
x
L
xx
L
xx
L
xx
L
x
L
xx
L
xx
L
xx
L
x
L
L
xxx
x
x
x
xL RÎ
ú
ú
ú
ú
ú
ú
ú
ú
ú
û
ù
ê
ê
ê
ê
ê
ê
ê
ê
ê
ë
é
¶
¶
¶¶
¶
¶¶
¶
¶¶
¶
¶
¶
¶¶
¶
¶¶
¶
¶¶
¶
¶
¶
=
ú
û
ù
ê
ë
é
¶
¶
¶
¶
¶
¶
ú
ú
ú
ú
ú
ú
ú
ú
ú
û
ù
ê
ê
ê
ê
ê
ê
ê
ê
ê
ë
é
¶
¶
¶
¶
¶
¶
=ÑÑ
2
2
2
2
1
2
2
2
2
2
2
12
2
1
2
21
2
2
1
2
21
2
1
The Cramér-Rao Lower Bound on the Variance of the Estimator – Multivariable Case
10
Cramér-Rao Lower Bound (CRLB)
SOLO
Gradient
Gradient of a Vector() ( ) ( )
nT
n
pT
p
p
xxxxaaaaxa RRR Î=Î=Î ,,,,,,
2121
() [ ]
nxp
n
p
nn
p
p
p
n
T
x
x
a
x
a
x
a
x
a
x
a
x
a
x
a
x
a
x
a
aaa
x
x
x
xa RÎ
ú
ú
ú
ú
ú
ú
ú
ú
ú
û
ù
ê
ê
ê
ê
ê
ê
ê
ê
ê
ë
é
¶
¶
¶
¶
¶
¶
¶
¶
¶
¶
¶
¶
¶
¶
¶
¶
¶
¶
=
ú
ú
ú
ú
ú
ú
ú
ú
ú
û
ù
ê
ê
ê
ê
ê
ê
ê
ê
ê
ë
é
¶
¶
¶
¶
¶
¶
=Ñ
21
22
2
2
1
11
2
1
1
212
1
nT
x
T
x
nxnT
x vvxxvIx RR Î=Ñ=ÑÎ=Ñ &1
()()[ ] ()[ ]() ()[ ]()
nT
x
T
x
abba
T
x xaxbxbxaxbxa
TT
RÎÑ+Ñ=Ñ
=
2
( )( ) ( ) ( )xMMxxMxMxxMx
T
Mx
T
x
T
x
T
x
TT
+=Ñ+Ñ=Ñ
3
The Cramér-Rao Lower Bound on the Variance of the Estimator – Multivariable Case
Return to Table of Content
Cramér-Rao Lower Bound (CRLB)
SOLO
Gradient
Gradient of a Vector() ( ) ( )
nT
n
pT
p
p
xxxxaaaaxa RRR Î=Î=Î ,,,,,,
2121
() [ ]
nxp
n
p
nn
p
p
p
n
T
x
x
a
x
a
x
a
x
a
x
a
x
a
x
a
x
a
x
a
aaa
x
x
x
xa RÎ
ú
ú
ú
ú
ú
ú
ú
ú
ú
û
ù
ê
ê
ê
ê
ê
ê
ê
ê
ê
ë
é
¶
¶
¶
¶
¶
¶
¶
¶
¶
¶
¶
¶
¶
¶
¶
¶
¶
¶
=
ú
ú
ú
ú
ú
ú
ú
ú
ú
û
ù
ê
ê
ê
ê
ê
ê
ê
ê
ê
ë
é
¶
¶
¶
¶
¶
¶
=Ñ
21
22
2
2
1
11
2
1
1
212
1
nT
x
T
x
nxnT
x vvxxvIx RR Î=Ñ=ÑÎ=Ñ &1
()()[ ] ()[ ]() ()[ ]()
nT
x
T
x
abba
T
x xaxbxbxaxbxa
TT
RÎÑ+Ñ=Ñ
=
2
( )( ) ( ) ( )xMMxxMxMxxMx
T
Mx
T
x
T
x
T
x
TT
+=Ñ+Ñ=Ñ
3
The Cramér-Rao Lower Bound on the Variance of the Estimator – Multivariable Case
Return to Table of Content
11
Cramér-Rao Lower Bound (CRLB)
SOLO
Matrix Inversion Relations
ú
ú
û
ù
ê
ê
ë
é
DD
D-D+
=
ú
û
ù
ê
ë
é
---
------
-
111
111111
1
AD
BAADBAA
CD
BA
1
2
AofcomplementSchurBADC
1
:
-
-=D
( ) ( )
1
1
111
1
1 -
-
---
-
-
+-=+ ADBADCBAADBCA
The Cramér-Rao Lower Bound on the Variance of the Estimator – Multivariable Case
Return to Table of Content
Cramér-Rao Lower Bound (CRLB)
SOLO
Matrix Inversion Relations
ú
ú
û
ù
ê
ê
ë
é
DD
D-D+
=
ú
û
ù
ê
ë
é
---
------
-
111
111111
1
AD
BAADBAA
CD
BA
1
2
AofcomplementSchurBADC
1
:
-
-=D
( ) ( )
1
1
111
1
1 -
-
---
-
-
+-=+ ADBADCBAADBCA
The Cramér-Rao Lower Bound on the Variance of the Estimator – Multivariable Case
Return to Table of Content
12
Cramér-Rao Lower Bound (CRLB)
SOLO
Helpfully Relations
( )
( )
( ) ( ) ( )zxfzxfzxf
zxf
zxf
T
xx
T
xx
T
xx
,ln,ln,
,
1
,ln ÑÑ-ÑÑ=ÑÑ
Proof:
Start with: ( )
( )
( )zxf
zxf
zxf
xx ,
,
1
,ln Ñ=Ñ
( )
( )
( )
( )
( )
( )
( )
( )
( )
( )
( ) ( )
( )
( )
( )
( ) ( )
( )
( ) ( ) ( )zxfzxfzxf
zxf
zxfzxf
zxf
zxf
zxf
zxfzxf
zxf
zxf
zxf
zxf
zxf
zxf
zxf
zxf
zxf
zxf
T
xx
T
xx
T
xx
T
xx
T
xx
T
xx
T
xx
T
xx
T
xx
T
xx
,ln,ln,
,
1
,ln,
,
1
,
,
1
,ln,
,
1
,
,
1
,
,
1
,
,
1
,
,
1
,ln
ÑÑ-ÑÑ=
ÑÑ-ÑÑ=Ñ
ú
û
ù
ê
ë
é
Ñ+ÑÑ=
Ñ
ú
û
ù
ê
ë
é
Ñ+ÑÑ=
ú
û
ù
ê
ë
é
ÑÑ=ÑÑ
( )
+
+
®= RR
pn
zxf :,Lemma 1: Given a function the following relations holds:
The Cramér-Rao Lower Bound on the Variance of the Estimator – Multivariable Case
Cramér-Rao Lower Bound (CRLB)
SOLO
Helpfully Relations
( )
( )
( ) ( ) ( )zxfzxfzxf
zxf
zxf
T
xx
T
xx
T
xx
,ln,ln,
,
1
,ln ÑÑ-ÑÑ=ÑÑ
Proof:
Start with: ( )
( )
( )zxf
zxf
zxf
xx ,
,
1
,ln Ñ=Ñ
( )
( )
( )
( )
( )
( )
( )
( )
( )
( )
( ) ( )
( )
( )
( )
( ) ( )
( )
( ) ( ) ( )zxfzxfzxf
zxf
zxfzxf
zxf
zxf
zxf
zxfzxf
zxf
zxf
zxf
zxf
zxf
zxf
zxf
zxf
zxf
zxf
T
xx
T
xx
T
xx
T
xx
T
xx
T
xx
T
xx
T
xx
T
xx
T
xx
,ln,ln,
,
1
,ln,
,
1
,
,
1
,ln,
,
1
,
,
1
,
,
1
,
,
1
,
,
1
,ln
ÑÑ-ÑÑ=
ÑÑ-ÑÑ=Ñ
ú
û
ù
ê
ë
é
Ñ+ÑÑ=
Ñ
ú
û
ù
ê
ë
é
Ñ+ÑÑ=
ú
û
ù
ê
ë
é
ÑÑ=ÑÑ
( )
+
+
®= RR
pn
zxf :,Lemma 1: Given a function the following relations holds:
The Cramér-Rao Lower Bound on the Variance of the Estimator – Multivariable Case
13
Cramér-Rao Lower Bound (CRLB)
SOLO
Helpfully Relations
( )
( )
( ) ( ) ( )zxfzxfzxf
zxf
zxf
T
xx
T
xx
T
xx ,ln,ln,
,
1
,ln ÑÑ-ÑÑ=ÑÑ
Proof:
( )
+
+
®= RR
pn
zxf :,Lemma 1: Given a function the following relations holds:
p
zRÎLemma 2: Let be a random vector with density p (y|x) parameterized by the
nonrandom vector , then:
n
xRÎ
( ) ( ){ } ( ){ }xzpExzpxzpE
T
xxz
T
xxz
|ln|ln|ln ÑÑ-=ÑÑ
( ){ }
( )
( ) ( ) ( ){ }xzpxzpExzp
xzp
ExzpE
T
xxz
T
xxz
T
xxz
|ln|ln|
|
1
|ln
0
ÑÑ-
þ
ý
ü
î
í
ì
ÑÑ=ÑÑ
()
()
()
()() () 0|||
|
1
|
|
1
1
=ÑÑ=
ú
û
ù
ê
ë
é
ÑÑ=
þ
ý
ü
î
í
ì
ÑÑ òò
pp
zdxzpzdxzpxzp
xzp
xzp
xzp
E
T
xx
T
xx
T
xxz
RR
Proof:
( ) ( ){ } ( ){ }zxpEzxpzxpE
T
xxzx
T
xxzx ,ln,ln,ln
,, ÑÑ-=ÑÑ
( ){ }
( )
( ) ( ) ( ){ }zxpzxpEzxp
zxp
EzxpE
T
xxzx
T
xxzx
T
xxzx ,ln,ln,
,
1
,ln
,
0
,, ÑÑ-
þ
ý
ü
î
í
ì
ÑÑ=ÑÑ
Lemma 3: Let be random vectors with joint density p (x,y), then:
pn
zx RRÎÎ,
( )
( )
( )
( ) ( ) ( ) 0,,,
,
1
,
,
1
1
,
=ÑÑ=
ú
û
ù
ê
ë
é
ÑÑ=
þ
ý
ü
î
í
ì
ÑÑ òò
++
pnpn
zdxdzxpzdxdzxpzxp
zxp
zxp
zxp
E
T
xx
T
xx
T
xxzx
RR
Return to Table of Content
Cramér-Rao Lower Bound (CRLB)
SOLO
Helpfully Relations
( )
( )
( ) ( ) ( )zxfzxfzxf
zxf
zxf
T
xx
T
xx
T
xx ,ln,ln,
,
1
,ln ÑÑ-ÑÑ=ÑÑ
Proof:
( )
+
+
®= RR
pn
zxf :,Lemma 1: Given a function the following relations holds:
p
zRÎLemma 2: Let be a random vector with density p (y|x) parameterized by the
nonrandom vector , then:
n
xRÎ
( ) ( ){ } ( ){ }xzpExzpxzpE
T
xxz
T
xxz
|ln|ln|ln ÑÑ-=ÑÑ
( ){ }
( )
( ) ( ) ( ){ }xzpxzpExzp
xzp
ExzpE
T
xxz
T
xxz
T
xxz
|ln|ln|
|
1
|ln
0
ÑÑ-
þ
ý
ü
î
í
ì
ÑÑ=ÑÑ
()
()
()
()() () 0|||
|
1
|
|
1
1
=ÑÑ=
ú
û
ù
ê
ë
é
ÑÑ=
þ
ý
ü
î
í
ì
ÑÑ òò
pp
zdxzpzdxzpxzp
xzp
xzp
xzp
E
T
xx
T
xx
T
xxz
RR
Proof:
( ) ( ){ } ( ){ }zxpEzxpzxpE
T
xxzx
T
xxzx ,ln,ln,ln
,, ÑÑ-=ÑÑ
( ){ }
( )
( ) ( ) ( ){ }zxpzxpEzxp
zxp
EzxpE
T
xxzx
T
xxzx
T
xxzx ,ln,ln,
,
1
,ln
,
0
,, ÑÑ-
þ
ý
ü
î
í
ì
ÑÑ=ÑÑ
Lemma 3: Let be random vectors with joint density p (x,y), then:
pn
zx RRÎÎ,
( )
( )
( )
( ) ( ) ( ) 0,,,
,
1
,
,
1
1
,
=ÑÑ=
ú
û
ù
ê
ë
é
ÑÑ=
þ
ý
ü
î
í
ì
ÑÑ òò
++
pnpn
zdxdzxpzdxdzxpzxp
zxp
zxp
zxp
E
T
xx
T
xx
T
xxzx
RR
Return to Table of Content
14
Cramér-Rao Lower Bound (CRLB)
SOLO
Nonrandom Parameters
n
xRÎ
p
zRÎ
The Score of the estimation is defined by the logarithm of the likelihood( )xzp
x
|lnÑ
In Maximum Likelihood Estimation (MLE), this function returns a vector valued
Score given by the observations and a candidate parameter vector .
Score close to zero are good scores since they indicate that is close to a local
optimum of , since
p
zRÎ
n
xRÎ
x
( )xzp|
( )
( )
( )xzp
xzp
xzp
xx |
|
1
|ln Ñ=Ñ
Since the measurement vector is stochastic the Expected Value of the Score
is given by:
p
zRÎ
( ){ } ( )( )
( )
( )( ) ( ) ( ) 0||||
|
1
||ln|ln
1
=Ñ=Ñ=Ñ=
Ñ=Ñ
òòò
ò
ppp
p
zdxzpzdxzpzdxzpxzp
xzp
zdxzpxzpxzpE
xxx
xxz
RRR
R
v
( )vxh,
z
x
Estimator
x
The parameters are regarded as unknown but fixed.
The measurements are
n
xRÎ
p
zRÎ
Cramér-Rao Lower Bound (CRLB)
SOLO
Nonrandom Parameters
n
xRÎ
p
zRÎ
The Score of the estimation is defined by the logarithm of the likelihood( )xzp
x
|lnÑ
In Maximum Likelihood Estimation (MLE), this function returns a vector valued
Score given by the observations and a candidate parameter vector .
Score close to zero are good scores since they indicate that is close to a local
optimum of , since
p
zRÎ
n
xRÎ
x
( )xzp|
( )
( )
( )xzp
xzp
xzp
xx |
|
1
|ln Ñ=Ñ
Since the measurement vector is stochastic the Expected Value of the Score
is given by:
p
zRÎ
( ){ } ( )( )
( )
( )( ) ( ) ( ) 0||||
|
1
||ln|ln
1
=Ñ=Ñ=Ñ=
Ñ=Ñ
òòò
ò
ppp
p
zdxzpzdxzpzdxzpxzp
xzp
zdxzpxzpxzpE
xxx
xxz
RRR
R
v
( )vxh,
z
x
Estimator
x
The parameters are regarded as unknown but fixed.
The measurements are
n
xRÎ
p
zRÎ
15
Cramér-Rao Lower Bound (CRLB)
() ( ) ( ){ } ( ){ }xzpExzpxzpExJ
T
xxz
T
xxz |ln|ln|ln: ÑÑ-=ÑÑ=
SOLO
The Fisher Information Matrix (FIM)
Fisher, Sir Ronald Aylmer
1890 - 1962
The Fisher Information Matrix (FIM) was defined by Ronald Aylmer
Fisher as the Covariance Matrix of the Score
( ){ } ()( ) 0||ln|ln =Ñ=Ñ ò
p
zdxzpxpxzpE
xxz
R
The Expected Value of the Score is given by:
The Covariance of the Score is given by:
( ) ( ){ } () ( )( )ò
ÑÑ=ÑÑ
p
zdxzpxzpxpxzpxzpE
T
xx
T
xxz
R
||ln|ln|ln|ln
Nonrandom Parameters
The Cramér-Rao Lower Bound on the Variance of the Estimator – Multivariable Case
Cramér-Rao Lower Bound (CRLB)
() ( ) ( ){ } ( ){ }xzpExzpxzpExJ
T
xxz
T
xxz |ln|ln|ln: ÑÑ-=ÑÑ=
SOLO
The Fisher Information Matrix (FIM)
Fisher, Sir Ronald Aylmer
1890 - 1962
The Fisher Information Matrix (FIM) was defined by Ronald Aylmer
Fisher as the Covariance Matrix of the Score
( ){ } ()( ) 0||ln|ln =Ñ=Ñ ò
p
zdxzpxpxzpE
xxz
R
The Expected Value of the Score is given by:
The Covariance of the Score is given by:
( ) ( ){ } () ( )( )ò
ÑÑ=ÑÑ
p
zdxzpxzpxpxzpxzpE
T
xx
T
xxz
R
||ln|ln|ln|ln
Nonrandom Parameters
The Cramér-Rao Lower Bound on the Variance of the Estimator – Multivariable Case
16
Fisher, Sir Ronald Aylmer (1890-1962)
The Fisher information is the amount of information that
an observable random variable z carries about an unknown
parameter x upon which the likelihood of z, L(x) = f (Z; x),
depends. The likelihood function is the joint probability of
the data, the Zs, conditional on the value of x, as a function
of x. Since the expectation of the score is zero, the variance
is simply the second moment of the score, the derivative of
the lan of the likelihood function with respect to x. Hence
the Fisher information can be written
() [ ]( ) [ ]( ){ } [ ]( ){ }
x
k
xx
x
T
k
x
k
x
xZLExZLxZLEx ,ln,ln,ln: ÑÑ-=ÑÑ=J
Cramér-Rao Lower Bound (CRLB)
Return to Table of Content
Fisher, Sir Ronald Aylmer (1890-1962)
The Fisher information is the amount of information that
an observable random variable z carries about an unknown
parameter x upon which the likelihood of z, L(x) = f (Z; x),
depends. The likelihood function is the joint probability of
the data, the Zs, conditional on the value of x, as a function
of x. Since the expectation of the score is zero, the variance
is simply the second moment of the score, the derivative of
the lan of the likelihood function with respect to x. Hence
the Fisher information can be written
() [ ]( ) [ ]( ){ } [ ]( ){ }
x
k
xx
x
T
k
x
k
x
xZLExZLxZLEx ,ln,ln,ln: ÑÑ-=ÑÑ=J
Cramér-Rao Lower Bound (CRLB)
Return to Table of Content
17
Cramér-Rao Lower Bound (CRLB)SOLO
( ){ } ()
rxn
yy
T
y
Trxr
yy
T
yyz ytMyzpEJ RR ÎÑ=ÎÑÑ-=
== **
:&|ln:
Nonrandom Parameters
The Likelihood p (z|x) may be over-parameterized so that some of x or combination of
elements of x do not affect p (z|x). In such a case the FIM for the parameters x becomes
singular. This leads to problems of computing the Cramér – Rao bounds. Let
(r ≤ n) be an alternative parameterization of the Likelihood such that p (z|y) is a well
defined density function for z given and the corresponding FIM is non-singular.
We define a possible non-invertible coordinate transformation .
r
yRÎ
r
yRÎ
()ytx=
Theorem 1: Nonrandom Parametric Cramér – Rao Bound
Assume that the observation has a well defined probability density function p (z|y)
for all , and let denote the parameter that yields the true distribution of .
Moreover, let be an Unbiased Estimator of , and let .
The estimation error covariance of is bounded for below by
p
zRÎ
r
yRÎ *y y
()
n
zx RΈ ()ytx= ( )**ytx=
()zxˆ
( )( ){ }
TT
z
MJMxxxxE
1
*ˆ*ˆ
-
³--
where
are matrices that depend on the true unknown parameter vector .*y
Cramér-Rao Lower Bound (CRLB)SOLO
( ){ } ()
rxn
yy
T
y
Trxr
yy
T
yyz ytMyzpEJ RR ÎÑ=ÎÑÑ-=
== **
:&|ln:
Nonrandom Parameters
The Likelihood p (z|x) may be over-parameterized so that some of x or combination of
elements of x do not affect p (z|x). In such a case the FIM for the parameters x becomes
singular. This leads to problems of computing the Cramér – Rao bounds. Let
(r ≤ n) be an alternative parameterization of the Likelihood such that p (z|y) is a well
defined density function for z given and the corresponding FIM is non-singular.
We define a possible non-invertible coordinate transformation .
r
yRÎ
r
yRÎ
()ytx=
Theorem 1: Nonrandom Parametric Cramér – Rao Bound
Assume that the observation has a well defined probability density function p (z|y)
for all , and let denote the parameter that yields the true distribution of .
Moreover, let be an Unbiased Estimator of , and let .
The estimation error covariance of is bounded for below by
p
zRÎ
r
yRÎ *y y
()
n
zx RΈ ()ytx= ( )**ytx=
()zxˆ
( )( ){ }
TT
z
MJMxxxxE
1
*ˆ*ˆ
-
³--
where
are matrices that depend on the true unknown parameter vector .*y
18
Cramér-Rao Lower Bound (CRLB)SOLO
( ){ } ()
**
:&|ln:
yy
T
y
T
yy
T
yyz ytMyzpEJ
==
Ñ=ÑÑ-=
Nonrandom Parameters
Theorem 1: Nonrandom Parametric Cramér – Rao Bound
Assume that the observation has a well defined probability density function p (z|y)
for all , and let denote the parameter that yields the true distribution of .
Moreover, let be an Unbiased Estimator of , and let .
The estimation error covariance of is bounded for below by
p
zRÎ
r
yRÎ *y y
()
n
zx RΈ ()ytx= ( )**ytx=
()zxˆ ( )( ){ }
TT
z
MJMxxxxE
1
*ˆ*ˆ
-
³--
where
are matrices that depend on the true unknown parameter vector .*y
Proof:
() ()[ ] ( ){ } 0|ˆ =-Ñò
p
zdyzpytzx
T
y
R
Tacking the gradient w.r.t. on both sides of this relation we obtain:y
( ){ }() ()[ ] (){ }( ) 0|ˆ| =Ñ--Ñ òò
pp
zdyzpytzdytzxyzp
T
y
T
y
RR
( ){ }() ()[ ] ( ) () ( )
1
||ˆ|ln òò
Ñ=-Ñ
pp
zdyzpytzdyzpytzxyzp
T
y
T
y
RR
( ){ }() ()[ ] ( ) ()ytzdyzpytzxyzp
T
y
T
y
p
Ñ=-Ñò
R
|ˆ|ln
Consider the Random Vector:
( )
÷
÷
ø
ö
ç
ç
è
æ
Ñ
-
yzp
xx
y
|ln
ˆ
where:
( )
{ }
( ){ } ÷
÷
ø
ö
ç
ç
è
æ
=
÷
÷
ø
ö
ç
ç
è
æ
Ñ
-
=
ï
þ
ï
ý
ü
ï
î
ï
í
ì
÷
÷
ø
ö
ç
ç
è
æ
Ñ
-
0
0
|ln
ˆ
|ln
ˆ
yzpE
xxE
yzp
xx
E
yz
z
y
z
()[ ] ( ) () ()[ ] ( )
()
0|ˆ|ˆ
ˆ
sUnbiasenes
zxof
TT
pp
zdyzpytzxzdxzpxzx =-=- òò
RR
Using the Unbiasedness of Estimator:
Cramér-Rao Lower Bound (CRLB)SOLO
( ){ } ()
**
:&|ln:
yy
T
y
T
yy
T
yyz ytMyzpEJ
==
Ñ=ÑÑ-=
Nonrandom Parameters
Theorem 1: Nonrandom Parametric Cramér – Rao Bound
Assume that the observation has a well defined probability density function p (z|y)
for all , and let denote the parameter that yields the true distribution of .
Moreover, let be an Unbiased Estimator of , and let .
The estimation error covariance of is bounded for below by
p
zRÎ
r
yRÎ *y y
()
n
zx RΈ ()ytx= ( )**ytx=
()zxˆ ( )( ){ }
TT
z
MJMxxxxE
1
*ˆ*ˆ
-
³--
where
are matrices that depend on the true unknown parameter vector .*y
Proof:
() ()[ ] ( ){ } 0|ˆ =-Ñò
p
zdyzpytzx
T
y
R
Tacking the gradient w.r.t. on both sides of this relation we obtain:y
( ){ }() ()[ ] (){ }( ) 0|ˆ| =Ñ--Ñ òò
pp
zdyzpytzdytzxyzp
T
y
T
y
RR
( ){ }() ()[ ] ( ) () ( )
1
||ˆ|ln òò
Ñ=-Ñ
pp
zdyzpytzdyzpytzxyzp
T
y
T
y
RR
( ){ }() ()[ ] ( ) ()ytzdyzpytzxyzp
T
y
T
y
p
Ñ=-Ñò
R
|ˆ|ln
Consider the Random Vector:
( )
÷
÷
ø
ö
ç
ç
è
æ
Ñ
-
yzp
xx
y
|ln
ˆ
where:
( )
{ }
( ){ } ÷
÷
ø
ö
ç
ç
è
æ
=
÷
÷
ø
ö
ç
ç
è
æ
Ñ
-
=
ï
þ
ï
ý
ü
ï
î
ï
í
ì
÷
÷
ø
ö
ç
ç
è
æ
Ñ
-
0
0
|ln
ˆ
|ln
ˆ
yzpE
xxE
yzp
xx
E
yz
z
y
z
()[ ] ( ) () ()[ ] ( )
()
0|ˆ|ˆ
ˆ
sUnbiasenes
zxof
TT
pp
zdyzpytzxzdxzpxzx =-=- òò
RR
Using the Unbiasedness of Estimator:
19
Cramér-Rao Lower Bound (CRLB)SOLO
( ){ } ()
**
:&|ln:
yy
T
y
T
yy
T
yyz ytMyzpEJ
==
Ñ=ÑÑ-=
Nonrandom Parameters
Theorem 1: Nonrandom Parametric Cramér – Rao Bound
Assume that the observation has a well defined probability density function p (z|y)
for all , and let denote the parameter that yields the true distribution of .
Moreover, let be an Unbiased Estimator of , and let .
The estimation error covariance of is bounded for below by
p
zRÎ
r
yRÎ *y y
()
n
zx RΈ ()ytx= ( )**ytx=
()zxˆ ( )( ){ }
TT
z
MJMxxxxE
1
*ˆ*ˆ
-
³--
where
are matrices that depend on the true unknown parameter vector .*y
Proof (continue – 1):
Consider the Random Vector:
( )
÷
÷
ø
ö
ç
ç
è
æ
Ñ
-
yzp
xx
y
|ln
ˆ
The Covariance Matrix is Positive Semi-definite by construction:
( )
{ }
( ){ } ÷
÷
ø
ö
ç
ç
è
æ
=
÷
÷
ø
ö
ç
ç
è
æ
Ñ
-
=
ï
þ
ï
ý
ü
ï
î
ï
í
ì
÷
÷
ø
ö
ç
ç
è
æ
Ñ
-
0
0
|ln
ˆ
|ln
ˆ
yzpE
xxE
yzp
xx
E
yz
z
y
z
( ) ( )
0
0
0
0
0|ln
ˆ
|ln
ˆ
1
11
definiteSemi
Positive
T
T
T
T
yy
z
IMJ
I
J
MJMC
I
JMI
JM
MC
yzp
xx
yzp
xx
E
-
-
--
³ú
û
ù
ê
ë
é
ú
û
ù
ê
ë
é-
ú
û
ù
ê
ë
é
=ú
û
ù
ê
ë
é
=
ï
þ
ï
ý
ü
ï
î
ï
í
ì
÷
÷
ø
ö
ç
ç
è
æ
Ñ
-
÷
÷
ø
ö
ç
ç
è
æ
Ñ
-
( )( ){ }
T
z xxxxEC --= ˆˆ: ( ) ( ){ } ( ){ }yzpEyzpyzpEJ
T
yyz
T
yyz |ln|ln|ln: ÑÑ-=ÑÑ=
( )( )( ){ } ()ytxxyzpEM
T
y
T
yz
T
Ñ=-Ñ= ˆ|ln:
( )( ){ }
TT
z
Notations
Equivalent
definiteSemi
Positive
T
MJMxxxxECMJMC
11
ˆˆ:0
-
-
-
³--=Û³-
( ){ }() ()[ ] ( ) ()ytzdyzpytzxyzp
T
y
T
y
p
Ñ=-Ñò
R
|ˆ|lnWe found:
q.e.d.
where:
Cramér-Rao Lower Bound (CRLB)SOLO
( ){ } ()
**
:&|ln:
yy
T
y
T
yy
T
yyz ytMyzpEJ
==
Ñ=ÑÑ-=
Nonrandom Parameters
Theorem 1: Nonrandom Parametric Cramér – Rao Bound
Assume that the observation has a well defined probability density function p (z|y)
for all , and let denote the parameter that yields the true distribution of .
Moreover, let be an Unbiased Estimator of , and let .
The estimation error covariance of is bounded for below by
p
zRÎ
r
yRÎ *y y
()
n
zx RΈ ()ytx= ( )**ytx=
()zxˆ ( )( ){ }
TT
z
MJMxxxxE
1
*ˆ*ˆ
-
³--
where
are matrices that depend on the true unknown parameter vector .*y
Proof (continue – 1):
Consider the Random Vector:
( )
÷
÷
ø
ö
ç
ç
è
æ
Ñ
-
yzp
xx
y
|ln
ˆ
The Covariance Matrix is Positive Semi-definite by construction:
( )
{ }
( ){ } ÷
÷
ø
ö
ç
ç
è
æ
=
÷
÷
ø
ö
ç
ç
è
æ
Ñ
-
=
ï
þ
ï
ý
ü
ï
î
ï
í
ì
÷
÷
ø
ö
ç
ç
è
æ
Ñ
-
0
0
|ln
ˆ
|ln
ˆ
yzpE
xxE
yzp
xx
E
yz
z
y
z
( ) ( )
0
0
0
0
0|ln
ˆ
|ln
ˆ
1
11
definiteSemi
Positive
T
T
T
T
yy
z
IMJ
I
J
MJMC
I
JMI
JM
MC
yzp
xx
yzp
xx
E
-
-
--
³ú
û
ù
ê
ë
é
ú
û
ù
ê
ë
é-
ú
û
ù
ê
ë
é
=ú
û
ù
ê
ë
é
=
ï
þ
ï
ý
ü
ï
î
ï
í
ì
÷
÷
ø
ö
ç
ç
è
æ
Ñ
-
÷
÷
ø
ö
ç
ç
è
æ
Ñ
-
( )( ){ }
T
z xxxxEC --= ˆˆ: ( ) ( ){ } ( ){ }yzpEyzpyzpEJ
T
yyz
T
yyz |ln|ln|ln: ÑÑ-=ÑÑ=
( )( )( ){ } ()ytxxyzpEM
T
y
T
yz
T
Ñ=-Ñ= ˆ|ln:
( )( ){ }
TT
z
Notations
Equivalent
definiteSemi
Positive
T
MJMxxxxECMJMC
11
ˆˆ:0
-
-
-
³--=Û³-
( ){ }() ()[ ] ( ) ()ytzdyzpytzxyzp
T
y
T
y
p
Ñ=-Ñò
R
|ˆ|lnWe found:
q.e.d.
where:
20
Cramér-Rao Lower Bound (CRLB)SOLO
( ){ } ()
nxn
yy
T
y
Tnxn
yy
T
yyz ybIMyzpEJ RR ÎÑ+=ÎÑÑ-=
== **
:&|ln:
Nonrandom Parameters
Corollary 1: Nonrandom Parametric Cramér – Rao Bound (Baiased Estimator)
Consider an estimaton problem defined by the likelihood p (y|z), and the fixed unknown
parameter . Any estimator with unknown bias has a mean square error
bounded from below by
*y ()zyˆ ()yb
( )( ){ } ( ) ( )***ˆ*ˆ
1
ybybMJMyyyyE
TTT
z +³--
-
where
are matrices that depend on the true unknown parameter vector .*y
Proof:
Theorem 1 yields that:
Introduce the quantity , the estimator is an unbiased estimator of .()ybyx+=: () ()zyzx ˆˆ= x
( )( ){ } ()[ ] ( ){ }( ) ()[ ]ybIyzpEybIxxxxE
T
y
T
yyz
T
T
y
T
z
Ñ+ÑÑ-Ñ+³--
-1
|lnˆˆ
Using , we obtain:()ybyx+=:
( )( ){ } ()[ ] ( ){ }( ) ()[ ] () ()ybybybIyzpEybIyyyyE
TT
y
T
yyz
T
T
y
T
z +Ñ+ÑÑ-Ñ+³--
-1
|lnˆˆ
after suitably inserting the true parameter .*y
Cramér-Rao Lower Bound (CRLB)SOLO
( ){ } ()
nxn
yy
T
y
Tnxn
yy
T
yyz ybIMyzpEJ RR ÎÑ+=ÎÑÑ-=
== **
:&|ln:
Nonrandom Parameters
Corollary 1: Nonrandom Parametric Cramér – Rao Bound (Baiased Estimator)
Consider an estimaton problem defined by the likelihood p (y|z), and the fixed unknown
parameter . Any estimator with unknown bias has a mean square error
bounded from below by
*y ()zyˆ ()yb
( )( ){ } ( ) ( )***ˆ*ˆ
1
ybybMJMyyyyE
TTT
z +³--
-
where
are matrices that depend on the true unknown parameter vector .*y
Proof:
Theorem 1 yields that:
Introduce the quantity , the estimator is an unbiased estimator of .()ybyx+=: () ()zyzx ˆˆ= x
( )( ){ } ()[ ] ( ){ }( ) ()[ ]ybIyzpEybIxxxxE
T
y
T
yyz
T
T
y
T
z
Ñ+ÑÑ-Ñ+³--
-1
|lnˆˆ
Using , we obtain:()ybyx+=:
( )( ){ } ()[ ] ( ){ }( ) ()[ ] () ()ybybybIyzpEybIyyyyE
TT
y
T
yyz
T
T
y
T
z +Ñ+ÑÑ-Ñ+³--
-1
|lnˆˆ
after suitably inserting the true parameter .*y
21
Cramér-Rao Lower Bound (CRLB)
[]( )[]( ) [ ] []( )[]( ){ }
() [ ] [ ] ()
() ()
() [ ] ()
() ()xbxb
x
xb
I
x
xZL
E
x
xb
I
xbxb
x
xb
I
x
xZL
x
xZL
E
x
xb
I
xZxxZxEZdxZLxZxxZx
T
x
k
T
T
x
T
kk
T
x
T
kkkk
T
kk
+
ú
û
ù
ê
ë
é
¶
¶
+
÷
÷
ø
ö
ç
ç
è
æ
ï
þ
ï
ý
ü
ï
î
ï
í
ì
¶
¶
ú
û
ù
ê
ë
é
¶
¶
+-=
+
ú
û
ù
ê
ë
é
¶
¶
+
÷
÷
÷
ø
ö
ç
ç
ç
è
æ
ï
þ
ï
ý
ü
ï
î
ï
í
ì
÷
÷
ø
ö
ç
ç
è
æ
¶
¶
÷
÷
ø
ö
ç
ç
è
æ
¶
¶
ú
û
ù
ê
ë
é
¶
¶
+³
--=--
-
-
ò
1
2
2
1
,ln
,ln,ln
,
SOLO
The Cramér-Rao Lower Bound on the Variance of the Estimator
The multivariable form of the Cramér-Rao Lower Bound is:
[]( )
[]
[]
÷
÷
÷
ø
ö
ç
ç
ç
è
æ
-
-
=-
n
k
n
k
k
xZx
xZx
xZx
11
[ ]( )
[ ]
[ ]
[ ]
÷
÷
÷
÷
÷
÷
ø
ö
ç
ç
ç
ç
ç
ç
è
æ
¶
¶
¶
¶
=
÷
÷
ø
ö
ç
ç
è
æ
¶
¶
=Ñ
n
k
k
k
k
x
x
xZL
x
xZL
x
xZL
xZL
,ln
,ln
,ln
,ln
1
Fisher Information Matrix
[ ] [ ] [ ]
ï
þ
ï
ý
ü
ï
î
ï
í
ì
¶
¶
-=
ï
þ
ï
ý
ü
ï
î
ï
í
ì
÷
÷
ø
ö
ç
ç
è
æ
¶
¶
÷
÷
ø
ö
ç
ç
è
æ
¶
¶
=
x
k
x
T
kk
x
xZL
E
x
xZL
x
xZL
E
2
2
,ln,ln,ln
:J
Fisher, Sir Ronald Aylmer
1890 - 1962
Return to Table of Content
Cramér-Rao Lower Bound (CRLB)
[]( )[]( ) [ ] []( )[]( ){ }
() [ ] [ ] ()
() ()
() [ ] ()
() ()xbxb
x
xb
I
x
xZL
E
x
xb
I
xbxb
x
xb
I
x
xZL
x
xZL
E
x
xb
I
xZxxZxEZdxZLxZxxZx
T
x
k
T
T
x
T
kk
T
x
T
kkkk
T
kk
+
ú
û
ù
ê
ë
é
¶
¶
+
÷
÷
ø
ö
ç
ç
è
æ
ï
þ
ï
ý
ü
ï
î
ï
í
ì
¶
¶
ú
û
ù
ê
ë
é
¶
¶
+-=
+
ú
û
ù
ê
ë
é
¶
¶
+
÷
÷
÷
ø
ö
ç
ç
ç
è
æ
ï
þ
ï
ý
ü
ï
î
ï
í
ì
÷
÷
ø
ö
ç
ç
è
æ
¶
¶
÷
÷
ø
ö
ç
ç
è
æ
¶
¶
ú
û
ù
ê
ë
é
¶
¶
+³
--=--
-
-
ò
1
2
2
1
,ln
,ln,ln
,
SOLO
The Cramér-Rao Lower Bound on the Variance of the Estimator
The multivariable form of the Cramér-Rao Lower Bound is:
[]( )
[]
[]
÷
÷
÷
ø
ö
ç
ç
ç
è
æ
-
-
=-
n
k
n
k
k
xZx
xZx
xZx
11
[ ]( )
[ ]
[ ]
[ ]
÷
÷
÷
÷
÷
÷
ø
ö
ç
ç
ç
ç
ç
ç
è
æ
¶
¶
¶
¶
=
÷
÷
ø
ö
ç
ç
è
æ
¶
¶
=Ñ
n
k
k
k
k
x
x
xZL
x
xZL
x
xZL
xZL
,ln
,ln
,ln
,ln
1
Fisher Information Matrix
[ ] [ ] [ ]
ï
þ
ï
ý
ü
ï
î
ï
í
ì
¶
¶
-=
ï
þ
ï
ý
ü
ï
î
ï
í
ì
÷
÷
ø
ö
ç
ç
è
æ
¶
¶
÷
÷
ø
ö
ç
ç
è
æ
¶
¶
=
x
k
x
T
kk
x
xZL
E
x
xZL
x
xZL
E
2
2
,ln,ln,ln
:J
Fisher, Sir Ronald Aylmer
1890 - 1962
Return to Table of Content
22
Cramér-Rao Lower Bound (CRLB)SOLO
Random Parameters
Theorem 2: Random Parameters (Posterior Cramér – Rao Bound)
() ()[ ]( )ò
-=
p
zdyzpytxyb
R
|ˆ
( ){ } (){ }
rxnT
yz
TrxrT
yyyz
ytEMyzpEJ RR ÎÑ=ÎÑÑ-= :&,ln:
,
where
then the Mean Square of the Estimate is Bounded from Below
y
nr
t RR®: x
For Random Parameters there is no true parameter value. Instead, the prior assumption
on the parameter distribution determines the probability of different parameter vectors.
Like in the nonrandom parametric case, we assume a possible non-invertible mapping
between a parameter vector and the sought parameter . The vector
is assumed to have been chosen such that the joint probability density p (y,z) is a well
defined density.
y
Let be two random vectors with a well defined joint density
p (y,z), and let be an estimate of . If the estimator bias
pr
zandy RR ÎÎ
()
n
zx RΈ ()ytx=
satisfies ()() njandriallforypyb
j
z
i
,,1,,10lim ===
±¥®
( )( ){ }
TT
yz
MJMxxxxE
1
,
ˆˆ
-
³--
( )( ){ } 0ˆˆ
1
,
definiteSemi
Positive
TT
yz
MJMxxxxE
-
-
³---
Equivalent
Notations
Cramér-Rao Lower Bound (CRLB)SOLO
Random Parameters
Theorem 2: Random Parameters (Posterior Cramér – Rao Bound)
() ()[ ]( )ò
-=
p
zdyzpytxyb
R
|ˆ
( ){ } (){ }
rxnT
yz
TrxrT
yyyz
ytEMyzpEJ RR ÎÑ=ÎÑÑ-= :&,ln:
,
where
then the Mean Square of the Estimate is Bounded from Below
y
nr
t RR®: x
For Random Parameters there is no true parameter value. Instead, the prior assumption
on the parameter distribution determines the probability of different parameter vectors.
Like in the nonrandom parametric case, we assume a possible non-invertible mapping
between a parameter vector and the sought parameter . The vector
is assumed to have been chosen such that the joint probability density p (y,z) is a well
defined density.
y
Let be two random vectors with a well defined joint density
p (y,z), and let be an estimate of . If the estimator bias
pr
zandy RR ÎÎ
()
n
zx RΈ ()ytx=
satisfies ()() njandriallforypyb
j
z
i
,,1,,10lim ===
±¥®
( )( ){ }
TT
yz
MJMxxxxE
1
,
ˆˆ
-
³--
( )( ){ } 0ˆˆ
1
,
definiteSemi
Positive
TT
yz
MJMxxxxE
-
-
³---
Equivalent
Notations
23
Cramér-Rao Lower Bound (CRLB)SOLO
Random Parameters
Theorem 2: Random Parameters (Posterior Cramér – Rao Bound)
( )( ){ }
TT
yz
MJMxxxxE
1
,
ˆˆ
-
³-- ( ){ } (){ }ytEMyzpEJ
T
yz
TT
yyyz Ñ=ÑÑ-= :&,ln:
,
then the Mean Square of the Estimate is Bounded from Below
Proof:
Let be two random vectors with a well defined joint density
p (y,z), and let be an estimate of . If the estimator bias
pr
zandy RR ÎÎ
()
n
zx RΈ ()ytx=
() ()[ ]( )ò
-=
p
zdyzpytxyb
R
|ˆ and ()() njandriallforypyb
j
z
i
,,1,,10lim ===
±¥®
Compute
()()[ ] () ()[ ] ( )()
( )
() ( )
()
( )[ ]() ()[ ]òòò
-Ñ+-Ñ=-Ñ=Ñ
ppp
zdytzxyzpzdyzpytzdypyzpytzxypyb
T
y
yp
T
y
yzp
T
y
T
y
RRR
ˆ,,|ˆ
,
Integrating both sides w.r.t. over its complete range R
r
yieldsy
()()[ ] ()() ( )[ ]() ()[ ]òòò
+
-Ñ+Ñ-=Ñ
rprr
ydzdytzxyzpydypytydypyb
T
y
T
y
T
y
RRR
ˆ,
The (i,j) element of the left hand side matrix is:
()()
()() ()()
riiii
y
j
y
j
i
j
ydydydydydydypybypybyd
y
ypyb
r
ii
r
111
00
0
+---
-¥=+¥=
==
ú
ú
ú
û
ù
ê
ê
ê
ë
é
-=
¶
¶
òò
RR
Cramér-Rao Lower Bound (CRLB)SOLO
Random Parameters
Theorem 2: Random Parameters (Posterior Cramér – Rao Bound)
( )( ){ }
TT
yz
MJMxxxxE
1
,
ˆˆ
-
³-- ( ){ } (){ }ytEMyzpEJ
T
yz
TT
yyyz Ñ=ÑÑ-= :&,ln:
,
then the Mean Square of the Estimate is Bounded from Below
Proof:
Let be two random vectors with a well defined joint density
p (y,z), and let be an estimate of . If the estimator bias
pr
zandy RR ÎÎ
()
n
zx RΈ ()ytx=
() ()[ ]( )ò
-=
p
zdyzpytxyb
R
|ˆ and ()() njandriallforypyb
j
z
i
,,1,,10lim ===
±¥®
Compute
()()[ ] () ()[ ] ( )()
( )
() ( )
()
( )[ ]() ()[ ]òòò
-Ñ+-Ñ=-Ñ=Ñ
ppp
zdytzxyzpzdyzpytzdypyzpytzxypyb
T
y
yp
T
y
yzp
T
y
T
y
RRR
ˆ,,|ˆ
,
Integrating both sides w.r.t. over its complete range R
r
yieldsy
()()[ ] ()() ( )[ ]() ()[ ]òòò
+
-Ñ+Ñ-=Ñ
rprr
ydzdytzxyzpydypytydypyb
T
y
T
y
T
y
RRR
ˆ,
The (i,j) element of the left hand side matrix is:
()()
()() ()()
riiii
y
j
y
j
i
j
ydydydydydydypybypybyd
y
ypyb
r
ii
r
111
00
0
+---
-¥=+¥=
==
ú
ú
ú
û
ù
ê
ê
ê
ë
é
-=
¶
¶
òò
RR
24
Cramér-Rao Lower Bound (CRLB)SOLO
Random Parameters
Theorem 2: Random Parameters (Posterior Cramér – Rao Bound)
( )( ){ }
TT
yz
MJMxxxxE
1
,
ˆˆ
-
³-- ( ){ } (){ }ytEMyzpEJ
T
yz
TT
yyyz
Ñ=ÑÑ-= :&,ln:
,
then the Mean Square of the Estimate is Bounded from Below
Let be two random vectors with a well defined joint density
p (y,z), and let be an estimate of . If the estimator bias
pr
zandy RR ÎÎ
()
n
zx RΈ ()ytx=
() ()[ ]( )ò
-=
p
zdyzpytxyb
R
|ˆ and ()() njandriallforypyb
j
z
i
,,1,,10lim ===
±¥®
Proof (continue – 1):We found ( )[ ]() ()[ ] ()()òò
Ñ=-Ñ
+ rrp
ydypytydzdytzxyzp
T
y
T
y
RR
ˆ,
( )[ ]() ()[ ] ( ) ()() (){ }ytEydypytydzdyzpytzxyzp
T
yz
T
y
T
y
rrp
Ñ=Ñ=-Ñ òò
+
RR
,ˆ,ln
Consider the Random Vector:
( )
÷
÷
ø
ö
ç
ç
è
æ
Ñ
-
yzp
xx
y
,ln
ˆ
The Covariance Matrix is Positive Semi-definite by construction:
( ) ( )
0
0
0
0
0,ln
ˆ
,ln
ˆ
1
11
,
definiteSemi
Positive
T
T
T
T
yy
yz
IMJ
I
J
MJMC
I
JMI
JM
MC
yzp
xx
yzp
xx
E
-
-
--
³ú
û
ù
ê
ë
é
ú
û
ù
ê
ë
é-
ú
û
ù
ê
ë
é
=ú
û
ù
ê
ë
é
=
ï
þ
ï
ý
ü
ï
î
ï
í
ì
÷
÷
ø
ö
ç
ç
è
æ
Ñ
-
÷
÷
ø
ö
ç
ç
è
æ
Ñ
-
( )( )( ){ } (){ }ytExxyzpEM
T
yz
T
yyz
T
Ñ=-Ñ= ˆ,ln:
,
( )( ){ }
TT
z
Notations
Equivalent
definiteSemi
Positive
T
MJMxxxxECMJMC
11
ˆˆ:0
-
-
-
³--=Û³- q.e.d.
( )( ){ }
T
yz xxxxEC --= ˆˆ:
, ( ) ( ){ } ( ){ }yzpEyzpyzpEJ
T
yyyz
T
yyz ,ln,ln|ln:
,ÑÑ-=ÑÑ=
where:
( )
{ }
( ){ }
÷
÷
ø
ö
ç
ç
è
æ
=
÷
÷
ø
ö
ç
ç
è
æ
Ñ
-
=
ï
þ
ï
ý
ü
ï
î
ï
í
ì
÷
÷
ø
ö
ç
ç
è
æ
Ñ
-
0
0
,ln
ˆ
,ln
ˆ
,
,
,
yzpE
xxE
yzp
xx
E
yyz
yz
y
yz
Return to Table of Content
Cramér-Rao Lower Bound (CRLB)SOLO
Random Parameters
Theorem 2: Random Parameters (Posterior Cramér – Rao Bound)
( )( ){ }
TT
yz
MJMxxxxE
1
,
ˆˆ
-
³-- ( ){ } (){ }ytEMyzpEJ
T
yz
TT
yyyz
Ñ=ÑÑ-= :&,ln:
,
then the Mean Square of the Estimate is Bounded from Below
Let be two random vectors with a well defined joint density
p (y,z), and let be an estimate of . If the estimator bias
pr
zandy RR ÎÎ
()
n
zx RΈ ()ytx=
() ()[ ]( )ò
-=
p
zdyzpytxyb
R
|ˆ and ()() njandriallforypyb
j
z
i
,,1,,10lim ===
±¥®
Proof (continue – 1):We found ( )[ ]() ()[ ] ()()òò
Ñ=-Ñ
+ rrp
ydypytydzdytzxyzp
T
y
T
y
RR
ˆ,
( )[ ]() ()[ ] ( ) ()() (){ }ytEydypytydzdyzpytzxyzp
T
yz
T
y
T
y
rrp
Ñ=Ñ=-Ñ òò
+
RR
,ˆ,ln
Consider the Random Vector:
( )
÷
÷
ø
ö
ç
ç
è
æ
Ñ
-
yzp
xx
y
,ln
ˆ
The Covariance Matrix is Positive Semi-definite by construction:
( ) ( )
0
0
0
0
0,ln
ˆ
,ln
ˆ
1
11
,
definiteSemi
Positive
T
T
T
T
yy
yz
IMJ
I
J
MJMC
I
JMI
JM
MC
yzp
xx
yzp
xx
E
-
-
--
³ú
û
ù
ê
ë
é
ú
û
ù
ê
ë
é-
ú
û
ù
ê
ë
é
=ú
û
ù
ê
ë
é
=
ï
þ
ï
ý
ü
ï
î
ï
í
ì
÷
÷
ø
ö
ç
ç
è
æ
Ñ
-
÷
÷
ø
ö
ç
ç
è
æ
Ñ
-
( )( )( ){ } (){ }ytExxyzpEM
T
yz
T
yyz
T
Ñ=-Ñ= ˆ,ln:
,
( )( ){ }
TT
z
Notations
Equivalent
definiteSemi
Positive
T
MJMxxxxECMJMC
11
ˆˆ:0
-
-
-
³--=Û³- q.e.d.
( )( ){ }
T
yz xxxxEC --= ˆˆ:
, ( ) ( ){ } ( ){ }yzpEyzpyzpEJ
T
yyyz
T
yyz ,ln,ln|ln:
,ÑÑ-=ÑÑ=
where:
( )
{ }
( ){ }
÷
÷
ø
ö
ç
ç
è
æ
=
÷
÷
ø
ö
ç
ç
è
æ
Ñ
-
=
ï
þ
ï
ý
ü
ï
î
ï
í
ì
÷
÷
ø
ö
ç
ç
è
æ
Ñ
-
0
0
,ln
ˆ
,ln
ˆ
,
,
,
yzpE
xxE
yzp
xx
E
yyz
yz
y
yz
Return to Table of Content
25
Cramér-Rao Lower Bound (CRLB)SOLO
Nonrandom and Random Parameters Cramér – Rao Bounds
For the Nonrandom Parameters the Cramér – Rao Bound depends on the true unknown
parameter vector y , and on the model of the problem defined by p (z|y) and the mapping
x = t (y). Hence the bound can only be computed by using simulations, when the true value
of the sought parameter vector y is known.
For the Random Parameters the Cramér – Rao Bound can be computed even in real
applications. Since the parameters are random there is no unknown true parameter value.
Instead, in the posterior Cramér – Rao Bound the matrices J and M are computed by
mathematical expectation performed with respect to the prior distribution of the parameters.
Return to Table of Content
Cramér-Rao Lower Bound (CRLB)SOLO
Nonrandom and Random Parameters Cramér – Rao Bounds
For the Nonrandom Parameters the Cramér – Rao Bound depends on the true unknown
parameter vector y , and on the model of the problem defined by p (z|y) and the mapping
x = t (y). Hence the bound can only be computed by using simulations, when the true value
of the sought parameter vector y is known.
For the Random Parameters the Cramér – Rao Bound can be computed even in real
applications. Since the parameters are random there is no unknown true parameter value.
Instead, in the posterior Cramér – Rao Bound the matrices J and M are computed by
mathematical expectation performed with respect to the prior distribution of the parameters.
Return to Table of Content
26
Cramér-Rao Lower Bound (CRLB)SOLO
Discrete Time Nonlinear Estimation
( )
( )
p
kkk
n
kkk
vxhz
wxfx
R
R
Î=
Î=
--
,
,
11
kk
vw&
1- are system and measurement white-noise sequences
independent of past and current states and on each other and
having known P.D.F.s ( ) ()
kk vpwp &
1-
()
0
xpIn addition the P.D.F. of the initial state , is also given.
We found that the Cramér – Rao Lower Bound for the Random Parameters is given by:
( )( ){ } ( ) ( ){ } ( ){ }
1
:1:1,
1
:1:1:1:1,:1:1|:1:1:1|:1,
,ln,ln,ln
:1:1:1:1
--
ÑÑ-=Ñѳ--
kk
T
XXZXkk
T
XkkXZX
T
kkkkkkZX
XZpEXZpXZpEXXXXE
kkkk
( )
1-=
kk xfxIf we have a deterministic state model, i.e. then we can use the Nonrandom
Parametric Cramér – Rao Lower Bound
( )( ){ } ( ) ( ){ } ( ){ }
1
:1:1
1
:1:1:1:1:1:1|:1:1:1|:1
|ln|ln|ln
:1:1:1:1
--
ÑÑ-=Ñѳ--
kk
T
XXZkk
T
XkkXZ
T
kkkkkkZ
XZpEXZpXZpEXXXXE
kkkk
After k cycles we have k measurements and k random parameters
estimated by an Unbiased Estimator as .
[ ]
T
kk zzzZ ,,,:
21:1 =
[ ]
T
kk
xxxxX ,,,,:
210:0
= [ ]
T
kkkk xxxX
|2|21|1:1|:1
ˆ,,ˆ,ˆ:
ˆ
=
The CRLB provides a lower bound for second-order (mean-squared) error only. Posterior
densities, which result from Nonlinear Filtering, are in general non-Gaussian. A full
statistical characterization of a non-Gaussian density requires higher order moments, in
addition to mean and covariance. Therefore, the CRLB for Nonlinear Filtering does not
fully characterize the accuracy of Filtering Algorithms.
Cramér-Rao Lower Bound (CRLB)SOLO
Discrete Time Nonlinear Estimation
( )
( )
p
kkk
n
kkk
vxhz
wxfx
R
R
Î=
Î=
--
,
,
11
kk
vw&
1- are system and measurement white-noise sequences
independent of past and current states and on each other and
having known P.D.F.s ( ) ()
kk vpwp &
1-
()
0
xpIn addition the P.D.F. of the initial state , is also given.
We found that the Cramér – Rao Lower Bound for the Random Parameters is given by:
( )( ){ } ( ) ( ){ } ( ){ }
1
:1:1,
1
:1:1:1:1,:1:1|:1:1:1|:1,
,ln,ln,ln
:1:1:1:1
--
ÑÑ-=Ñѳ--
kk
T
XXZXkk
T
XkkXZX
T
kkkkkkZX
XZpEXZpXZpEXXXXE
kkkk
( )
1-=
kk xfxIf we have a deterministic state model, i.e. then we can use the Nonrandom
Parametric Cramér – Rao Lower Bound
( )( ){ } ( ) ( ){ } ( ){ }
1
:1:1
1
:1:1:1:1:1:1|:1:1:1|:1
|ln|ln|ln
:1:1:1:1
--
ÑÑ-=Ñѳ--
kk
T
XXZkk
T
XkkXZ
T
kkkkkkZ
XZpEXZpXZpEXXXXE
kkkk
After k cycles we have k measurements and k random parameters
estimated by an Unbiased Estimator as .
[ ]
T
kk zzzZ ,,,:
21:1 =
[ ]
T
kk
xxxxX ,,,,:
210:0
= [ ]
T
kkkk xxxX
|2|21|1:1|:1
ˆ,,ˆ,ˆ:
ˆ
=
The CRLB provides a lower bound for second-order (mean-squared) error only. Posterior
densities, which result from Nonlinear Filtering, are in general non-Gaussian. A full
statistical characterization of a non-Gaussian density requires higher order moments, in
addition to mean and covariance. Therefore, the CRLB for Nonlinear Filtering does not
fully characterize the accuracy of Filtering Algorithms.
27
Cramér-Rao Lower Bound (CRLB)SOLO
Discrete Time Nonlinear Estimation
Theorem 3: The Cramér – Rao Lower Bound for the Random Parameters is given by:
Let perform the partitioning[ ]
1
1:1:1 ,:
xnkT
kkk xXX RÎ=
- [ ]
1
|1:1|1:1:1|:1
ˆ,
ˆ
:
ˆ
xnk
T
kkkkkk xXX RÎ=
--
( )
( )
p
kkk
n
kkk
vxhz
wxfx
R
R
Î=
Î=
--
,
,
11
kk
vw&
1- are system and measurement white-noise sequences
independent of past and current states and on each other and
having known P.D.F.s ( ) ()
kk vpwp &
1-
()
0
xpIn addition the P.D.F. of the initial state , is also given.
After k cycles we have k measurements and k random parameters
estimated by an Unbiased Estimator as .
[ ]
T
kk zzzZ ,,,:
21:1 =
[ ]
T
kk
xxxxX ,,,,:
210:0
= [ ]
T
kkkk xxxX
|2|21|1:1|:1
ˆ,,ˆ,ˆ:
ˆ
=
( ){ }
( ) ( )
( ){ }
( )
( ){ }
nxn
kk
T
xxZXk
nxkn
kk
T
xXZXk
knxkn
kk
T
XXZXk
XZpEC
XZpEB
XZpEA
kk
kk
kk
R
R
R
ÎÑÑ-=
ÎÑÑ-=
ÎÑÑ-=
-
--
-
--
:1:1,
1
:1:1,
11
:1:1,
,ln:
,ln:
,ln:
1:1
1:11:1
( )( ){ } ( )
nxn
kk
T
kkk
T
kkkkkkZX BABCJxxxxE RÎ-=³--
-
--
1
11
||, :ˆˆ( )( ){ } ( ) 0ˆˆ
1
1
||,
definiteSemi
Positive
kk
T
kk
T
kkkkkkZX BABCxxxxE
-
-
-
³----
Equivalent
Notations
Cramér-Rao Lower Bound (CRLB)SOLO
Discrete Time Nonlinear Estimation
Theorem 3: The Cramér – Rao Lower Bound for the Random Parameters is given by:
Let perform the partitioning[ ]
1
1:1:1 ,:
xnkT
kkk xXX RÎ=
- [ ]
1
|1:1|1:1:1|:1
ˆ,
ˆ
:
ˆ
xnk
T
kkkkkk xXX RÎ=
--
( )
( )
p
kkk
n
kkk
vxhz
wxfx
R
R
Î=
Î=
--
,
,
11
kk
vw&
1- are system and measurement white-noise sequences
independent of past and current states and on each other and
having known P.D.F.s ( ) ()
kk vpwp &
1-
()
0
xpIn addition the P.D.F. of the initial state , is also given.
After k cycles we have k measurements and k random parameters
estimated by an Unbiased Estimator as .
[ ]
T
kk zzzZ ,,,:
21:1 =
[ ]
T
kk
xxxxX ,,,,:
210:0
= [ ]
T
kkkk xxxX
|2|21|1:1|:1
ˆ,,ˆ,ˆ:
ˆ
=
( ){ }
( ) ( )
( ){ }
( )
( ){ }
nxn
kk
T
xxZXk
nxkn
kk
T
xXZXk
knxkn
kk
T
XXZXk
XZpEC
XZpEB
XZpEA
kk
kk
kk
R
R
R
ÎÑÑ-=
ÎÑÑ-=
ÎÑÑ-=
-
--
-
--
:1:1,
1
:1:1,
11
:1:1,
,ln:
,ln:
,ln:
1:1
1:11:1
( )( ){ } ( )
nxn
kk
T
kkk
T
kkkkkkZX BABCJxxxxE RÎ-=³--
-
--
1
11
||, :ˆˆ( )( ){ } ( ) 0ˆˆ
1
1
||,
definiteSemi
Positive
kk
T
kk
T
kkkkkkZX BABCxxxxE
-
-
-
³----
Equivalent
Notations
28
Cramér-Rao Lower Bound (CRLB)SOLO
Discrete Time Nonlinear Estimation
The Cramér – Rao Bound for the Random Parameters is given by:
( ) ( ){ } 0,ln,ln
ˆˆ
1
:1:1,:1:1,,
|
1:11:1|1:1
|
1:11:1|1:1
,
1:11:1
definiteSemi
Positive
kk
T
xXkkxXZX
T
kkk
kkk
kkk
kkk
ZX
XZpXZpE
xx
XX
xx
XX
E
kkkk
-
-
------
³ÑÑ-
ï
þ
ï
ý
ü
ï
î
ï
í
ì
÷
÷
ø
ö
ç
ç
è
æ
-
-
÷
÷
ø
ö
ç
ç
è
æ
-
-
--
Proof Theorem 3: Let perform the partitioning[ ]
1
1:1:1
,:
xnkT
kkk
xXX RÎ=
- [ ]
1
|1:1|1:1:1|:1
ˆ,
ˆ
:
ˆ
xnk
T
kkkkkk xXX RÎ=
--
( ){ }
( ){ } ( ){ }
( ){ } ( ){ }
1
:1:1,:1:1,
:1:1,:1:1,1
:1:1,,,
,ln,ln
,ln,ln
,ln
1:1
1:11:11:1
1:11:1
-
-
ú
ú
û
ù
ê
ê
ë
é
ÑÑ-ÑÑ-
ÑÑ-ÑÑ-
=ÑÑ-
-
---
--
kk
T
xxZXkk
T
XxZX
kk
T
xXZXkk
T
XXZX
kk
T
xXxXZX
XZpEXZpE
XZpEXZpE
XZpE
kkkk
kkkk
kkkk
nkxnkkk
kk
T
kk
k
k
T
kk
T
k
kk
I
BAI
BABC
A
IAB
I
CB
BA
RÎ
÷
÷
ø
ö
ç
ç
è
æ
ú
û
ù
ê
ë
é
ú
û
ù
ê
ë
é
-
ú
û
ù
ê
ë
é
=ú
û
ù
ê
ë
é
=
-
-
--
- 1
1
11
1
00
00
:
( )
( )
p
kkk
n
kkk
vxhz
wxfx
R
R
Î=
Î=
--
,
,
11
kk
vw&
1- are system and measurement white-noise sequences
independent of past and current states and on each other and
having known P.D.F.s ( ) ()
kk vpwp &
1-
()
0
xpIn addition the P.D.F. of the initial state , is also given.
After k cycles we have k measurements and k random parameters
estimated by an Unbiased Estimator as .
[ ]
T
kk zzzZ ,,,:
21:1 =
[ ]
T
kk
xxxxX ,,,,:
210:0
= [ ]
T
kkkk xxxX
|2|21|1:1|:1
ˆ,,ˆ,ˆ:
ˆ
=
Cramér-Rao Lower Bound (CRLB)SOLO
Discrete Time Nonlinear Estimation
The Cramér – Rao Bound for the Random Parameters is given by:
( ) ( ){ } 0,ln,ln
ˆˆ
1
:1:1,:1:1,,
|
1:11:1|1:1
|
1:11:1|1:1
,
1:11:1
definiteSemi
Positive
kk
T
xXkkxXZX
T
kkk
kkk
kkk
kkk
ZX
XZpXZpE
xx
XX
xx
XX
E
kkkk
-
-
------
³ÑÑ-
ï
þ
ï
ý
ü
ï
î
ï
í
ì
÷
÷
ø
ö
ç
ç
è
æ
-
-
÷
÷
ø
ö
ç
ç
è
æ
-
-
--
Proof Theorem 3: Let perform the partitioning[ ]
1
1:1:1
,:
xnkT
kkk
xXX RÎ=
- [ ]
1
|1:1|1:1:1|:1
ˆ,
ˆ
:
ˆ
xnk
T
kkkkkk xXX RÎ=
--
( ){ }
( ){ } ( ){ }
( ){ } ( ){ }
1
:1:1,:1:1,
:1:1,:1:1,1
:1:1,,,
,ln,ln
,ln,ln
,ln
1:1
1:11:11:1
1:11:1
-
-
ú
ú
û
ù
ê
ê
ë
é
ÑÑ-ÑÑ-
ÑÑ-ÑÑ-
=ÑÑ-
-
---
--
kk
T
xxZXkk
T
XxZX
kk
T
xXZXkk
T
XXZX
kk
T
xXxXZX
XZpEXZpE
XZpEXZpE
XZpE
kkkk
kkkk
kkkk
nkxnkkk
kk
T
kk
k
k
T
kk
T
k
kk
I
BAI
BABC
A
IAB
I
CB
BA
RÎ
÷
÷
ø
ö
ç
ç
è
æ
ú
û
ù
ê
ë
é
ú
û
ù
ê
ë
é
-
ú
û
ù
ê
ë
é
=ú
û
ù
ê
ë
é
=
-
-
--
- 1
1
11
1
00
00
:
( )
( )
p
kkk
n
kkk
vxhz
wxfx
R
R
Î=
Î=
--
,
,
11
kk
vw&
1- are system and measurement white-noise sequences
independent of past and current states and on each other and
having known P.D.F.s ( ) ()
kk vpwp &
1-
()
0
xpIn addition the P.D.F. of the initial state , is also given.
After k cycles we have k measurements and k random parameters
estimated by an Unbiased Estimator as .
[ ]
T
kk zzzZ ,,,:
21:1 =
[ ]
T
kk
xxxxX ,,,,:
210:0
= [ ]
T
kkkk xxxX
|2|21|1:1|:1
ˆ,,ˆ,ˆ:
ˆ
=
29
Cramér-Rao Lower Bound (CRLB)SOLO
Discrete Time Nonlinear Estimation
( )( ){ } ( )( ){ }
( )( ){ } ( )( ){ }
( )
0
0
0
0
0
ˆˆˆ
ˆ
1
11
1
11
1
||,1:11:1|1:1|,
|1:11:1|1:1,1:11:1|1:11:11:1|1:1,
definiteSemi
Positive
k
T
k
kk
T
kk
k
kk
T
kkkkkkZX
T
kkkkkkZX
T
kkkkkkZX
T
kkkkkkZX
IAB
I
BABC
A
I
BAI
xxxxEXXxxE
xxXXEXXXXE
-
-
--
-
--
-
---
---------
³ú
û
ù
ê
ë
é
ú
ú
û
ù
ê
ê
ë
é
-
ú
û
ù
ê
ë
é
-
ú
ú
û
ù
ê
ê
ë
é
----
----
Proof Theorem 3 (continue – 1): We found
( )( ){ } ( )( ){ }
( )( ){ } ( )( ){ }
( )
0
0
0
0
ˆˆˆ
ˆ
0
1
1
1
1
||,1:11:1|1:1|,
|1:11:1|1:1,1:11:1|1:11:11:1|1:1,
1
definiteSemi
Positive
kk
T
kk
k
k
T
k
T
kkkkkkZX
T
kkkkkkZX
T
kkkkkkZX
T
kkkkkkZX
kk
BABC
A
IAB
I
xxxxEXXxxE
xxXXEXXXXE
I
BAI
-
-
-
-
-
---
---------
-
³
ú
ú
û
ù
ê
ê
ë
é
-
-
ú
û
ù
ê
ë
é
ú
ú
û
ù
ê
ê
ë
é
----
----
ú
û
ù
ê
ë
é
( )
( )
p
kkk
n
kkk
vxhz
wxfx
R
R
Î=
Î=
--
,
,
11
kk
vw&
1- are system and measurement white-noise sequences
independent of past and current states and on each other and
having known P.D.F.s ( ) ()
kk vpwp &
1-
()
0
xpIn addition the P.D.F. of the initial state , is also given.
Cramér-Rao Lower Bound (CRLB)SOLO
Discrete Time Nonlinear Estimation
( )( ){ } ( )( ){ }
( )( ){ } ( )( ){ }
( )
0
0
0
0
0
ˆˆˆ
ˆ
1
11
1
11
1
||,1:11:1|1:1|,
|1:11:1|1:1,1:11:1|1:11:11:1|1:1,
definiteSemi
Positive
k
T
k
kk
T
kk
k
kk
T
kkkkkkZX
T
kkkkkkZX
T
kkkkkkZX
T
kkkkkkZX
IAB
I
BABC
A
I
BAI
xxxxEXXxxE
xxXXEXXXXE
-
-
--
-
--
-
---
---------
³ú
û
ù
ê
ë
é
ú
ú
û
ù
ê
ê
ë
é
-
ú
û
ù
ê
ë
é
-
ú
ú
û
ù
ê
ê
ë
é
----
----
Proof Theorem 3 (continue – 1): We found
( )( ){ } ( )( ){ }
( )( ){ } ( )( ){ }
( )
0
0
0
0
ˆˆˆ
ˆ
0
1
1
1
1
||,1:11:1|1:1|,
|1:11:1|1:1,1:11:1|1:11:11:1|1:1,
1
definiteSemi
Positive
kk
T
kk
k
k
T
k
T
kkkkkkZX
T
kkkkkkZX
T
kkkkkkZX
T
kkkkkkZX
kk
BABC
A
IAB
I
xxxxEXXxxE
xxXXEXXXXE
I
BAI
-
-
-
-
-
---
---------
-
³
ú
ú
û
ù
ê
ê
ë
é
-
-
ú
û
ù
ê
ë
é
ú
ú
û
ù
ê
ê
ë
é
----
----
ú
û
ù
ê
ë
é
( )
( )
p
kkk
n
kkk
vxhz
wxfx
R
R
Î=
Î=
--
,
,
11
kk
vw&
1- are system and measurement white-noise sequences
independent of past and current states and on each other and
having known P.D.F.s ( ) ()
kk vpwp &
1-
()
0
xpIn addition the P.D.F. of the initial state , is also given.
30
Cramér-Rao Lower Bound (CRLB)
( )( ){ } ( )
1
11
||,
:ˆˆ
-
--
-=³--
kk
T
kkk
T
kkkkkkZX
BABCJxxxxE
SOLO
Discrete Time Nonlinear Estimation
Prof Theorem 3 (continue – 2): We found
( )( ){ } ( )
0
0
0
ˆˆ*
**
1
1
1
||,
definiteSemi
Positive
kk
T
kk
k
T
kkkkkkZX BABC
A
xxxxE
-
-
-
-
³
ú
ú
û
ù
ê
ê
ë
é
-
-
ú
ú
û
ù
ê
ê
ë
é
--
( )( ){ } ( ) 0ˆˆ
1
1
||,
definiteSemi
Positive
kk
T
kk
T
kkkkkkZX BABCxxxxE
-
-
-
³----
Equivalent
Notations
( ){ }
( ) ( )
( ){ }
( )
( ){ }
nxn
kk
T
xxZXk
nxkn
kk
T
xXZXk
knxkn
kk
T
XXZXk
XZpEC
XZpEB
XZpEA
kk
kk
kk
R
R
R
ÎÑÑ-=
ÎÑÑ-=
ÎÑÑ-=
-
--
-
--
:1:1,
1
:1:1,
11
:1:1,
,ln:
,ln:
,ln:
1:1
1:11:1
( )
( )
p
kkk
n
kkk
vxhz
wxfx
R
R
Î=
Î=
--
,
,
11
kk
vw&
1- are system and measurement white-noise sequences
independent of past and current states and on each other and
having known P.D.F.s ( ) ()
kk vpwp &
1-
()
0
xpIn addition the P.D.F. of the initial state , is also given.
q.e.d.
Cramér-Rao Lower Bound (CRLB)
( )( ){ } ( )
1
11
||,
:ˆˆ
-
--
-=³--
kk
T
kkk
T
kkkkkkZX
BABCJxxxxE
SOLO
Discrete Time Nonlinear Estimation
Prof Theorem 3 (continue – 2): We found
( )( ){ } ( )
0
0
0
ˆˆ*
**
1
1
1
||,
definiteSemi
Positive
kk
T
kk
k
T
kkkkkkZX BABC
A
xxxxE
-
-
-
-
³
ú
ú
û
ù
ê
ê
ë
é
-
-
ú
ú
û
ù
ê
ê
ë
é
--
( )( ){ } ( ) 0ˆˆ
1
1
||,
definiteSemi
Positive
kk
T
kk
T
kkkkkkZX BABCxxxxE
-
-
-
³----
Equivalent
Notations
( ){ }
( ) ( )
( ){ }
( )
( ){ }
nxn
kk
T
xxZXk
nxkn
kk
T
xXZXk
knxkn
kk
T
XXZXk
XZpEC
XZpEB
XZpEA
kk
kk
kk
R
R
R
ÎÑÑ-=
ÎÑÑ-=
ÎÑÑ-=
-
--
-
--
:1:1,
1
:1:1,
11
:1:1,
,ln:
,ln:
,ln:
1:1
1:11:1
( )
( )
p
kkk
n
kkk
vxhz
wxfx
R
R
Î=
Î=
--
,
,
11
kk
vw&
1- are system and measurement white-noise sequences
independent of past and current states and on each other and
having known P.D.F.s ( ) ()
kk vpwp &
1-
()
0
xpIn addition the P.D.F. of the initial state , is also given.
q.e.d.
31
Cramér-Rao Lower Bound (CRLB)
( ){ }
( )( )
( ){ }
( )
( ){ }
nxn
kk
T
xxZXk
nxkn
kk
T
xXZXk
knxkn
kk
T
XXZXk
XZpEC
XZpEB
XZpEA
kk
kk
kk
R
R
R
ÎÑÑ-=
ÎÑÑ-=
ÎÑÑ-=
-
--
-
--
:1:1,
1
:1:1,
11
:1:1,
,ln:
,ln:
,ln:
1:1
1:11:1
SOLO
Discrete Time Nonlinear Estimation – Recursive Cramér–Rao Lower Bound
We found
We want to compute J
k
recursively, without the need for inverting large matrices as A
k
.
( )( ){ } ( )
1
11
||, :ˆˆ
-
--
-=³--
kk
T
kkk
T
kkkkkkZX BABCJxxxxE
( )
( )
p
kkk
n
kkk
vxhz
wxfx
R
R
Î=
Î=
--
,
,
11
kkvw&
1- are system and measurement white-noise sequences
independent of past and current states and on each other and
having known P.D.F.s ( ) ()
kk vpwp &
1-
()
0xpIn addition the P.D.F. of the initial state , is also given.
Theorem 4:The Recursive Cramér–Rao Lower Bound for the Random Parameters is given by:
( )( ){ } [ ]( )
nxn
kkkkkk
T
kkkkkkZX DDJDDJxxxxE RÎ+-=³--
-
-
-
+--+--+
1
12
1
1121221
111|111|1, :ˆˆ
( )( ){ } ( )
nxn
kk
T
kkk
T
kkkkkkZX BABCJxxxxE RÎ-=³--
-
--
1
11
||, :ˆˆ
( ){ }
( )[ ]{ }[ ]
( ){ } ( ){ }
nxn
kk
T
kxxzkk
T
kxxxk
nxn
T
kkk
T
kxxxk
nxn
kk
T
kxxxk
xzpExxpED
DxxpED
xxpED
kkkkkk
kkk
kkk
R
R
R
ÎÑÑ-ÑÑ-=
Î=ÑÑ-=
ÎÑÑ-=
+++++
++
+
+++++
+
+
111|11|
22
21
11|
12
1|
11
|ln|ln:
|ln:
|ln:
11111
1
1
() (){ }
000
lnln
000
xpxpEJ
T
xxx
ÑÑ=
The recursions start with the initial
information matrix J
0
,
Cramér-Rao Lower Bound (CRLB)
( ){ }
( )( )
( ){ }
( )
( ){ }
nxn
kk
T
xxZXk
nxkn
kk
T
xXZXk
knxkn
kk
T
XXZXk
XZpEC
XZpEB
XZpEA
kk
kk
kk
R
R
R
ÎÑÑ-=
ÎÑÑ-=
ÎÑÑ-=
-
--
-
--
:1:1,
1
:1:1,
11
:1:1,
,ln:
,ln:
,ln:
1:1
1:11:1
SOLO
Discrete Time Nonlinear Estimation – Recursive Cramér–Rao Lower Bound
We found
We want to compute J
k
recursively, without the need for inverting large matrices as A
k
.
( )( ){ } ( )
1
11
||, :ˆˆ
-
--
-=³--
kk
T
kkk
T
kkkkkkZX BABCJxxxxE
( )
( )
p
kkk
n
kkk
vxhz
wxfx
R
R
Î=
Î=
--
,
,
11
kkvw&
1- are system and measurement white-noise sequences
independent of past and current states and on each other and
having known P.D.F.s ( ) ()
kk vpwp &
1-
()
0xpIn addition the P.D.F. of the initial state , is also given.
Theorem 4:The Recursive Cramér–Rao Lower Bound for the Random Parameters is given by:
( )( ){ } [ ]( )
nxn
kkkkkk
T
kkkkkkZX DDJDDJxxxxE RÎ+-=³--
-
-
-
+--+--+
1
12
1
1121221
111|111|1, :ˆˆ
( )( ){ } ( )
nxn
kk
T
kkk
T
kkkkkkZX BABCJxxxxE RÎ-=³--
-
--
1
11
||, :ˆˆ
( ){ }
( )[ ]{ }[ ]
( ){ } ( ){ }
nxn
kk
T
kxxzkk
T
kxxxk
nxn
T
kkk
T
kxxxk
nxn
kk
T
kxxxk
xzpExxpED
DxxpED
xxpED
kkkkkk
kkk
kkk
R
R
R
ÎÑÑ-ÑÑ-=
Î=ÑÑ-=
ÎÑÑ-=
+++++
++
+
+++++
+
+
111|11|
22
21
11|
12
1|
11
|ln|ln:
|ln:
|ln:
11111
1
1
() (){ }
000
lnln
000
xpxpEJ
T
xxx
ÑÑ=
The recursions start with the initial
information matrix J
0
,
32
Cramér-Rao Lower Bound (CRLB)SOLO
( ){ }
( ){ }
( ){ }
kk
T
xxZXk
kk
T
xXZXk
kk
T
XXZXk
XZpEC
XZpEB
XZpEA
kk
kk
kk
:1:1,
:1:1,
:1:1,
,ln:
,ln:
,ln:
1:1
1:11:1
ÑÑ-=
ÑÑ-=
ÑÑ-=
-
--
We found
We want to compute J
k
recursively, without the need for inverting large matrices as A
k
.
( )( ){ } ( )
1
11
||, :ˆˆ
-
--
-=³--
kk
T
kkk
T
kkkkkkZX BABCJxxxxE
Start with:
( ) ( ) ( ) ( )
kkkkkkkkkkkkk
XxZpXxZzpXxZzpXZp
:11:1:11:11:11:111:11:1
,,,,|,,,,
+++++++
==
( )
( )
( )
( )
( )
kk
xxpMarkov
kkk
xzpMarkov
kkkk
XZpXZxpXxZzp
kkkk
:1:1
|
:1:11
|
:11:11
,,|,,|
111
+++
®
+
®
++
=
( )( )( )
1:11:11
,||
---
=
kkkkkk
XZpxxpxzp
( )
( )
p
kkk
n
kkk
vxhz
wxfx
R
R
Î=
Î=
--
,
,
11
kk
vw&
1- are system and measurement white-noise sequences
independent of past and current states and on each other and
having known P.D.F.s ( ) ()
kk vpwp &
1-
()
0
xpIn addition the P.D.F. of the initial state , is also given.
Proof of Theorem 4:
Discrete Time Nonlinear Estimation – Recursive Cramér–Rao Lower Bound
Cramér-Rao Lower Bound (CRLB)SOLO
( ){ }
( ){ }
( ){ }
kk
T
xxZXk
kk
T
xXZXk
kk
T
XXZXk
XZpEC
XZpEB
XZpEA
kk
kk
kk
:1:1,
:1:1,
:1:1,
,ln:
,ln:
,ln:
1:1
1:11:1
ÑÑ-=
ÑÑ-=
ÑÑ-=
-
--
We found
We want to compute J
k
recursively, without the need for inverting large matrices as A
k
.
( )( ){ } ( )
1
11
||, :ˆˆ
-
--
-=³--
kk
T
kkk
T
kkkkkkZX BABCJxxxxE
Start with:
( ) ( ) ( ) ( )
kkkkkkkkkkkkk
XxZpXxZzpXxZzpXZp
:11:1:11:11:11:111:11:1
,,,,|,,,,
+++++++
==
( )
( )
( )
( )
( )
kk
xxpMarkov
kkk
xzpMarkov
kkkk
XZpXZxpXxZzp
kkkk
:1:1
|
:1:11
|
:11:11
,,|,,|
111
+++
®
+
®
++
=
( )( )( )
1:11:11
,||
---
=
kkkkkk
XZpxxpxzp
( )
( )
p
kkk
n
kkk
vxhz
wxfx
R
R
Î=
Î=
--
,
,
11
kk
vw&
1- are system and measurement white-noise sequences
independent of past and current states and on each other and
having known P.D.F.s ( ) ()
kk vpwp &
1-
()
0
xpIn addition the P.D.F. of the initial state , is also given.
Proof of Theorem 4:
Discrete Time Nonlinear Estimation – Recursive Cramér–Rao Lower Bound
33
Cramér-Rao Lower Bound (CRLB)
( )
1
1
1
111
111
111
1
1:11:1,
11|1
|
1:11:1|1:1
11|1
|
1:11:1|1:1
, :,ln
ˆ
ˆ
ˆ
ˆ
1111:11
11:1
11:11:11:11:1
-
+
-
+++
+++
+++
-
++
+++
---
+++
---
=
ú
ú
ú
û
ù
ê
ê
ê
ë
é
=
ï
ï
þ
ï
ï
ý
ü
ï
ï
î
ï
ï
í
ì
÷
÷
÷
÷
ø
ö
ç
ç
ç
ç
è
æ
ÑÑÑÑÑÑ
ÑÑÑÑÑÑ
ÑÑÑÑÑÑ
-³
ï
ï
þ
ï
ï
ý
ü
ï
ï
î
ï
ï
í
ì
÷
÷
÷
÷
ø
ö
ç
ç
ç
ç
è
æ
-
-
-
÷
÷
÷
÷
ø
ö
ç
ç
ç
ç
è
æ
-
-
-
+++-+
+-
+----
k
k
T
k
T
k
kk
T
k
kkk
kk
T
xx
T
xx
T
Xx
T
xx
T
xx
T
Xx
T
xX
T
xX
T
XX
ZX
T
kkk
kkk
kkk
kkk
kkk
kkk
ZX I
FEL
ECB
LBA
XZpE
xx
xx
XX
xx
xx
XX
E
kkkkkk
kkkkkk
kkkkkk
SOLO
Proof of Theorem 4 (continue – 1):
Compute:
( ) ( )( )( )
kkkkkkkk XZpxxpxzpXZp
:1:11111:11:1 ,||,
+++++ =
( ){ } ( ) ( ) ( )[ ]{ }
( )[ ]{ }
kkk
T
XXZX
kkkkkk
T
XXZXkk
T
XXZXk
AXZpE
XZpxxpxzpEXZpEA
kk
kkkk
=ÑÑ-+=
++ÑÑ-=ÑÑ-=
--
----
++++++
:1:1,
:1:1111,1:11:1,1
,ln00
,ln|ln|ln,ln:
1:11:1
1:11:11:11:1
( ){ } ( ) ( ) ( )[ ]{ }
( )[ ]{ }
kkk
T
xXZX
kkkkkk
T
xXZXkk
T
xXZXk
BXZpE
XZpxxpxzpEXZpEB
kk
kkkk
=ÑÑ-+=
++ÑÑ-=ÑÑ-=
-
--
++++++
:1:1,
:1:1111,1:11:1,1
,ln00
,ln|ln|ln,ln:
1:1
1:11:1
( ){ } ( ) ( ) ( )[ ]{ }
( )[ ]{ } ( )[ ]{ }
11
:1:1,1|
:1:1111,1:11:1,1
,ln|ln0
,ln|ln|ln,ln:
11
1 kk
C
kk
T
xxZX
D
kk
T
xxxx
kkkkkk
T
xxZXkk
T
xxZXk
DCXZpExxpE
XZpxxpxzpEXZpEC
k
kk
k
kkkk
kkkk
+=ÑÑ-ÑÑ-=
++ÑÑ-=ÑÑ-=
+
++++++
+
( )
( )
p
kkk
n
kkk
vxhz
wxfx
R
R
Î=
Î=
--
,
,
11
kk
vw&
1- are system and measurement white-noise sequences
independent of past and current states and on each other and
having known P.D.F.s ( ) ()
kk vpwp &
1-
()
0
xpIn addition the P.D.F. of the initial state , is also given.
Discrete Time Nonlinear Estimation – Recursive Cramér–Rao Lower Bound
Cramér-Rao Lower Bound (CRLB)
( )
1
1
1
111
111
111
1
1:11:1,
11|1
|
1:11:1|1:1
11|1
|
1:11:1|1:1
, :,ln
ˆ
ˆ
ˆ
ˆ
1111:11
11:1
11:11:11:11:1
-
+
-
+++
+++
+++
-
++
+++
---
+++
---
=
ú
ú
ú
û
ù
ê
ê
ê
ë
é
=
ï
ï
þ
ï
ï
ý
ü
ï
ï
î
ï
ï
í
ì
÷
÷
÷
÷
ø
ö
ç
ç
ç
ç
è
æ
ÑÑÑÑÑÑ
ÑÑÑÑÑÑ
ÑÑÑÑÑÑ
-³
ï
ï
þ
ï
ï
ý
ü
ï
ï
î
ï
ï
í
ì
÷
÷
÷
÷
ø
ö
ç
ç
ç
ç
è
æ
-
-
-
÷
÷
÷
÷
ø
ö
ç
ç
ç
ç
è
æ
-
-
-
+++-+
+-
+----
k
k
T
k
T
k
kk
T
k
kkk
kk
T
xx
T
xx
T
Xx
T
xx
T
xx
T
Xx
T
xX
T
xX
T
XX
ZX
T
kkk
kkk
kkk
kkk
kkk
kkk
ZX I
FEL
ECB
LBA
XZpE
xx
xx
XX
xx
xx
XX
E
kkkkkk
kkkkkk
kkkkkk
SOLO
Proof of Theorem 4 (continue – 1):
Compute:
( ) ( )( )( )
kkkkkkkk XZpxxpxzpXZp
:1:11111:11:1 ,||,
+++++ =
( ){ } ( ) ( ) ( )[ ]{ }
( )[ ]{ }
kkk
T
XXZX
kkkkkk
T
XXZXkk
T
XXZXk
AXZpE
XZpxxpxzpEXZpEA
kk
kkkk
=ÑÑ-+=
++ÑÑ-=ÑÑ-=
--
----
++++++
:1:1,
:1:1111,1:11:1,1
,ln00
,ln|ln|ln,ln:
1:11:1
1:11:11:11:1
( ){ } ( ) ( ) ( )[ ]{ }
( )[ ]{ }
kkk
T
xXZX
kkkkkk
T
xXZXkk
T
xXZXk
BXZpE
XZpxxpxzpEXZpEB
kk
kkkk
=ÑÑ-+=
++ÑÑ-=ÑÑ-=
-
--
++++++
:1:1,
:1:1111,1:11:1,1
,ln00
,ln|ln|ln,ln:
1:1
1:11:1
( ){ } ( ) ( ) ( )[ ]{ }
( )[ ]{ } ( )[ ]{ }
11
:1:1,1|
:1:1111,1:11:1,1
,ln|ln0
,ln|ln|ln,ln:
11
1 kk
C
kk
T
xxZX
D
kk
T
xxxx
kkkkkk
T
xxZXkk
T
xxZXk
DCXZpExxpE
XZpxxpxzpEXZpEC
k
kk
k
kkkk
kkkk
+=ÑÑ-ÑÑ-=
++ÑÑ-=ÑÑ-=
+
++++++
+
( )
( )
p
kkk
n
kkk
vxhz
wxfx
R
R
Î=
Î=
--
,
,
11
kk
vw&
1- are system and measurement white-noise sequences
independent of past and current states and on each other and
having known P.D.F.s ( ) ()
kk vpwp &
1-
()
0
xpIn addition the P.D.F. of the initial state , is also given.
Discrete Time Nonlinear Estimation – Recursive Cramér–Rao Lower Bound
34
Cramér-Rao Lower Bound (CRLB)
( )
1
1
1
111
111
111
1
1:11:1,
11|1
|
1:11:1|1:1
11|1
|
1:11:1|1:1
, :,ln
ˆ
ˆ
ˆ
ˆ
1111:11
11:1
11:11:11:11:1
-
+
-
+++
+++
+++
-
++
+++
---
+++
---
=
ú
ú
ú
û
ù
ê
ê
ê
ë
é
=
ï
ï
þ
ï
ï
ý
ü
ï
ï
î
ï
ï
í
ì
÷
÷
÷
÷
ø
ö
ç
ç
ç
ç
è
æ
ÑÑÑÑÑÑ
ÑÑÑÑÑÑ
ÑÑÑÑÑÑ
-³
ï
ï
þ
ï
ï
ý
ü
ï
ï
î
ï
ï
í
ì
÷
÷
÷
÷
ø
ö
ç
ç
ç
ç
è
æ
-
-
-
÷
÷
÷
÷
ø
ö
ç
ç
ç
ç
è
æ
-
-
-
+++-+
+-
+----
k
k
T
k
T
k
kk
T
k
kkk
kk
T
xx
T
xx
T
Xx
T
xx
T
xx
T
Xx
T
xX
T
xX
T
XX
ZX
T
kkk
kkk
kkk
kkk
kkk
kkk
ZX I
FEL
ECB
LBA
XZpE
xx
xx
XX
xx
xx
XX
E
kkkkkk
kkkkkk
kkkkkk
SOLO
Proof of Theorem 4 (continue – 2):
Compute:
( ) ( )( )( )
kkkkkkkk XZpxxpxzpXZp
:1:11111:11:1 ,||,
+++++ =
( ){ } ( ) ( ) ( )[ ]{ }
( )[ ] ( )[ ] ( )[ ]0,ln|ln|ln
,ln|ln|ln,ln:
0
:1:1,
0
1,
0
11,
:1:1111,1:11:1,1
11:111:111:1
11:111:1
=
ï
þ
ï
ý
ü
ï
î
ï
í
ì
ÑÑ-
ï
þ
ï
ý
ü
ï
î
ï
í
ì
ÑÑ-
ï
þ
ï
ý
ü
ï
î
ï
í
ì
ÑÑ-=
++ÑÑ-=ÑÑ-=
+-+-+-
+-+-
+++
++++++
kk
T
xXZXkk
T
xXZXkk
T
xXZX
kkkkkk
T
xXZXkk
T
xXZXk
XZpExxpExzpE
XZpxxpxzpEXZpEL
kkkkkk
kkkk
( ){ } ( ) ( ) ( )[ ]{ }
( )[ ] ( )[ ]{ } ( )[ ] ( )[ ]{ }
12
1|
0
:1:1,1,
0
11,
:1:1111,1:11:1,1
:|ln,ln|ln|ln
,ln|ln|ln,ln:
11111
11
kkk
T
xxxxkk
T
xxZXkk
T
xxZXkk
T
xxZX
kkkkkk
T
xxZXkk
T
xxZXk
DxxpEXZpExxpExzpE
XZpxxpxzpEXZpEE
kkkkkkkkkk
kkkk
=ÑÑ-=
ï
þ
ï
ý
ü
ï
î
ï
í
ì
ÑÑ-ÑÑ-
ï
þ
ï
ý
ü
ï
î
ï
í
ì
ÑÑ-=
++ÑÑ-=ÑÑ-=
++++
++++++
+++++
++
( ){ } ( ) ( ) ( )[ ]{ }
( )[ ]{ } ( )[ ]{ } ( )[ ]
22
0
:1:1,1|11|
:1:1111,1:11:1,1
,ln|ln|ln
,ln|ln|ln,ln:
111111111
1111
kkk
T
xxZXkk
T
xxxxkk
T
xxxz
kkkkkk
T
xxZXkk
T
xxZXk
DXZpExxpExzpE
XZpxxpxzpEXZpEF
kkkkkkkkkk
kkkk
=
ï
þ
ï
ý
ü
ï
î
ï
í
ì
ÑÑ-ÑÑ-ÑÑ-=
++ÑÑ-=ÑÑ-=
+++++++++
++++
+++
++++++
( )
( )
p
kkk
n
kkk
vxhz
wxfx
R
R
Î=
Î=
--
,
,
11
kk
vw&
1- are system and measurement white-noise sequences
independent of past and current states and on each other and
having known P.D.F.s ( ) ()
kk vpwp &
1-
()
0
xpIn addition the P.D.F. of the initial state , is also given.
Discrete Time Nonlinear Estimation – Recursive Cramér–Rao Lower Bound
Cramér-Rao Lower Bound (CRLB)
( )
1
1
1
111
111
111
1
1:11:1,
11|1
|
1:11:1|1:1
11|1
|
1:11:1|1:1
, :,ln
ˆ
ˆ
ˆ
ˆ
1111:11
11:1
11:11:11:11:1
-
+
-
+++
+++
+++
-
++
+++
---
+++
---
=
ú
ú
ú
û
ù
ê
ê
ê
ë
é
=
ï
ï
þ
ï
ï
ý
ü
ï
ï
î
ï
ï
í
ì
÷
÷
÷
÷
ø
ö
ç
ç
ç
ç
è
æ
ÑÑÑÑÑÑ
ÑÑÑÑÑÑ
ÑÑÑÑÑÑ
-³
ï
ï
þ
ï
ï
ý
ü
ï
ï
î
ï
ï
í
ì
÷
÷
÷
÷
ø
ö
ç
ç
ç
ç
è
æ
-
-
-
÷
÷
÷
÷
ø
ö
ç
ç
ç
ç
è
æ
-
-
-
+++-+
+-
+----
k
k
T
k
T
k
kk
T
k
kkk
kk
T
xx
T
xx
T
Xx
T
xx
T
xx
T
Xx
T
xX
T
xX
T
XX
ZX
T
kkk
kkk
kkk
kkk
kkk
kkk
ZX I
FEL
ECB
LBA
XZpE
xx
xx
XX
xx
xx
XX
E
kkkkkk
kkkkkk
kkkkkk
SOLO
Proof of Theorem 4 (continue – 2):
Compute:
( ) ( )( )( )
kkkkkkkk XZpxxpxzpXZp
:1:11111:11:1 ,||,
+++++ =
( ){ } ( ) ( ) ( )[ ]{ }
( )[ ] ( )[ ] ( )[ ]0,ln|ln|ln
,ln|ln|ln,ln:
0
:1:1,
0
1,
0
11,
:1:1111,1:11:1,1
11:111:111:1
11:111:1
=
ï
þ
ï
ý
ü
ï
î
ï
í
ì
ÑÑ-
ï
þ
ï
ý
ü
ï
î
ï
í
ì
ÑÑ-
ï
þ
ï
ý
ü
ï
î
ï
í
ì
ÑÑ-=
++ÑÑ-=ÑÑ-=
+-+-+-
+-+-
+++
++++++
kk
T
xXZXkk
T
xXZXkk
T
xXZX
kkkkkk
T
xXZXkk
T
xXZXk
XZpExxpExzpE
XZpxxpxzpEXZpEL
kkkkkk
kkkk
( ){ } ( ) ( ) ( )[ ]{ }
( )[ ] ( )[ ]{ } ( )[ ] ( )[ ]{ }
12
1|
0
:1:1,1,
0
11,
:1:1111,1:11:1,1
:|ln,ln|ln|ln
,ln|ln|ln,ln:
11111
11
kkk
T
xxxxkk
T
xxZXkk
T
xxZXkk
T
xxZX
kkkkkk
T
xxZXkk
T
xxZXk
DxxpEXZpExxpExzpE
XZpxxpxzpEXZpEE
kkkkkkkkkk
kkkk
=ÑÑ-=
ï
þ
ï
ý
ü
ï
î
ï
í
ì
ÑÑ-ÑÑ-
ï
þ
ï
ý
ü
ï
î
ï
í
ì
ÑÑ-=
++ÑÑ-=ÑÑ-=
++++
++++++
+++++
++
( ){ } ( ) ( ) ( )[ ]{ }
( )[ ]{ } ( )[ ]{ } ( )[ ]
22
0
:1:1,1|11|
:1:1111,1:11:1,1
,ln|ln|ln
,ln|ln|ln,ln:
111111111
1111
kkk
T
xxZXkk
T
xxxxkk
T
xxxz
kkkkkk
T
xxZXkk
T
xxZXk
DXZpExxpExzpE
XZpxxpxzpEXZpEF
kkkkkkkkkk
kkkk
=
ï
þ
ï
ý
ü
ï
î
ï
í
ì
ÑÑ-ÑÑ-ÑÑ-=
++ÑÑ-=ÑÑ-=
+++++++++
++++
+++
++++++
( )
( )
p
kkk
n
kkk
vxhz
wxfx
R
R
Î=
Î=
--
,
,
11
kk
vw&
1- are system and measurement white-noise sequences
independent of past and current states and on each other and
having known P.D.F.s ( ) ()
kk vpwp &
1-
()
0
xpIn addition the P.D.F. of the initial state , is also given.
Discrete Time Nonlinear Estimation – Recursive Cramér–Rao Lower Bound
35
Cramér-Rao Lower Bound (CRLB)
1
2221
1211
1
111
111
111
1
1
11|1
|
1:11:1|1:1
11|1
|
1:11:1|1:1
,
0
0
:
ˆ
ˆ
ˆ
ˆ
--
+++
+++
+++
-
+
+++
---
+++
---
ú
ú
ú
û
ù
ê
ê
ê
ë
é
+=
ú
ú
ú
û
ù
ê
ê
ê
ë
é
=³
ï
ï
þ
ï
ï
ý
ü
ï
ï
î
ï
ï
í
ì
÷
÷
÷
÷
ø
ö
ç
ç
ç
ç
è
æ
-
-
-
÷
÷
÷
÷
ø
ö
ç
ç
ç
ç
è
æ
-
-
-
kk
kkk
T
k
kk
k
T
k
T
k
kk
T
k
kkk
k
T
kkk
kkk
kkk
kkk
kkk
kkk
ZX
DD
DDCB
BA
FEL
ECB
LBA
I
xx
xx
XX
xx
xx
XX
E
SOLO
Proof of Theorem 4 (continue – 3):
We found:
[ ]
[ ]
ú
ú
ú
û
ù
ê
ê
ê
ë
é
ú
û
ù
ê
ë
é
ú
û
ù
ê
ë
é
+
ú
ú
ú
ú
ú
û
ù
ê
ê
ê
ê
ê
ë
é
ú
û
ù
ê
ë
é
ú
û
ù
ê
ë
é
+
-
ú
û
ù
ê
ë
é
+
ú
ú
ú
û
ù
ê
ê
ê
ë
é
ú
û
ù
ê
ë
é
+
=
-
-
-
+
I
DDCB
BA
I
DDCB
BA
DD
DCB
BA
I
DCB
BA
D
I
I
kkk
T
k
kk
kkk
T
k
kk
kk
kk
T
k
kk
kk
T
k
kk
k
k
0
0
0
00
0
0
0
12
1
11
12
1
11
2122
11
1
11
21
1
Therefore: ( )( ){ }
[ ] [ ]
12
1
1112122
12
1
11
2122
1
1
111|111|1,
0
0:
ˆˆ
kkk
T
kkkkk
kkk
T
k
kk
kkk
k
T
kkkkkkZX
DBABDCDD
DDCB
BA
DDJ
JxxxxE
-
-
-
+
-
+++++++
-+-=ú
û
ù
ê
ë
é
ú
û
ù
ê
ë
é
+
-=
³--
( )
( )
p
kkk
n
kkk
vxhz
wxfx
R
R
Î=
Î=
--
,
,
11
kk
vw&
1- are system and measurement white-noise sequences
independent of past and current states and on each other and
having known P.D.F.s ( ) ()
kk vpwp &
1-
()
0
xpIn addition the P.D.F. of the initial state , is also given.
Discrete Time Nonlinear Estimation – Recursive Cramér–Rao Lower Bound
Cramér-Rao Lower Bound (CRLB)
1
2221
1211
1
111
111
111
1
1
11|1
|
1:11:1|1:1
11|1
|
1:11:1|1:1
,
0
0
:
ˆ
ˆ
ˆ
ˆ
--
+++
+++
+++
-
+
+++
---
+++
---
ú
ú
ú
û
ù
ê
ê
ê
ë
é
+=
ú
ú
ú
û
ù
ê
ê
ê
ë
é
=³
ï
ï
þ
ï
ï
ý
ü
ï
ï
î
ï
ï
í
ì
÷
÷
÷
÷
ø
ö
ç
ç
ç
ç
è
æ
-
-
-
÷
÷
÷
÷
ø
ö
ç
ç
ç
ç
è
æ
-
-
-
kk
kkk
T
k
kk
k
T
k
T
k
kk
T
k
kkk
k
T
kkk
kkk
kkk
kkk
kkk
kkk
ZX
DD
DDCB
BA
FEL
ECB
LBA
I
xx
xx
XX
xx
xx
XX
E
SOLO
Proof of Theorem 4 (continue – 3):
We found:
[ ]
[ ]
ú
ú
ú
û
ù
ê
ê
ê
ë
é
ú
û
ù
ê
ë
é
ú
û
ù
ê
ë
é
+
ú
ú
ú
ú
ú
û
ù
ê
ê
ê
ê
ê
ë
é
ú
û
ù
ê
ë
é
ú
û
ù
ê
ë
é
+
-
ú
û
ù
ê
ë
é
+
ú
ú
ú
û
ù
ê
ê
ê
ë
é
ú
û
ù
ê
ë
é
+
=
-
-
-
+
I
DDCB
BA
I
DDCB
BA
DD
DCB
BA
I
DCB
BA
D
I
I
kkk
T
k
kk
kkk
T
k
kk
kk
kk
T
k
kk
kk
T
k
kk
k
k
0
0
0
00
0
0
0
12
1
11
12
1
11
2122
11
1
11
21
1
Therefore: ( )( ){ }
[ ] [ ]
12
1
1112122
12
1
11
2122
1
1
111|111|1,
0
0:
ˆˆ
kkk
T
kkkkk
kkk
T
k
kk
kkk
k
T
kkkkkkZX
DBABDCDD
DDCB
BA
DDJ
JxxxxE
-
-
-
+
-
+++++++
-+-=ú
û
ù
ê
ë
é
ú
û
ù
ê
ë
é
+
-=
³--
( )
( )
p
kkk
n
kkk
vxhz
wxfx
R
R
Î=
Î=
--
,
,
11
kk
vw&
1- are system and measurement white-noise sequences
independent of past and current states and on each other and
having known P.D.F.s ( ) ()
kk vpwp &
1-
()
0
xpIn addition the P.D.F. of the initial state , is also given.
Discrete Time Nonlinear Estimation – Recursive Cramér–Rao Lower Bound
36
Cramér-Rao Lower Bound (CRLB)
SOLO
The recursions start with the initial information matrix J
0
, which can be computed
from the initial density p (x
0
) as follows:
( )( ){ } [ ]( )
1
12
1
1121221
111|111|1, :ˆˆ
-
-
-
+--+--+ +-=³--
kkkkkk
T
kkkkkkZX DDJDDJxxxxE
( ){ }
( ){ }
( ){ }
kk
T
xxZXk
kk
T
xXZXk
kk
T
XXZXk
XZpEC
XZpEB
XZpEA
kk
kk
kk
:1:1,
:1:1,
:1:1,
,ln:
,ln:
,ln:
1:1
1:11:1
ÑÑ-=
ÑÑ-=
ÑÑ-=
-
--
Proof of Theorem 4 (continue – 4):
( )( ){ } ( )
1
11
||, :ˆˆ
-
--
-=³--
kk
T
kkk
T
kkkkkkZX BABCJxxxxE
( ){ }
( )[ ]{ } [ ]
( ){ } ( ){ }
11|1|
22
21
1|
12
1|
11
|ln|ln:
|ln:
|ln:
1111111
11
1
+++
+
+
+++++++
++
+
ÑÑ-ÑÑ-=
=ÑÑ-=
ÑÑ-=
kk
T
xxxzkk
T
xxxxk
T
kkk
T
xxxxk
kk
T
xxxxk
xzpExxpED
DxxpED
xxpED
kkkkkkkk
kkkk
kkkk
( ) ( ){ }
000 lnln
000
xpxpEJ
T
xxx ÑÑ=
( )
( )
p
kkk
n
kkk
vxhz
wxfx
R
R
Î=
Î=
--
,
,
11
kk
vw&
1- are system and measurement white-noise sequences
independent of past and current states and on each other and
having known P.D.F.s ( ) ()
kk vpwp &
1-
()
0
xpIn addition the P.D.F. of the initial state , is also given.
Discrete Time Nonlinear Estimation – Recursive Cramér–Rao Lower Bound
Cramér-Rao Lower Bound (CRLB)
SOLO
The recursions start with the initial information matrix J
0
, which can be computed
from the initial density p (x
0
) as follows:
( )( ){ } [ ]( )
1
12
1
1121221
111|111|1, :ˆˆ
-
-
-
+--+--+ +-=³--
kkkkkk
T
kkkkkkZX DDJDDJxxxxE
( ){ }
( ){ }
( ){ }
kk
T
xxZXk
kk
T
xXZXk
kk
T
XXZXk
XZpEC
XZpEB
XZpEA
kk
kk
kk
:1:1,
:1:1,
:1:1,
,ln:
,ln:
,ln:
1:1
1:11:1
ÑÑ-=
ÑÑ-=
ÑÑ-=
-
--
Proof of Theorem 4 (continue – 4):
( )( ){ } ( )
1
11
||, :ˆˆ
-
--
-=³--
kk
T
kkk
T
kkkkkkZX BABCJxxxxE
( ){ }
( )[ ]{ } [ ]
( ){ } ( ){ }
11|1|
22
21
1|
12
1|
11
|ln|ln:
|ln:
|ln:
1111111
11
1
+++
+
+
+++++++
++
+
ÑÑ-ÑÑ-=
=ÑÑ-=
ÑÑ-=
kk
T
xxxzkk
T
xxxxk
T
kkk
T
xxxxk
kk
T
xxxxk
xzpExxpED
DxxpED
xxpED
kkkkkkkk
kkkk
kkkk
( ) ( ){ }
000 lnln
000
xpxpEJ
T
xxx ÑÑ=
( )
( )
p
kkk
n
kkk
vxhz
wxfx
R
R
Î=
Î=
--
,
,
11
kk
vw&
1- are system and measurement white-noise sequences
independent of past and current states and on each other and
having known P.D.F.s ( ) ()
kk vpwp &
1-
()
0
xpIn addition the P.D.F. of the initial state , is also given.
Discrete Time Nonlinear Estimation – Recursive Cramér–Rao Lower Bound
37
Cramér-Rao Lower Bound (CRLB)
SOLO
( )( ){ } [ ]( )
1
12
1
1121221
111|111|1, :ˆˆ
-
-
-
+--+--+ +-=³--
kkkkkk
T
kkkkkkZX DDJDDJxxxxE
Proof of Theorem 4 (continue – 5):
( ){ }
( )[ ]{ }[ ]
() ()
() ( ){ } () ( ){ }
nxn
kk
T
xxxzk
nxn
kk
T
xxxxk
kkk
nxn
T
kkk
T
xxxxk
nxn
kk
T
xxxxk
xzpEDxxpED
DDD
DxxpED
xxpED
kkkkkkkk
kkkk
kkkk
RR
R
R
ÎÑÑ-=ÎÑÑ-=
+=
Î=ÑÑ-=
ÎÑÑ-=
+++
+
+
+++++++
++
+
11|
22
1|
22
222222
21
1|
12
1|
11
|ln:2|ln:1
21:
|ln:
|ln:
1111111
11
1
( )
( )
p
kkk
n
kkk
vxhz
wxfx
R
R
Î=
Î=
--
,
,
11
kk
vw&
1- are system and measurement white-noise sequences
independent of past and current states and on each other and
having known P.D.F.s ( ) ()
kk vpwp &
1-
()
0
xpIn addition the P.D.F. of the initial state , is also given.
q.e.d.
() [ ] ()
tMeasuremen
Updated
22
ModelProcess
UsingPrediction
12
1
112122
1
21:
kkkkkkk
DDDJDDJ ++-=
-
+
Discrete Time Nonlinear Estimation – Recursive Cramér–Rao Lower Bound
Cramér-Rao Lower Bound (CRLB)
SOLO
( )( ){ } [ ]( )
1
12
1
1121221
111|111|1, :ˆˆ
-
-
-
+--+--+ +-=³--
kkkkkk
T
kkkkkkZX DDJDDJxxxxE
Proof of Theorem 4 (continue – 5):
( ){ }
( )[ ]{ }[ ]
() ()
() ( ){ } () ( ){ }
nxn
kk
T
xxxzk
nxn
kk
T
xxxxk
kkk
nxn
T
kkk
T
xxxxk
nxn
kk
T
xxxxk
xzpEDxxpED
DDD
DxxpED
xxpED
kkkkkkkk
kkkk
kkkk
RR
R
R
ÎÑÑ-=ÎÑÑ-=
+=
Î=ÑÑ-=
ÎÑÑ-=
+++
+
+
+++++++
++
+
11|
22
1|
22
222222
21
1|
12
1|
11
|ln:2|ln:1
21:
|ln:
|ln:
1111111
11
1
( )
( )
p
kkk
n
kkk
vxhz
wxfx
R
R
Î=
Î=
--
,
,
11
kk
vw&
1- are system and measurement white-noise sequences
independent of past and current states and on each other and
having known P.D.F.s ( ) ()
kk vpwp &
1-
()
0
xpIn addition the P.D.F. of the initial state , is also given.
q.e.d.
() [ ] ()
tMeasuremen
Updated
22
ModelProcess
UsingPrediction
12
1
112122
1
21:
kkkkkkk
DDDJDDJ ++-=
-
+
Discrete Time Nonlinear Estimation – Recursive Cramér–Rao Lower Bound
38
Cramér-Rao Lower Bound (CRLB)
SOLO
Discrete Time Nonlinear Estimation –Special Cases
() ( ) ( ) ( )
ú
û
ù
ê
ë
é
---==
-
00
1
000
0
0000
ˆˆ
2
1
exp
2
1
,ˆ;
0
xxPxx
P
Pxxxp
T
x
p
N
( )
( )
p
kkk
n
kkk
vxhz
wxfx
R
R
Î=
Î=
--
,
,
11
kk
vw&
1- are system and measurement white-noise sequences
independent of past and current states and on each other and
having known P.D.F.s ( ) ()
kk vpwp &
1-
()
0
xpIn addition the P.D.F. of the initial state , is also given.
Probability Density Function of is Gaussian0
x
() ( ) ( ) ( )
00
1
000
1
0000
ˆˆˆ
2
1
ln
000
xxPxxPxxcxp
T
xxx --=
þ
ý
ü
î
í
ì
---Ñ=Ñ
--
() (){ } ( )( )[ ]{ }
( )( ){ }
1
0
1
00
1
0
1
00000
1
0
1
00000
1
0000
ˆˆ
ˆˆlnln
0
000000
-----
--
==--=
--=ÑÑ=
PPPPPxxxxEP
PxxxxPExpxpEJ
T
x
TT
xx
T
xxxx
Return to Table of Content
Cramér-Rao Lower Bound (CRLB)
SOLO
Discrete Time Nonlinear Estimation –Special Cases
() ( ) ( ) ( )
ú
û
ù
ê
ë
é
---==
-
00
1
000
0
0000
ˆˆ
2
1
exp
2
1
,ˆ;
0
xxPxx
P
Pxxxp
T
x
p
N
( )
( )
p
kkk
n
kkk
vxhz
wxfx
R
R
Î=
Î=
--
,
,
11
kk
vw&
1- are system and measurement white-noise sequences
independent of past and current states and on each other and
having known P.D.F.s ( ) ()
kk vpwp &
1-
()
0
xpIn addition the P.D.F. of the initial state , is also given.
Probability Density Function of is Gaussian0
x
() ( ) ( ) ( )
00
1
000
1
0000
ˆˆˆ
2
1
ln
000
xxPxxPxxcxp
T
xxx --=
þ
ý
ü
î
í
ì
---Ñ=Ñ
--
() (){ } ( )( )[ ]{ }
( )( ){ }
1
0
1
00
1
0
1
00000
1
0
1
00000
1
0000
ˆˆ
ˆˆlnln
0
000000
-----
--
==--=
--=ÑÑ=
PPPPPxxxxEP
PxxxxPExpxpEJ
T
x
TT
xx
T
xxxx
Return to Table of Content
39
Cramér-Rao Lower Bound (CRLB)
SOLO
Discrete Time Nonlinear Estimation –Special Cases
( ) () ( ) ()( ) ()( )
ú
û
ù
ê
ë
é
---===
+
-
++ kkkk
T
kkk
k
kkkwkk
xfxQxfx
Q
Qwwpxxp
1
1
11
2
1
exp
2
1
,0;|
p
N
()
( )
p
kkkk
n
kkkk
vxhz
wxfx
R
R
Î+=
Î+=
++++
+
1111
1
1
&
+kk
vw are system and measurement Gaussian white-noise
sequences, independent of past and current states and on each
other with covariances Q
k
and R
k+1
, respectively
()
0
xpIn addition the P.D.F. of the initial state , is also given.
Additive Gaussian Noises
( ) ()( ) ()( ) ()[ ] ()( )
kkkkk
T
kxkkkk
T
kkkxkkx
xfxQxfxfxQxfxcxxp
kkk
-Ñ=
þ
ý
ü
î
í
ì
---Ñ=Ñ
+
-
+
-
++ 1
1
1
1
11
2
1
|ln
( ) ( ) ( ) ( )( ) ( )( )
ú
û
ù
ê
ë
é
---===
+++
-
++++
+
+++++ 111
1
1111
1
11111
2
1
exp
2
1
,0;|
kkkk
T
kkk
k
kkkvkk xhzRxhz
R
Rvvpxzp
p
N
( ) ( )( ) ( )( ) ( )[ ] ( )( )
111
1
111111
1
1111211
111
2
1
|ln
+++
-
++++++
-
++++++
-Ñ=
þ
ý
ü
î
í
ì
---Ñ=Ñ
+++ kkkkk
T
kxkkkk
T
kkkxkkx
xhzRxhxhzRxhzcxzp
kkk
( ) ()( ) ()[ ]{ } ()[ ]
11
11
11
|ln
--
++ Ñ=Ñ-Ñ=ÑÑ
++
kk
T
kx
T
k
T
kxk
T
kkkxkk
T
xx QxfxfQxfxxxp
kkkkk
()[ ] ( )[ ]
T
k
T
kxk
T
k
T
kxk xhHxfF
kk
11
1
:
~
&:
~
++
+
Ñ=Ñ=
Cramér-Rao Lower Bound (CRLB)
SOLO
Discrete Time Nonlinear Estimation –Special Cases
( ) () ( ) ()( ) ()( )
ú
û
ù
ê
ë
é
---===
+
-
++ kkkk
T
kkk
k
kkkwkk
xfxQxfx
Q
Qwwpxxp
1
1
11
2
1
exp
2
1
,0;|
p
N
()
( )
p
kkkk
n
kkkk
vxhz
wxfx
R
R
Î+=
Î+=
++++
+
1111
1
1
&
+kk
vw are system and measurement Gaussian white-noise
sequences, independent of past and current states and on each
other with covariances Q
k
and R
k+1
, respectively
()
0
xpIn addition the P.D.F. of the initial state , is also given.
Additive Gaussian Noises
( ) ()( ) ()( ) ()[ ] ()( )
kkkkk
T
kxkkkk
T
kkkxkkx
xfxQxfxfxQxfxcxxp
kkk
-Ñ=
þ
ý
ü
î
í
ì
---Ñ=Ñ
+
-
+
-
++ 1
1
1
1
11
2
1
|ln
( ) ( ) ( ) ( )( ) ( )( )
ú
û
ù
ê
ë
é
---===
+++
-
++++
+
+++++ 111
1
1111
1
11111
2
1
exp
2
1
,0;|
kkkk
T
kkk
k
kkkvkk xhzRxhz
R
Rvvpxzp
p
N
( ) ( )( ) ( )( ) ( )[ ] ( )( )
111
1
111111
1
1111211
111
2
1
|ln
+++
-
++++++
-
++++++
-Ñ=
þ
ý
ü
î
í
ì
---Ñ=Ñ
+++ kkkkk
T
kxkkkk
T
kkkxkkx
xhzRxhxhzRxhzcxzp
kkk
( ) ()( ) ()[ ]{ } ()[ ]
11
11
11
|ln
--
++ Ñ=Ñ-Ñ=ÑÑ
++
kk
T
kx
T
k
T
kxk
T
kkkxkk
T
xx QxfxfQxfxxxp
kkkkk
()[ ] ( )[ ]
T
k
T
kxk
T
k
T
kxk xhHxfF
kk
11
1
:
~
&:
~
++
+
Ñ=Ñ=
40
Cramér-Rao Lower Bound (CRLB)
SOLO
Discrete Time Nonlinear Estimation –Special Cases
()
( )
p
kkkk
n
kkkk
vxhz
wxfx
R
R
Î+=
Î+=
++++
+
1111
1
1
&
+kk
vw are system and measurement Gaussian white-noise
sequences, independent of past and current states and on each
other with covariances Q
k
and R
k+1
, respectively
()
0
xpIn addition the P.D.F. of the initial state , is also given.
Additive Gaussian Noises
( )[ ] ( )( ) ( )( ) ( )[ ]{ } { }
1
1
11|111111111
1
111|
~~
111111 +
-
++++
-
+++++++
-
+++
++++++
=Ñ--Ñ=
kk
T
kxz
T
k
T
kx
T
k
T
kkkkkkkk
T
kxxz
HRHExhRxhzxhzRxhE
kkkkkk
( ){ } (){ } { }
1
|
1
|1|
12 ~
|ln
1111
--
+
++++
-=Ñ-=ÑÑ-=
k
T
kxxkkk
T
xxxkk
T
xxxxk QFEQxfExxpED
kkkkkkkkk
()[ ] ( )[ ]
T
k
T
kxk
T
k
T
kxk xhHxfF
kk
11
1
:
~
&:
~
++
+
Ñ=Ñ=
( ){ } ( ) ( ){ }
()[ ] ()( ) ()( ) ()[ ]{ }
{ }
kk
T
kxx
T
k
T
kx
T
k
T
kkkkkkkk
T
kxxx
kk
T
xkkxxxkk
T
xxxxk
FQFE
xfQxfxxfxQxfE
xxpxxpExxpED
kk
kkkk
kkkkkkkk
~~
|ln|ln|ln:
1
|
?
11
1
|
11|1|
11
1
1
11
-
-
++
-
+++
+
+
++
=
Ñ--Ñ=
ÑÑ=ÑÑ-=
() ( ){ } ( ) ( ){ }
1111|11|
22
|ln|ln|ln:2
11111111
++++++
++++++++
ÑÑ=ÑÑ-=
kk
T
xkkxxzkk
T
xxxzk xzpxzpExzpED
kkkkkkkk
The Jacobians of
computed at , respectively.
() ( )
11
&
++ kkkk
xhxf
1
&
+kk
xx
() ( ){ } ()( )[ ]{ }
1
1
1
|1|
22
11111
|ln:1
-
+
-
+
=-Ñ=ÑÑ-=
+++++ kkkkk
T
xxxkk
T
xxxxk
QxfxQExxpED
kkkkkkk
Cramér-Rao Lower Bound (CRLB)
SOLO
Discrete Time Nonlinear Estimation –Special Cases
()
( )
p
kkkk
n
kkkk
vxhz
wxfx
R
R
Î+=
Î+=
++++
+
1111
1
1
&
+kk
vw are system and measurement Gaussian white-noise
sequences, independent of past and current states and on each
other with covariances Q
k
and R
k+1
, respectively
()
0
xpIn addition the P.D.F. of the initial state , is also given.
Additive Gaussian Noises
( )[ ] ( )( ) ( )( ) ( )[ ]{ } { }
1
1
11|111111111
1
111|
~~
111111 +
-
++++
-
+++++++
-
+++
++++++
=Ñ--Ñ=
kk
T
kxz
T
k
T
kx
T
k
T
kkkkkkkk
T
kxxz
HRHExhRxhzxhzRxhE
kkkkkk
( ){ } (){ } { }
1
|
1
|1|
12 ~
|ln
1111
--
+
++++
-=Ñ-=ÑÑ-=
k
T
kxxkkk
T
xxxkk
T
xxxxk QFEQxfExxpED
kkkkkkkkk
()[ ] ( )[ ]
T
k
T
kxk
T
k
T
kxk xhHxfF
kk
11
1
:
~
&:
~
++
+
Ñ=Ñ=
( ){ } ( ) ( ){ }
()[ ] ()( ) ()( ) ()[ ]{ }
{ }
kk
T
kxx
T
k
T
kx
T
k
T
kkkkkkkk
T
kxxx
kk
T
xkkxxxkk
T
xxxxk
FQFE
xfQxfxxfxQxfE
xxpxxpExxpED
kk
kkkk
kkkkkkkk
~~
|ln|ln|ln:
1
|
?
11
1
|
11|1|
11
1
1
11
-
-
++
-
+++
+
+
++
=
Ñ--Ñ=
ÑÑ=ÑÑ-=
() ( ){ } ( ) ( ){ }
1111|11|
22
|ln|ln|ln:2
11111111
++++++
++++++++
ÑÑ=ÑÑ-=
kk
T
xkkxxzkk
T
xxxzk xzpxzpExzpED
kkkkkkkk
The Jacobians of
computed at , respectively.
() ( )
11
&
++ kkkk
xhxf
1
&
+kk
xx
() ( ){ } ()( )[ ]{ }
1
1
1
|1|
22
11111
|ln:1
-
+
-
+
=-Ñ=ÑÑ-=
+++++ kkkkk
T
xxxkk
T
xxxxk
QxfxQExxpED
kkkkkkk
41
Cramér-Rao Lower Bound (CRLB)
{ }
{ }
()
() { }
1
1
11|
22
122
1
|
12
1
|
11
~~
2
1
~
~~
11
1
1
+
-
++
-
-
-
++
+
+
=
=
-=
=
kk
T
kxzk
kk
k
T
kxxk
kk
T
kxxk
HRHED
QD
QFED
FQFED
kk
kk
kk
SOLO
Discrete Time Nonlinear Estimation –Special Cases
()
( )
p
kkkk
n
kkkk
vxhz
wxfx
R
R
Î+=
Î+=
++++
+
1111
1
1
&
+kk
vw are system and measurement Gaussian white-noise
sequences, independent of past and current states and on each
other with covariances Q
k
and R
k+1
, respectively
()
0
xpIn addition the P.D.F. of the initial state , is also given.
Additive Gaussian Noises
()[ ] ( )[ ]
T
k
T
kxk
T
k
T
kxk xhHxfF
kk
11
1
:
~
&:
~
++
+
Ñ=Ñ= The Jacobians of
computed at , respectively.
() ( )
11
&
++ kkkk
xhxf
1
&
+kk
xx
() [ ] ()
tMeasuremen
Updated
22
ModelProcess
UsingPrediction
12
1
112122
1
21:
kkkkkkk
DDDJDDJ ++-=
-
+
We can calculate the expectations using a Monte Carlo
Simulation. Using we draw ()( ) ()
01
&, xpvpwp
kk +
()
() ( ) Nivpvwpw
xpx
k
i
kk
i
k ,,2,1~&~
~
11
00
=
++
We Simulate System States and Measurements
()
( )
Ni
vxhz
wxfx
i
k
i
kk
i
k
i
k
i
kk
i
k
,,2,1
1111
1
=
ï
î
ï
í
ì
+=
+=
++++
+
We then average over realizations to get J
0
.
We average over realization to get next terms and so forth.
0
x
1
x
Return to Table of Content
Cramér-Rao Lower Bound (CRLB)
{ }
{ }
()
() { }
1
1
11|
22
122
1
|
12
1
|
11
~~
2
1
~
~~
11
1
1
+
-
++
-
-
-
++
+
+
=
=
-=
=
kk
T
kxzk
kk
k
T
kxxk
kk
T
kxxk
HRHED
QD
QFED
FQFED
kk
kk
kk
SOLO
Discrete Time Nonlinear Estimation –Special Cases
()
( )
p
kkkk
n
kkkk
vxhz
wxfx
R
R
Î+=
Î+=
++++
+
1111
1
1
&
+kk
vw are system and measurement Gaussian white-noise
sequences, independent of past and current states and on each
other with covariances Q
k
and R
k+1
, respectively
()
0
xpIn addition the P.D.F. of the initial state , is also given.
Additive Gaussian Noises
()[ ] ( )[ ]
T
k
T
kxk
T
k
T
kxk xhHxfF
kk
11
1
:
~
&:
~
++
+
Ñ=Ñ= The Jacobians of
computed at , respectively.
() ( )
11
&
++ kkkk
xhxf
1
&
+kk
xx
() [ ] ()
tMeasuremen
Updated
22
ModelProcess
UsingPrediction
12
1
112122
1
21:
kkkkkkk
DDDJDDJ ++-=
-
+
We can calculate the expectations using a Monte Carlo
Simulation. Using we draw ()( ) ()
01
&, xpvpwp
kk +
()
() ( ) Nivpvwpw
xpx
k
i
kk
i
k ,,2,1~&~
~
11
00
=
++
We Simulate System States and Measurements
()
( )
Ni
vxhz
wxfx
i
k
i
kk
i
k
i
k
i
kk
i
k
,,2,1
1111
1
=
ï
î
ï
í
ì
+=
+=
++++
+
We then average over realizations to get J
0
.
We average over realization to get next terms and so forth.
0
x
1
x
Return to Table of Content
42
Cramér-Rao Lower Bound (CRLB)
() ()
1
1
11
22122112111
2&1&&
+
-
++
---
==-==
kk
T
kkkkk
T
kkkk
T
kk HRHDQDQFDFQFD
SOLO
Discrete Time Nonlinear Estimation –Special Cases
p
kkkk
n
kkkk
vxHz
wxFx
R
R
Î+=
Î+=
++++
+
1111
1
1
&
+kk
vw are system and measurement Gaussian white-noise
sequences, independent of past and current states and on each
other with covariances Q
k
and R
k+1
, respectively
()
0
xpIn addition the P.D.F. of the initial state , is also given.
Linear/ Gaussian System
( ) ( )
1
1
11
1
1
tsMeasuremen
Updated
1
1
11
ModelProcess
UsingPrediction
1
1
111
1 +
-
++
-
-
+
-
++
-
-
---
+
++=++-=
kk
T
k
T
kkkk
Lemma
Inverse
Matrix
kk
T
kk
T
kkk
T
kkkkkk
HRHFJFQHRHQFFQFJFQQJ
Define ( )
T
kkkkkkkkkkkkk FPFQPPJPJ
|
1
|1
1
|
1
1|11 :&:&: +===
-
+
--
+++
( )
1
1
11
1
|11
1
11
1
|
1
1|1 +
-
++
-
++
-
++
-
-
++
+=++=
kk
T
kkkkk
T
k
T
kkkkkkk
HRHPHRHFPFQP
The conclusion is that CRLB for the Linear Gaussian Filtering Problem is
Equivalent to the Covariance Matrix of the Kalman Filter.
Return to Table of Content
Cramér-Rao Lower Bound (CRLB)
() ()
1
1
11
22122112111
2&1&&
+
-
++
---
==-==
kk
T
kkkkk
T
kkkk
T
kk HRHDQDQFDFQFD
SOLO
Discrete Time Nonlinear Estimation –Special Cases
p
kkkk
n
kkkk
vxHz
wxFx
R
R
Î+=
Î+=
++++
+
1111
1
1
&
+kk
vw are system and measurement Gaussian white-noise
sequences, independent of past and current states and on each
other with covariances Q
k
and R
k+1
, respectively
()
0
xpIn addition the P.D.F. of the initial state , is also given.
Linear/ Gaussian System
( ) ( )
1
1
11
1
1
tsMeasuremen
Updated
1
1
11
ModelProcess
UsingPrediction
1
1
111
1 +
-
++
-
-
+
-
++
-
-
---
+
++=++-=
kk
T
k
T
kkkk
Lemma
Inverse
Matrix
kk
T
kk
T
kkk
T
kkkkkk
HRHFJFQHRHQFFQFJFQQJ
Define ( )
T
kkkkkkkkkkkkk FPFQPPJPJ
|
1
|1
1
|
1
1|11 :&:&: +===
-
+
--
+++
( )
1
1
11
1
|11
1
11
1
|
1
1|1 +
-
++
-
++
-
++
-
-
++
+=++=
kk
T
kkkkk
T
k
T
kkkkkkk
HRHPHRHFPFQP
The conclusion is that CRLB for the Linear Gaussian Filtering Problem is
Equivalent to the Covariance Matrix of the Kalman Filter.
Return to Table of Content
43
Cramér-Rao Lower Bound (CRLB)
SOLO
Discrete Time Nonlinear Estimation –Special Cases
p
kkkk
n
kkk
vxHz
xFx
R
R
Î+=
Î=
++++
+
1111
1
1+k
v
are measurement Gaussian white-noise sequence,
independent of past and current states with covariance R
k+1
.
Q
k
= 0.
()
0
xpIn addition the P.D.F. of the initial state , is also given.
Linear System with Zero System Noise
Define ( )
1
|
0
1
|1
1
|
1
1|11
:&:&:
-
=
-
+
--
+++
===
T
kkkk
Q
kkkkkkkk
FPFPPJPJ
k
( )
1
1
11
1
|11
1
11
1
|
1
1|1 +
-
++
-
++
-
++
-
-
++
+=+=
kk
T
kkkkk
T
k
T
kkkkkk
HRHPHRHFPFP
Return to Table of Content
Cramér-Rao Lower Bound (CRLB)
SOLO
Discrete Time Nonlinear Estimation –Special Cases
p
kkkk
n
kkk
vxHz
xFx
R
R
Î+=
Î=
++++
+
1111
1
1+k
v
are measurement Gaussian white-noise sequence,
independent of past and current states with covariance R
k+1
.
Q
k
= 0.
()
0
xpIn addition the P.D.F. of the initial state , is also given.
Linear System with Zero System Noise
Define ( )
1
|
0
1
|1
1
|
1
1|11
:&:&:
-
=
-
+
--
+++
===
T
kkkk
Q
kkkkkkkk
FPFPPJPJ
k
( )
1
1
11
1
|11
1
11
1
|
1
1|1 +
-
++
-
++
-
++
-
-
++
+=+=
kk
T
kkkkk
T
k
T
kkkkkk
HRHPHRHFPFP
Return to Table of Content
44
Cramér-Rao Lower Bound (CRLB)
SOLO
References
http://en.wikipedia.org/wiki/Cramer_Rao_bound
Bergman, N., “Recursive Bayesian Estimation - Navigation and Tracking Applications”,
PhD Thesis, Linköping University, 1999, Dissertation No. 579, Ch. 4
Van Trees, H., L., “Detection, Estimation and Modulation Theory”, Wiley,
New York, 1968, 2001, pp. 146, 66, 72, 79,84,
Tichavský, P., Muravchik, C, Nehorai. A., “Posterior Cramér – Rao bounds for
Discrete-Time Nonlinear Filtering”, IEEE Transactions on Signal Processing, 46(5),
1998, pp. 1386 - 1396
Ristic, B., Arulampalam, S., N., Gordon, N., “Beyond the Kalman Filter – Particle Filters
for Tracking Applications”, Artech House, 2004, Ch. 4: “Cramér – Rao Bounds for
Nonlinear Filtering”
Ristic, B., “Cramér – Rao Bounds for Target Tracking”, Int. Conf. Intelligent Sensors,
Sensor Networks and Information Processing, 6 Dec., 2005,
http://www.issnip.org/2005/branko_05.pdf
Van Trees, H., L., “Bayesian Bounds”, Keynote Speech, 2005 Adaptive Sensor and Array
Processing Workshop, 7 June 2005,
http://www.ll.mit.edu/asap/asap_05/pdf/Presentations/01_vantrees.pdf
Return to Table of Content
Cramér-Rao Lower Bound (CRLB)
SOLO
References
http://en.wikipedia.org/wiki/Cramer_Rao_bound
Bergman, N., “Recursive Bayesian Estimation - Navigation and Tracking Applications”,
PhD Thesis, Linköping University, 1999, Dissertation No. 579, Ch. 4
Van Trees, H., L., “Detection, Estimation and Modulation Theory”, Wiley,
New York, 1968, 2001, pp. 146, 66, 72, 79,84,
Tichavský, P., Muravchik, C, Nehorai. A., “Posterior Cramér – Rao bounds for
Discrete-Time Nonlinear Filtering”, IEEE Transactions on Signal Processing, 46(5),
1998, pp. 1386 - 1396
Ristic, B., Arulampalam, S., N., Gordon, N., “Beyond the Kalman Filter – Particle Filters
for Tracking Applications”, Artech House, 2004, Ch. 4: “Cramér – Rao Bounds for
Nonlinear Filtering”
Ristic, B., “Cramér – Rao Bounds for Target Tracking”, Int. Conf. Intelligent Sensors,
Sensor Networks and Information Processing, 6 Dec., 2005,
http://www.issnip.org/2005/branko_05.pdf
Van Trees, H., L., “Bayesian Bounds”, Keynote Speech, 2005 Adaptive Sensor and Array
Processing Workshop, 7 June 2005,
http://www.ll.mit.edu/asap/asap_05/pdf/Presentations/01_vantrees.pdf
Return to Table of Content
January 11, 2015 45
SOLO
Technion
Israeli Institute of Technology
1964 – 1968 BSc EE
1968 – 1971 MSc EE
Israeli Air Force
1970 – 1974
RAFAEL
Israeli Armament Development Authority
1974 – 2013
Stanford University
1983 – 1986 PhD AA
SOLO
Technion
Israeli Institute of Technology
1964 – 1968 BSc EE
1968 – 1971 MSc EE
Israeli Air Force
1970 – 1974
RAFAEL
Israeli Armament Development Authority
1974 – 2013
Stanford University
1983 – 1986 PhD AA
46
Harry L. Van Trees
http://teal.gmu.edu/faculty_info/van.html
Harald Cramér
1893 – 1985
Cayampudi Radhakrishna
Rao
1920 -
Fisher, Sir Ronald Aylmer
1890 - 1962
Branko Ristic
Niclas Bergman
Arye Nehorai
Carlos H. Muravchik
Petr Tichavsky
Harry L. Van Trees
http://teal.gmu.edu/faculty_info/van.html
Harald Cramér
1893 – 1985
Cayampudi Radhakrishna
Rao
1920 -
Fisher, Sir Ronald Aylmer
1890 - 1962
Branko Ristic
Niclas Bergman
Arye Nehorai
Carlos H. Muravchik
Petr Tichavsky
Categories
GeneralDownload
Download Slideshow Get the original presentation fileQuick Actions
Statistics
- Views 3,418
- Slides 46
- Favorites 3
- Age 3979 days
Related Slideshows
22
Pray For The Peace Of Jerusalem and You Will Prosper
RodolfoMoralesMarcuc
32 views
26
Don_t_Waste_Your_Life_God.....powerpoint
chalobrido8
34 views
31
VILLASUR_FACTORS_TO_CONSIDER_IN_PLATING_SALAD_10-13.pdf
JaiJai148317
31 views
14
Fertility awareness methods for women in the society
Isaiah47
30 views
35
Chapter 5 Arithmetic Functions Computer Organisation and Architecture
RitikSharma297999
28 views
5
syakira bhasa inggris (1) (1).pptx.......
ourcommunity56
30 views