Optimisation and optimal control 2: Tutorial solutions (optimal control)

Question 1
First, we form the Hamiltonean for the LQR problem:
1
H(x, u) = (xT Qx + uT Ru) + pT (Ax + Bu)
2
To obtain the minimising solution we need to solve:
H
= 0 Ru + B T p = 0 u = R1 B T p
u
H
p = = Qx AT p
x
M (T )
p(T ) = = Sx(T )
x
together with the dynamic equations. In matrix form:
! ! !
x A BR1 B T x
= T
p Q A p

which is a two-point boundary-value problem with boundary conditions x(0) = x0 and

p(T ) = Sx(T ).
Next, use suggested substitution p(t) = P (t)x(t) for some unknown matrix function P (t), which
satisfies P (T ) = S. Differentiating the co-state equations we get:

p = P x + P x = P x + P (Ax + Bu) = P x + P (Ax BR1 B T p)

= P x + P Ax P BR1 B T p = P x + P Ax P BR1 B T P x

Using the co-state equation p = Qx AT p = Qx AT P x gives

P x = (AT P + P A P BR1 B T P + Q)x

for t [0 T ]. Since this must hold for all state-trajectories given any x(0) = x0 , it is necessary
that:
P = AT P + P A P BR1 B T P + Q

for t [0 T ] which is a differential matrix Riccati equation.

Question 2
Obtain a spectral decomposition of the Hamiltonean:
! !
P1 P1
H =
P2 P2

1
where [P1T P2T ]T is a basis of the stable invariant subspace of H (e.g. the columns of [P1T P2T ]T
are the eigenvectors and generalised eigenvectors of H and is the part of the Jordan block
of H corresponding to all stable eigenvalues). Then P = P2 (P1 )1 . This may fail if we cannot
choose an n-dimensional (where n is the dimension of A) stable invariant subspace for H (i.e.
if H has eigenvalues on the imaginary axis) or if P1 is singular. Both conditions are satisfied if
(A, B) stabilisable and (A, C) detectable.

Question 3
(i) Using the suggested substitution,
u(t) 1 u(t)u(t) u2 (t) 1 u(t) 1 u2 (t)
p(t) = p(t) = = +
au(t) a u2 (t) a u(t) a u2 (t)
and
u2 (t) b u(t)
ap2 (t) + bp(t) + c = a 2 2
+c
a u (t) a u(t)
Hence
u2 (t) bu(t) u(t) u2 (t)
+ c = +
au2 (t) au(t) au(t) au2 (t)
Cancelling and rearranging
u(t) bu(t)
+ c = 0 u(t) bu(t) + cau(t) = 0
au(t) au(t)
which is a linear second-order differential equation.
(ii) Again applying suggested substitution gives:
u(t) u(t)u(t) u2 (t) u(t) u(t) u2 (t)
p(t) = (t) (t) = (t) (t) + (t)
u(t) u2 (t) u(t) u(t) u2 (t)
Hence
u(t) u(t) u2 (t) u2 (t)
(t)((t) + p(t)) = p2 (t) (t)((t) (t) (t) + (t) 2 (t)) = 2 (t) 2
u(t) u(t) u u (t)
Simplifying,
u(t) u(t) d
(t) + (t) (t) = 0 (t)u(t) + (t)u(t) (t)u(t) = 0 ((t)u(t)) = (t)u(t)
u(t) u(t) dt
as required.

Question 4
Here A = 0, B = 1, R = 1 and Q = 0, P (1) = S = 1. The ARE is
Z 1 Z 1 1
2 2 dp 1
p(t) = p (t) p(t) = p (t) 2
= dt =1t
t p t p t
1 1 1
= 1 t p(t) =
p(t) p(1) 2t

2
Thus
1
u(t) = R1 B T P (t)x(t) = x(t)
t2
Hence
Z x(t) Z t
1 dx dt x(t) 2t
x(t) = x(t) = [ln(x)]x(t) t
x0 = [ln(2 t)]0 ln = ln
t2 x0 x 0 2t x0 2

and hence
t
x(t) = 1 x0
2
and therefore
1 t 1
u(t) = 1 x0 = x0
t2 2 2
(constant).

Question 5
This is an LQR finite horizon problem with A = 0, B = R = S = I2 , and Q = 0. The
differential Riccati equation thus gives:

P (t) = P 2 (t), P (3) = I2

In matrix form this gives

! ! ! !
p1 p2 p1 p2 p1 p2 p21 + p2 p3 p1 p2 + p2 p3
= =
p2 p3 p2 p3 p2 p3 p1 p2 + p2 p3 p22 + p23

or in full form

p1 = p21 + p2 p3 , p1 (3) = 1
p2 = p2 (p1 + p3 ), p2 (3) = 0
p3 = p22 + p23 , p3 (3) = 1

The second equation implies that p2 (t) = 0 for all t. Then the first and third equations give
pi = p2i , i = 1, 3. Thus
Z 3 Z 3 3
dpi 1 1
= dt = 3 t pi (t) =
t p2i t pi t 4t

Thus !
1
4t
0
P (t) = 1
0 4t
1
and hence ui (t) = x (t),
t4 i
i = 1, 2. Integrating gives:
Z xi (t) Z t
1 dt dxi dt
xi (t) = xi (t) = =
4t 4t xi0 x(t) 0 4t

3
Thus
x (t) xi (t) 4t t
[ln xi ]xii = [ln(4 t)]t0 ln i = ln i
xi (t) = x0 1
0 x0 4 4
and hence
1 i t xi
ui (t) = x0 1 = 0
t4 4 4
for i = 1, 2 (constant).

Question 6
The ARE in this case gives:

2ap p2 + m = 0 p2 2ap m = 0 p = a a2 + m

Since (a, 1) is controllable and (a, m) is observable for m > 0 the stabilising solution must

be positive, and hence p = a + a2 + m > 0. The A matrix of the closed-loop system is

ac = ap = a(a+ a2 + m) = a2 + m < 0. Hence x(t) = exp( a2 + mt) 0 at t

and u(t) = (a + a2 + m) exp( a2 + mt), so that maxt0 |u(t)| = |u(0)| = a + a2 + m.
Note that as m increases the state transient response becomes faster and maxt0 |u(t)| increases

(as m asymptotically).

Question 7
The state-space model is
! ! ! !
x1 0 1 x1 0
= + u
x2 1 0 x2 1

The ARE is
AT P + P A P BR1 B T P + Q = 0
or
! ! ! ! ! !
0 1 p1 p2 p1 p2 0 1 p2 1 0
+ 8 p2 p3 +
1 0 p2 p3 p2 p3 1 0 p3 0 0

which gives:

2p2 8p22 + 1 = 0
p3 + p1 8p2 p3 = 0
2p2 8p23 = 0

The first equation gives

2
4 + 32 1 1
p2 = = or
16 4 2
2
while the third equation gives 4p3 = p2 . Choosing the negative solution for p2 thus gives p3 as
imaginary, so p2 = 14 and hence p23 = 16
1
, so that p3 = 14 (choosing p3 = 14 would correspond

4
to a solution P which is not positive semi-definite, so it is disregarded). The second equation
now gives
3
p1 = p3 + 8p2 p3 =
4
and !
3 1
4 4
P = 1 1
4 4

The optimal control is

1 T 1 1
u(t) = R B P x(t) = 8 4 4
x(t) = 2x1 (t) 2x2 (t)

Question 8
Here, ! ! !
0 1 C 1 0
B AB = and =
1 0 CA 0 1
and hence (A, B) is controllable and (A, C) is observable. Hence we expect a unique symmetric
(real) positive definite stabilising solution. The Hamiltonean matrix is:

0 1 0 0

0 0 0 1
H= 1 0 0

0

0 0 1 0

The characteristic equation of H is det(I4 H) = 0 or 4 + 1 = 0 and hence the eigenvalues

are 1 = exp(j/4), 2 = exp(j/4) (antistable) and 3 = exp(3j/4), 4 = exp(3j/4)
(stable) are the eigenvalues of H. If a = [x y z w]T is an eigenvector then Ha = a or y = x,
w = y, x = z and z = w. Thus [1 2 3 ]T is an eigenvector for each = i since
all four equations are satisfied (the first, second and fourth by construction and the third since
x = z = 4 x which implies that (1 + 4 )x = 0). Thus the stable spectral subspace is the
range of the matrix

1 1 1 1

3 4 exp(3j/4) exp(3j/4)

P = 3 =
3
3 4 exp(j/4) exp(j/4)
2 2
3 4 exp(j/2) exp(j/2)

Thus P = P2 P11 where

! !
1 1 33 34
P1 = and P2 =
3 4 23 24

Now !T !
1 4 3 1 4 1
P11 = =
4 3 1 1 4 3 3 1

5
and
! !
1 33 34 4 1
P = P2 P11 =
4 3 23 24 3 1
!
1 3 4 (3 + 4 )(3 4 ) (4 3 )(24 + 3 4 + 23
=
4 3 3 4 (4 3 ) (3 + 4 )(3 4
! !
3 4 (3 + 4 ) 23 + 3 4 + 24 (3 + 4 ) 1
= =
3 4 (3 + 4 ) 1 (3 + 4 )
or !

2 1
P =
1 2

since 23 = j, 24 = j and 3 +4 = 2Re(3 ) = 2. Clearly P is positive definite (eigenvalues

at i = 2 1 > 0) as expected since (A, C) is observable. The closed-loop A-matrix is
! ! ! !
0 1 0 2 1 0 1
Ac = A BB T P = 0 1 =
0 0 1 1 2 1 2

This has characteristic equation det(I Ac ) = 0, or 2 + 2 + 1 = 0 and hence closed-loop

eigenvalues are i = 22 j 22 as expected (stable eigenvalues of H).

Question 9
This is a standard LQR problem with R = 1, Q = diag(1, 1). Let the ARE solution be
!
p1 p2
P =
p2 p3

Then the Riccati equation gives:

! ! ! ! ! !
0 0 p1 p2 p1 p2 0 1 p2 1 0
+ p2 p3 + =0
1 0 p2 p3 p2 p3 0 0 p3 0 1
or ! ! ! !
0 0 0 p1 p22 p2 p3 1 0
+ + =0
p1 p2 0 p2 p2 p3 p23 0 1
The (1, 1) element gives p22 = 1 or p2 = 1. The (1, 2) element then gives p1 = p2 p3 , i.e.
p1 = p3 (if p2 = 1) or p1 = p3 (if p2 = 1). Clearly p1 = p3 does not correspond to a
positive semi-definite solution, so we must have p2 = 1 and p1 = p3 . The (2, 2) entry now gives

p23 = 2p2 + 1 = 3 and hence p3 = 3 (positive solution should be chosen). Thus
!
3 1
P =
1 3

6
is the stabilising solution and hence
! !
3 1 x1
u = R1 B T P x = 0 1 = x1 3x2
1 3 x2

The closed-loop A-matrix is then:

!
0 1
Acl = A BR1 B T P =
1 3

This has a characteristic equation det(I Acl ) = 2 + 3 + 1 = 0. Hence the closed-loop
poles are
1 1
1,2 = 3j
2 2
and hence Acl is asymptotically stable.

Question 10
The steady state Kalman filter equation gives:

2Pf Pf2 + 1 = 0 Pf = 21

(selecting positive solution of quadratic). The optimal filter gain is then:

L = Pf C T W 1 = 21

so that

A LC = 1 ( 2 1) = 2

(stable). The state-estimates are:

x = x + u + L(x + w x) = 2x + ( 2 1)x + u + ( 2 1)w

Hence the estimate error dynamics are:

e = x x = x + u + v 2x + ( 2 1)x + u + ( 2 1)w

= 2(x x) + v + (1 2)w

= 2e + v + (1 2)w

In the absence of process and measurement noise,

e(t) = 2e(t) e(t) = e(0) exp( 2t) 0

for any e(0).