MATLAB 6

Least Squares Approximation - Signal Processing(DFT)

Definition: Let the m x n matrix A have rank n. Then by the least squares solution of the system
Ax = b is meant the solution x of the corresponding normal system AT Ay AT b .

Note:

The normal system has a unique solution

x ( AT A) 1 AT b ;

The orthogonal projection p of b into Col(A) is given by p A x .

If the column vectors of A form an orthonormal set of vectors, then AT A I and the
solution to the least squares problem is x AT b
*********************************************************************
Let S be a subspace of an inner product space V and let x V . Let {v 1 , v 2 ,..., v n } be an
n
orthonormal basis for S. If p ci v i where ci x, v i , then p x S
i 1

p is the element of S that is closest to x and is said to be the projection of x onto S.

Let S be a nonempty subspace of R m and b R m . If {v 1 , v 2 ,..., v k } is an orthonormal basis
for S and V [ v 1 v 2 ... v k ] , then the projection p of b onto S is given by p VV T b
The matrix VV T is called the projection matrix corresponding to the subspace S of R m .
The projection matrix is unique (Prove)

Example 1: Let S {[ x, y,0]T | x, y are real} . Let w [5,3,4]T . If v 1 [1,0,0]T & v 2 [0,1,0]T ,
then p VV T w [5,3,0]T .

Example 2: Find the least squares approximation to e x on the interval [0,1] by a linear function.

Example 3:
The trigonometric polynomial of degree n is the function of the form
n
a0
t n ( x) (a k cos kx bk sin kx)
2 k1
1
{ , cos x, cos 2 x, ..., cos nx, sin x, sin 2 x, ..., sin nx} forms an orthonormal set with respect
2
1
to the inner product f , g f ( x) g ( x)dx .

The best least squares fit approximation to a continuous 2 periodic function f (x) by a
trigonometric polynomial of degree n or less has Fourier Coefficients given by:
 a0 f ,1 , a k f , cos kx , & bk f , sin kx .
1 1
Let ck (a k ibk ) and c k (a k ibk ) , k 0 . Using these, it can be shown that
2 2
n
t n ( x) c k e ikx
k n

where
1
c f ( x)e ikx dx
k 2
The Discrete Fourier Series (Page 267)

In what follows we assume that f (x) is a noisy signal (see figure 5.5.5 a in text).
We will choose [0,2 ] instead of [ , ] .
n
We want to approximate f (x) by t n ( x) c k e ikx
k n

The trigonometric approximation represents the function by simple harmonics where the
the k-th harmonic is rk cos(kx k ) , where k is the angular frequency.
A signal is smooth if the coefficients c k 0 rapidly as k
If some of the coefficients corresponding to larger frequencies are not small, the graph is
noisy.
We can filter the signal by setting these coefficients to zero. (see figure 5.5.5 b)
In actual signal processing we do not have a mathematical formula for f (x) . Rather, the
2j
signal is sampled over a sequence of times x j and the function is represented by
N
the N sample values y 0 f ( x0 ), y1 f ( x1 ),..., y N 1 f ( x N 1 ). [ y 0 f (0) & y N f (2 )]
In this case, we cannot compute c k . Instead we use a numerical integration method, the
trapezoidal rule, to approximate the integral. The approximations to the Fourier
coefficients are given by
1 N 1 ikx
j
d f ( x )e
k N j
j 0
2 i
2 2 1 N 1 jk
If e N cos( ) i sin( ) we can write d y
N N N k N j N
j 0
The finite sequence { d } is said to be the discrete Fourier transform of { y } .
k k
The Fourier Transform can be determined by a single matrix vector multiplication. For
example, if N = 4,
1
d0 ( y 0 y1 y 2 y3 )
4
 1 2 3
d1 ( y0 4 y1 4 y2 4 y3 )
4
1 2 4 6
d2 ( y0 4 y1 4 y2 4 y3 )
4
1 3 6 9
d3 ( y0 4 y1 4 y2 4 y3 )
4

1
If z = (1/4)y = [ y 0 y1 y 2 y3 ]T , then the vector d = [d 0 d1 d 2 d 3 ]T is determined by
4
multiplying z by the matrix
1 1 1 1 1 1 1 1
2 3
1 4 4 4 1 i 1 i
F4 2 4 6
called the Fourier Matrix
1 4 4 4
1 1 1 1
1
3 6 9 1 i 1 i
4 4 4
( j 1)( k 1)
FN is the N N matrix whose ( j, k ) element is f j ,k N .
DFT algorithm is the method of computing the discrete Fourier transform.

Exercises.
1. Derive the equation (5) on page 268.
2. Explain how the powers of 4 can be derived on the unit circle in the complex plane.
3. Determine all the powers of 8 by hand (using the unit circle) and write an exact form for
F8 (no decimal approximations)
4. Use MATLAB to compute the DFT { d 0 , d1 , d 2 , d 3 , d 4 , d 5 , d 6 , d 7 } for the following
data and plot the data
y 0 0.324
y1 0.456
y2 0.112
y3 0.202
y4 0.654
y 5 0.234
y 6 0.678
y 7 0.867
5. Plot the trigonometric polynomial t n (x) .
6. Compare the Exercise 8 with the built in MATLAB FFT (do an help fft)