Orthogonal Sets

A set of vectors{u1,u2,…,up} in Rⁿ is called an orthogonal set if u i ⋅ u j = 0

whenever i≠j.

Example.
1   1 0 
 
Is 1 , − 1, 0  an orthogonal set?
   
0  0 1  
      
Solution: Label the vectors u1, u2, and u3 respectively. Then
u1 ⋅ u 2 = 1(1) + 1(− 1) + 0(0) = 0
u1 ⋅ u 3 = 1(0) + 1(0) + 0(1) = 0
u 2 ⋅ u 3 = 1(0) + (− 1)(0 ) + 0(1) = 0
Since u1 ⋅ u 2 = u1 ⋅ u 3 = u 2 ⋅ u 3 = 0 , then {u1, u2, u3} is an orthogonal set.
1
Theorem 4
Suppose S={u1,u2,…,up} is an orthogonal set of nonzero vectors in Rⁿ and
W=span{u1,u2,…,up}
Then S is a linearly independent set and is therefore a basis for W.

Partial Proof: Suppose c1u1 +c2u2 +…+ cpup= 0

(c1u1 +c2u2 +…+ cpup)· u1 =0·
c1u1·u1+c2u2·u1+…+ cpup·u1=0
So c1u1·u1= 0 and since u1≠ 0, u1·u1>0 which means c1=0.
In a similar manner, c2,…,cp can be shown to by all 0. So S is a linearly
independent set. ■

Definition:
An orthogonal basis for a subspace W of Rⁿ is a basis for W that is also an
orthogonal set.
2
Orthonormal Sets

A set of vectors {u1,u2,…,up} in Rⁿ is called an orthonormal set if it is an

orthogonal set of unit vectors.

If W=span{u1,u2,…,up} then {u1,u2,…,up} is an orthonormal basis for W.

Recall that v is a unit vector if

2
v = v ⋅ v = vT v = 1

Therefore if {u1,u2,…,up} is an orthonormal set, then

2 T
uj = u j ⋅u j = u j u j =1
⇒ u j ⋅u j =1

3
Suppose U=[ u1 u2 u3] where {u1,u2,u3} is an orthonormal set.

u1T  u1Tu1 u1Tu 2 u1Tu 3  1 0 0

 T  
Then U U = u 2 [u1 u 2 u 3 ] = u T2 u1
T
u T2 u 2 u T2 u 3  = 0 1 0
 
u T  u T
u1 u 3Tu 2 u 3 u 3  0 0 1
T
 3  3

Theorem 6
An m×n matrix U has orthonormal columns if and only if UTU=I.

Theorem 7
Let U be an m×n matrix with orthonormal columns, and let x and y be in Rⁿ.
Then
a. Ux = x
b. (Ux)·(Uy)=x··y
c. (Ux)·(Uy)=0 if and only if x··y=0.
4
Using orthogonal basis to compute weights in linear combination
Theorem 5
Let {u1,u2,…,up} be an orthogonal basis for a subspace W of Rⁿ. Then each y in
W has a unique representation as a linear combination of u1,u2,…and up given by
y ⋅u j
y = c1u1 + c2u 2 + ... + c p u p where cj = ,… (j = 1,…,p)
u j ⋅u j
Proof: y ⋅ u i = (c1u1 + c2u 2 + ... + c p u p ) ⋅ u i
For i=1: y ⋅ u1 = (c1u1 + c2u 2 + ... + c p u p ) ⋅ u1
y ⋅ u1 = c1u1 ⋅ u1 + c2u 2 ⋅ u1 + ... + c p u p ⋅ u1
y ⋅ u1
⇒ y ⋅ u1 = c1u1 ⋅ u1 since u i ⋅ u j for i ≠ j ⇒ c1 =
u1 ⋅ u1
Similarly, c2 = (y ⋅ u 2 ) / (u 2 ⋅ u 2 ), ... , c p = (y ⋅ u p )/ (u p ⋅ u p )

If the basis is orthonormal then cj = y ⋅u j since u j ⋅ u j = 1.

Example Set 1 – Week 9
5
Orthogonal Projection of y onto u

For a nonzero vector u in Rⁿ, suppose we want to write y in Rⁿ as the following

y=(multiple of u)+(multiple a vector ⊥ to u)

y = yˆ + z
yˆ = αu ⇒ z = y − yˆ Since z ⊥ ŷ , therefore z ⊥ u
z ⋅ u = (y − αu ) ⋅ u = y ⋅ u − α u ⋅ u = 0
y ⋅u
⇒α =
u⋅u
y • orthogonal projection of y onto u
y-αu y ⋅u
yˆ = αu yˆ = αu = u = proju y
u ⋅u
u • component of y orthogonal to u
y ⋅u
z = y − yˆ = y − αu = y − u
u⋅u
6
Orthogonal Projection of y onto subspace L

L is subspace spanned by u: L = span{u}

ie. the set of vectors that can be written as multiple of u ie. αu

Projection of y onto L is given by

y ⋅ (cu ) y ⋅u L
yˆ = projL y = (cu ) = u
(cu ) ⋅ (cu ) u⋅u
y
y-αu
If the vector y is in subspace L then
ŷ
yˆ = y

Example Set 2 – Week 9

7
Orthogonal Projection onto subspace W

We have already seen that if {u1,u2,…,up} is an orthogonal basis for W then each
vector y in W can be written as
y ⋅ u1 y ⋅ u2 y ⋅u p
y= u1 + u 2 + ... + up
u1 ⋅ u1 u2 ⋅ u2 up ⋅up
This is the projection of y onto W.

What if y is not in the subspace W? y

We can write y = yˆ + z z W

where y is a vector in W and z is a vector in W ⊥ ŷ

0
y ⋅ u1 y ⋅ u2 y ⋅u p
yˆ = projW y = u1 + u 2 + ... + up
u1 ⋅ u1 u2 ⋅ u2 u p ⋅u p
z = y − yˆ
8
The Orthogonal Decomposition Theorem
Let W be a subspace of Rⁿ. Then each y in Rⁿ can be uniquely represented in the
form
y = yˆ + z
where ŷ is in W and z is in W ⊥ . In fact, if {u1,u2,…,up} is any orthogonal basis
of W, then
y ⋅ u1 y ⋅ u2 y ⋅u p
yˆ = u1 + u 2 + ... + up
u1 ⋅ u1 u2 ⋅ u2 up ⋅up
and
z = y − yˆ . y

z W
The vector ŷ is called the orthogonal
projection of y onto W.
ŷ
0

9
Geometric Interpretation of Orthogonal Projections

û 2

ŷ 2
y ⋅ u1 y ⋅ u2
yˆ = yˆ 1 + yˆ 2 = u1 + u2
u1 ⋅ u1 u2 ⋅ u2

ŷ1

û1

10
Theorem 10

If {u1,u2,…,up} is an orthonormal basis for a subspace W of Rⁿ, then

projWy=(y·u1) u1 +(y·u2)u2 +…+ (y·up)up

If U=[u1 u2 … up], then projWy =UUTy for all y in Rn.

Example Set 3 – Week 9

11
The Best Approximation Theorem
Let W be a subspace of Rn, y any vector in Rn, and ŷ the orthogonal projection
of y onto W. Then ŷ is the point in W closest to y, in the sense that
y − yˆ < y − v
for all v in W distinct from y.

z W

`
y =proj Wy
0

12
Outline of Proof:

Let v in W be distinct from y. Then v − yˆ is in W.

The vector z = y − yˆ is orthogonal to W and so y − yˆ is orthogonal to v − yˆ .

Since y−v= y − yˆ + yˆ − v we also have

2 2 2
y − v = y − yˆ + yˆ − v
Hence, y − yˆ < y − v

Example Set 4 –Week 9

13