Lecture 9: Linear Regression

0/23

Linear Regression

Roadmap
1

When Can Machines Learn?

Why Can Machines Learn?

Lecture 8: Noise and Error

learning can happen
with target distribution P(y |x) and low Ein w.r.t. err
3

How Can Machines Learn?

Lecture 9: Linear Regression

Linear Regression Problem
Linear Regression Algorithm
Generalization Issue
Linear Regression for Binary Classification
4

How Can Machines Learn Better?

1/23

Linear Regression

Linear Regression Problem

Credit Limit Problem

unknown target function

f: X Y

age
gender
annual salary
year in residence
year in job
current debt

credit limit? 5,000

(ideal credit limit formula)

training examples
D : (x1 , y1 ), , (xN , yN )

23 years
female
33,000 USD
1 year
0.5 year
20,000

learning
algorithm
A

(historical records in bank)

final hypothesis
gf
(learned formula to be used)

hypothesis set
H
(set of candidate formula)

Y = R: regression
2/23

Linear Regression

Linear Regression Problem

Linear Regression Hypothesis

age
annual salary
year in job
current debt

23 years
$ 33.000
0.5 year
20,000

For x = (x0 , x1 , x2 , , xd ) features of customer,

approximate the desired credit limit with a weighted sum:

y

d
X

wi xi

i=0

linear regression hypothesis: h(x) = wT x

h(x): like perceptron, but without the sign

3/23

Linear Regression

Linear Regression Problem

Illustration of Linear Regression

x = (x) R

x = (x1 , x2 ) R2

x1

x2

linear regression:
find lines/hyperplanes with small residuals
4/23

Linear Regression

Linear Regression Problem

The Error Measure

popular/historical error measure:
squared error err(y , y ) = (y y )2

in-sample

out-of-sample

N
1X
(h(xn ) yn )2
Ein (hw) =
| {z }
N
n=1

Eout (w) =

E
(x,y )P

(wT x y )2

wT xn

next: how to minimize Ein (w)?

5/23

Linear Regression

Linear Regression Problem

Fun Time
Consider using linear regression hypothesis h(x) = wT x to
predict the credit limit of customers x. Which feature below shall
have a positive weight in a good hypothesis for the task?
1

birth month

monthly income

current debt

number of credit cards owned

Reference Answer: 2
Customers with higher monthly income should
naturally be given a higher credit limit, which is
captured by the positive weight on the monthly
income feature.
6/23

Linear Regression

Linear Regression Algorithm

Matrix Form of Ein (w)

Ein (w) =

N
N
1X T
1X T
2
(w xn yn ) =
(xn w yn )2
N
N
n=1

n=1

2
T
x1 w y1

T

1
x
w

y
2

2


...
N

xT w y
N
N

xT1

T

1
x2 w

...
N

xT
N
1
k X
w y k2
N |{z} |{z} |{z}
Nd+1 d+11

2
y1


y2

...

yN

N1

7/23

Linear Regression

Linear Regression Algorithm

min Ein (w) =

w

1
kXw yk2
N

Ein (w): continuous, differentiable, convex

necessary condition of best w

Ein

Ein (w)

Ein
w 0 (w)
Ein
w 1 (w)

...
Ein
w d (w)

0
0
...
0

task: find wLIN such that Ein (wLIN ) = 0

8/23

Linear Regression

Linear Regression Algorithm

The Gradient Ein (w)

Ein (w) =

1
1
kXw yk2 = wT XT X w 2wT XT y + yT y
|{z}
| {z }
|{z}
N
N
b

vector w

one w only
Ein (w)= N1

aw 2bw + c



Ein (w)= N1 wT Aw 2wT b + c

Ein (w)= N1 (2aw 2b)

Ein (w)= N1 (2Aw 2b)

simple! :-)

similar (derived by definition)

Ein (w) =

2
N

XT Xw XT y

9/23

Linear Regression

Linear Regression Algorithm

Optimal Linear Regression Weights

task: find wLIN such that

2
N

invertible XT X
easy! unique solution

wLIN =

1
XT y
XT X
|
{z
}
pseudo-inverse X


often the case because

XT Xw XT y = Ein (w) = 0

singular XT X
many optimal solutions
one of the solutions

wLIN = X y
by defining X in other ways

N d +1
practical suggestion:
use well-implemented routine

1 T
instead of XT X
X
for numerical stability when almost-singular
10/23

Linear Regression

Linear Regression Algorithm

Linear Regression Algorithm

1

from D, construct input matrix X and output vector y by

xT1
y1

xT

2
y = y2
X=

T
yN
xN
| {z }
|
{z
}
N(d+1)

calculate pseudo-inverse

N1

X
|{z}

(d+1)N
3

return wLIN = X y
|{z}
(d+1)1

simple and efficient

with good routine
11/23

Linear Regression

Linear Regression Algorithm

Fun Time
After getting wLIN , we can calculate the predictions yn = wTLIN xn . If all yn
similar to how we form y, what is the matrix
are collected in a vector y
?
formula of y
1

XXT y

XX y

XX XXT y

Reference Answer: 3
= XwLIN . Then, a simple
Note that y
substitution of wLIN reveals the answer.

12/23

Linear Regression

Linear Regression for Binary Classification

Linear Classification vs. Linear Regression

Linear Regression

Linear Classification
Y = {1, +1}
T

Y = R

h(x) = sign(w x)
err(y , y ) = Jy 6= y K

h(x) = wT x
err(y , y ) = (y y )2

NP-hard to solve in general

efficient analytic solution

{1, +1} R: linear regression for classification?

1

run LinReg on binary classification data D (efficient)

return g(x) = sign(wTLIN x)

but explanation of this heuristic?
19/23

Linear Regression

Linear Regression for Binary Classification

Relation of Two Errors

r
z
err0/1 = sign(wT x) 6= y


2
errsqr = wT x y

desired y = 1

desired y = 1
squared
0/1

err

err

wTx

wTx

err0/1 errsqr

20/23

Linear Regression

Linear Regression for Binary Classification

Linear Regression for Binary Classification

err0/1 errsqr

classification Eout (w)

VC

classification Ein (w) + . . . . . .

regression Ein (w) + . . . . . .

(loose) upper bound errsqr as ec

rr to approximate err0/1
trade bound tightness for efficiency

wLIN : useful baseline classifier,

or as initial PLA/pocket vector

21/23

Linear Regression

Linear Regression for Binary Classification

Summary

1
2

When Can Machines Learn?

Why Can Machines Learn?

Lecture 8: Noise and Error

3

How Can Machines Learn?

Lecture 9: Linear Regression

Linear Regression Problem
use hyperplanes to approximate real values
Linear Regression Algorithm
analytic solution with pseudo-inverse
Generalization Issue
Eout Ein 2(d+1)
on average
N
Linear Regression for Binary Classification
0/1 error squared error
next: binary classification, regression, and then?
4

How Can Machines Learn Better?

23/23