Pattern Separability and

Regularization

1
 Radial Basis Function: Clustering

K-means clustering
Data

Cluster j

2
 Clustering is obviously based on a measure of similarity!

The cost function for clustering

Euclidean:

simplified to
3
 Variance of a cluster:

The clustering (iteration):

Start with arbitrary

4 Stop when the means / cluster assignment do not vary.

 K-means clustering and Cover’s theorem

RBF:

In k-means:

Equivalence in terms of Cover’s theorem, when #

of js are sufficiently large.

Least square estimation of weight vectors in a RBF network:

Recall:

5
6
 computation is heavy when is large
So:

7
 So:

in terms of
Sol:

Initialize:

8
 Essentially this is being minimized

Hybrid procedure for RBF learning:

Input layer Vector x of m0 dimension

Hidden layer
RBFs

means from K-means on

Max. distance
Gaussian mean with between centers

Output layer RLS

9 Till n=N
 Regularization theory
Input to output mapping problem, given only discrete points!

vs

A third term allowing examples without desired output

10
Hadamard’s conditions of well-posedness

Otherwise ill-posed!
11