Simulation of the complex

Hussler-von der Malsburg Dynamics

Dr. P. Nonnenmann
April 22,2008

Contents

Introduction
Neuro- biological Foundations

1.1

The central Vision track

1.2

Retinotopy

1.3

Goals of mathematical modelling and simulation

1.4

Results

The mathematical model

2.t

The dynamical system

2.2

The associated global attractor

.)2

t1
j)(

Stationary points

10
10

t2
13

On the existence of stationary points

15

A conjecture based on algebraic geometry

16

Convolution

3.1

18

The discrete lD-Fourier Transform

3.2 The discrete lD-convolutions

3.3

Discrete 2D-convolution

3.4

The associated C-Operators

Stability via Linearization

4.1 Linearization
4.2 Stability

of stationary points

18
19

20
2L

24
24
25

Numerical simulation

26

5.1

26

Computational details of the simulation

5.2 A combination

of cyclic and linear convolution

27

5.3

Linear versus cyclic dynamics

28

5.4

Cyclic dynamics with different starting points

30

5.5

A different C-operator: the

Sobel-fil1,er

35

5.6 Sensitive hehavior with respect to u,0

35

5.7

36

Eigenralues of the linearized system

Conclusion

39

Chapter 1
Neuro-biological Foundations
This Chapter introduces briefly the anatomy of the visual system and the problem
of self-organization in question.

1.1

The central Vision track

The level of the receptors lies on the opposite side of the retina, opposite to where
the light comes in (rrinside'r). Between the receptors and the rroutside", we have the
level of the horizontal cells, the bipolar cells, the amacrines and the ganglion cells. The level ofthe receptors consists of2 types ofphoto receptors: 120*106
Stbchen and 6 x 106 Zapfen. The Stbchen are mainly important during times
of low light intensity, whereas the Zapfen are responsible -for color daylight vision.
In the receptor cell there is a molecule called 11-cis-retinal. After absorption of a
photon, the 11-cis-retinal transforms into an isomer, called all-trans-retinal. After
a geometrical change in the relative positions of the molecules, the early receptor
potential is created.

This signal is being transduced to the mentioned cells and finally to the ganglion
cells. The visual information is being transferrpd to the brain via the nervus opticus, which is built out of the axons of the ganglion cells. The nervi optici unite to
the chiasma opticum, where the fibers stemming from the nasal part of the retina
cross to the opposite side. The fibers from the temporal part of the retina stay on
the same side and connect to the first central processing centers of the vision track:
the corpus geniculatum laterale (CGL) and the tectum opticum. Only
about 101,100 of the fibers end at the tectum, the others at the CGL.
The CGL axons t'run" as a fiber called radiatio optica to the primary visual
cortex (area l7). From there, connections exist to the secundary (area 18) and
ternary(area 19) visual cortex; see [ff] .

1.2

Retinotopy

The pa,per ILl anall'zes a probieur of biological rnorp]rogenesis ]tv self-organizatiril:

the formation of so-r:alled retinotopic prrojections. During ont,ogenesisi axons ol neurons of the retina start to grow to a region of the midclle brain: the tect,um. In the
beginrring, t,he synaptic connections are rather ranckrm. In the aclult, neighbouring cells o{ the retjna project orrto neighbouring cells of the tectum: retinotopir:
projection.
Assuuring there are two chains of ,A/ neurols in both retina antl tectum, we call the
strength of the connection between retina cell r ancl tectuur cell f the weight ur" IFR
The weights form arr (l/,ly')-rnatrix (u.,1.) and the two projectiorrs are represented by
uratrices which have nonzero entries only in the main- and antldiagonal ( definition
is given later ), respectively.

A mechanism ca,pable of generating retinotopy

cooperativity and competition, see Fig.1.

ha^s been

described

in [2, 31. It

Fig.1

Retins

--+++-E

a
U

Modification of weigts. Elevated weights whicl lie closely together iq tle conaection-matrix
cooperxivity (*)- Due to competitior this growth is compensated by a decrease in the

iocrease by

corresponding columns aod rorrs

(-).

uses

Adiacent, rieiglrts arnl-,lify tirernst-.lves through cot,r1-relaLiLrtr (*), ancl ilre being inhibi1,ed through comlret,ition trv the corresp,r.,nding rou,s and colun-rns (-) .

1.3

Goals of mathematical modelling and simula-

tion
A main feature is the formulation of cooperativity and comlretitivity in mathematical
terms, the main object of investigation is to explain the dynamical betravior of the
weights and to detect general structures.
By anal),tic reasoning several properties of the dynamical systeur modelling the problern aborre can be detected: continuous dependenc5., stationary points, boundedness
of orbits, attractors, . . . . By simulations some questions ma)' be checked concerning
the long-term-behavior of the system. For ttie theory of dl,n3pi.ol systems, see eg.
[16,17,181.

L.4 Results
In this thesis we cornpre tlrr: cyclic convoh tion, r'l here there are r:ilcles of r;ells. rvhich
is biologically unrealistic, u.itli the biologicalll, more plausible linear cornolution (In
[1] the cyrtlir: convolution ri,as used.). Linear convolution means ihat the boundarlcelis are being treated in tlie follou,ing u,ay : tectuur artd retina a,re being continuecl
b1' artificial cells lvith u,eight zero and convolution is done on this artificial fielcl of
c:ells.

Since for linear convolution some anai-vtic steps (unknown stationaly points, invariance of eigenspaces, .. . ) can not be done, a synergetic analysis seems no longer
possible: Therefore we use numerical methods lbr analyzing the dynautics ancl the

stabilit.y o[ tlre solutions.

Chapter

The mathematical model

Here we formulate the model of retinotopy by non-linear, high dimensional ordinary

differential equations, fo[owing

2.1

[11.

The dynamical system

As mentioned above, we assume ihat both retina and tectum are 1-dimensional
chains, each consisting of trf cells. Hence we consider the real (,n[, -n/)-matrix ttt :
(u1,.) of weights between cells t, r . The form of the equations describing the temporal change u' of w is given as follows:

w'r,,:F1,(w')

:: f,,,(w) - l,l(2N)ur, ( f4,(w)n j Ortrl)

t,T

:1,..., N,
(2. 1)

wherea)0and

ft,,(r)::a t

tt-4,C6,,(w)

Cr,,(r),:

k(t

t, ,\

- i,t' * i)rui,i

(2 3)

,i,j:|

(2.1) is a system of first order of cubic equations in N2 variables. ft,,(*) is the

growth rate of u.r, consisting of the growth rate a of the incoming retina axons to
ih. t""tr* and the cooperation term Q,,(u). The latter is the sum of products

k(t*i,r-i)utti
where k(t

* i,r - j)

describes the amplification of synapse (t,

10

{2.4)

r) by synapse (2, j).

The C-operator has been written as a general convolution: we will distinguish between the linear and cyclic convolution (see Chapter 3). The kernel k is normalized
as follows:
N

k(i,i)) 0,t,r : 1,....N, I,k(i,3) :

i:|,.i:\

1.

(2.5)

This property is assumed throughout the thesis without further mention. For most
of the considerations the kernel k is derived from a standard Gaussian:

k(r, :: erp(-(t;'
Usually we put

+ y'1112"'17

a:0.6.

Since the righthand-side F of the differential equation (2.1) is a system of polynomials in the variables arrr it is differentiable and so it is locally Lipschitz continuous.
Hence there exists a uniquely determined local solution for a given initial value 'u.,0:
ru'(s)

F(Tr,'(s)) ,

-s

i0,

7) , tu(0) :

,n0

,uthet'e T >

(2.6)

A;

see [r91.

To obtain a global solution it is sufficient to show that such a solution is bounded

in the domain [0, f) of definitiorr. A first step in this direction is to prove tha,t the

hypercube
K7,1

:: l0,l/] x .. . x [0,.n/]

is an invariant subset.

Lemma 2.L Let w0 e K1s .

ut(t') e K1s for all t> 0.

Th,en,

the a,,ssoc'iated solutic,n w erists for all tr,mes and

Proof:
As we knou,, rve have a uniquely determined local solution
ur :

10,

T)

R.N'N

utu(st) : 0 for som s1 [0,7). Then by uniqueness reasons tli.(s) :

foralls)sr.
Suppose wt (sr): N for sorl s1 ) 0 . Then

Suppose

w|, : f*(*) - lf 2u6lSu, *

SuTl

,A/

Su, : 1/N ft,(w) > llN(a*a1,,.C1,,(ru))+r/,nrf

i.:1

11

where we have estimated only ,S2., ,Sq is estimated courpletely analogous.

.|,,(st) :

tuy,,(s1)Ct,,{*)

- lf2u6lSu, *

Sull

* t)lNc')
-112(N-1)a<0,
-Lf 2w6(s1X(N

This

sho'ws

that the local solution stays in the hypercube K,nr. By a rvell

reasoning, the solution is deflned for all times and stays in the h.vpercube

2.2

knc,rvn

K1r. I

The associated global attractor

Here we define the irnportant notion of an attractor and determine the attractor
explicitly for the system in question. Notice that the dynamical systern (2.1) is only
meaningfulforbiologicalreasonsrvhenu,0 RI'',: {A : (ar)lau ) 0 for a\li,,i *:

1,...,1/).

\\re clefine the flow induced by the solution ur starting at tuo by:
O(f . uo)

::

tr,(l),

) 0. tr,o e Rf'N

Definition 2.2 A subset A c R{'N i,s called, an attractor

1. A i,s closed a'nd bou'nded
2. A i,s 'irtuariant, that i,s 0(R+, A) C A
3. A i,s attract'ing, that i,s
1U eU(A)1U

cVlty )0Vu6

i.ff

e VVt>ty@t(t)e

tJ')

Theorem 2.3 The hypercube Ky is a global attractor.

Proof:
Clearly, I{,nr is bounded and closed, hence compact.
Set

[.I

::

[0,

(N+e)]N,N, V ::[0, (l/*2e)]N,N,

We have to shorv that each orbit starting in I/ ends up

Lemma 2.5, we can rewrite our equation (2.1) as

w'r, :

o,(i

0.

U . As

in the proof of

- utt)*w6c6(w)(l*w*lN) - (tl2N)wnllrru,c*(r) +luactl@))

i.*t

12

j*r

Assume: There exists

t,r with w,(s) > l[+e

for all s

I c ]R and ut1,(s) I

IY

*2e.

Then

*'rr(t)

cy(1

-.)

+ (N +

z.)iI

k(t

- i,,r - j)(r/ + 2e)l(1 -

(r/ + dlN)

xl

a(1

-N*.) +l/(l/+2e)2(-e):: A,[10

Therefore

u(1)
rvhich shows that 'u(s)

s>*efl'L

IL

TL

I ,'G)ds<AI JoI tds<Mt

-u(0): Jo
< l/ + e for all s Z *e/l[.

Therefore, wb(s)

"

e [/ for all

10

Figure 2.1: The orbit ends in K1.

Figure 2.1 is an illustration of Theorem 2.3. Three components of the trajectory are
shorvn starting in u(0) : 61[ e IR5'5 ending in a stationary point (from the numerical
point of view) rvhich is approximately equal to the diagonal matrix with entries 4.7

in the diagonal.

2.3

Stationary points

Here we present some results concerning stationarl, points.

Definition 2.4 A matrir P e RN,N

ti,on F

i,s called a stat'ionary poi,nt

i.fJ

F(P)

13

o.

of the matri,r func-

Consider the property

: n.

c(r)

This property is satisfied when we use cyclic convolution as one can

see

very easily;

see also [1].

Lemma 2.5 Sup,tose that the property C(il) : fl zs satisfi,ed. Then ln the case
a : 0 we haue only one h,omogeneous real stat'ionary po,int, namely the matrir n
uhich'is a matrir w,ith entrtes 1.
Proof:
After some arithmetics,
wl,

1ve

can rervrite system (2.1)

: a(l - wt,) * w 6 C 7, (ut) (1 - u: r, I N) * (t I

as

N )w 6" [I

rr. C,, (r) +

u q C 71 (w)]

(2 7)

Setting lug: 0,[., n, C, u,e get:

F

Therefore o

(*)

r!'i

ll'l*4;

or

a: *J4,

i*,

)Ylfr;:).'
I

u,hich are trvo imaginary zeros.

Remark 2.6

The reformulatr,on of th,e rigltt-hand .side r,rt, (2.1) suggest t,wo types of
cand'idates of stat'ionary potnts P : (pni) IRN,N;

.
.

pt5

p4

x N for seuero,l i, j (P approrimately

1 euerywhere (P

n)
a,

loose Ttermutat,ion matru,r)

Following [10], rve define:

Definition 2.7 A

loose Ttermutati,on, matriris one that conta,ins at most a single

i,n each row a'nd column, and 0's euerywhere else.

Theorem 2.8

Suppose that the proyterty

lctose permutatr,on

(2 1).

C6): { z's satisJied,. If a : A, for arty

:: NP i,s a stationary Tto,int of system

matrh P of si,ze,Ay' x -Ay', W

Proof:
See

[101.

Thus the number of stationary points is at least exponential in N, in the case a

In general, the assertion of Theorem 2.8 does not hold when a > 0.

t4

0.

2.4

On the existence of stationary points

As rve have seen in Section 2.1 the sign of the value of the righthand side
dif{'erent boundaries of the hypercube K1,. is different:

F(...,0,...)r.>0, F'(...,N,...)r<9.

on

(28)

This suggests that F should have a zero P in the hypercube, that is F(P) : 0
due to reasoning as in the mean value theorem. But the mean value theorem is
in a straightfonvard manner applicable in a one dimensional situation only. Fortunately there is a generalization to the higher dimensional case which is knorvn as
the Poincare-\,[iranda Theoreur.; see [20].

{ n and
li,: {r e I" I x1, :0}, If ;: {r : rr :

Definition 2.9 Let 1" :: [0,1]' e R" ,k

called the k-th

oTtytos'i,te

1}

faces.

Theorem 2.1O (Poincare-Miranda ) Let

.f : I" '---* R", -f : (fr,.
be

a continuous ?nl,p. Ass'ume for all k

.,

f")

fnUi) < o, /e(r/) > o

Ttten there erists

reln: f(t):g

Theorem 2.LL There erists

a stat'ionary poi,nt P

in the

dynam,ical system (2.7)

for

the l'inear cose.

Prooft
Apply the Theorem in

[20].

Unfortunately, Theorem 2.11 gives no new information when the property C([) : g
is satisfied ,since we know already the stationary point I. But in the linear case,
which is in the focus of this thesis, we have now a stationary point by analytical
reasoning.

15

2.5 A conjecture based on algebraic geometry

In this section rve forurulate a conjecture concerning the stationary points seen from
a totally different subject: algebraic geometry.

Definition 2.12
a) An

algebraic system of equat'ions ot.rer lR is a system of the Jorrn

fo(Xr,..., Xn) :0
where

/, e RlXr,...,,Y,]

,'i:7,. -.,m,

(2 e)

are polynomials.

b) A suhset 1/ c CN is called an ffine R - uariety i,ff there i,s a ,systern (2.9)

.such that V i,s tl'r,e set of all soluti,ons of (2.9) .

Definition 2.L3 Let X be a set and Toyt be a set of subsets of X,

1. $,X e Top,
2. if (At)ier 'is any family Ai e Top, then f,,r, Ai Top,
3. if A1, Az e Top, thert' A1U A2 e Top

such that

X, Top

The sets A e Top are called closed sets'in

a totrtology and the pa,ir (X,To,p)

a topolog'ical space.

Remark 2.L4 A R-uariety V can be endowed

u:i,th the Zariski toytology, r'endering

it a letpologtcal spnce.
Definition 2.15 Let X be a topologi,cal suace.
Th,e Krull rli,ntensi,on dimX of X i,s the supremum of all lengths n of atl cha,ins

* X*t,'i:0,...tzn-r
subsets Xi . Also let dirn0 :: *1

Xo C Xr C
of closed, i,rreduci,ble

"'

C Xn wi,th

Xt.

(See [12] for the definition of "irreduciblerr.)

Remark 2.L6 Although this notion

'is uery general, ,it reduces ,in spec,ial cases to

ord,'inary not'ions of dimensr,on (eg. dimension of a mani,folcl ).

Definition 2.17 Let A be a commutatiue ring.

An i,deal I c A'is a subgroup of A, such that r[ C I for all r e A.

16

First, let

N:2.

Thenu,eassumefort,r:1,2, Ft,(a,u) :6, F beingthe

rigthhand-side of our equation. Let V be the corresponding affine varietl,. Let also

A
lvhere

is the ideal

::

IRlo, 1rtt, ..,ut22),

in generated b1,the

I ::

(Ftr, .., Frr) C A,

fl; . Then we use (see [12]);

Theorem 2.18 (The Krull dimension) W" haue the equ'iau,lence

ot' the followi,ng

assertions:

(a) di:,nV :

1.

(b) For all P e AlXr,...,)(r], P hom,ogeneous, ulith P([,rr,...,Frr):0

P e IAlXr,...,&]

impli,es

Conjecture

1:

dimT:1
\,Iost prolrably, it is possible to proof (b) in the Theorern, due to the structure of
the polynomial in cluestion.
dim I,' : 1 means, that the stationary points lbrm an algebraic curve, if cr is considered as a rariable.
If a is a constanl,, then the same conjecture shorvs dim v.- : 0. B), delinition of the
Krrrll dimension, there exist m stationary solutions. nt, ) 7 .

L7

Chapter

Convolution
This chapter gives some defrnitions and results concerning the operation "convolution", mainly because we did not find the discrete 1D, 2D linear convolution
properly defined in the literature. So it should serve as an overview; see [21]. Here
we use the notation which is comrnon in image processing.

3.1

The discrete lD-Fourier Transform

LetnN,n)1.\\reset
'

a.::

exp(2tri,f rt)

and call wn tlte primitive n-th root of unity. Clearly,

r,r,e

have a*.

The map^: C'-*--, C", rvhere

n.-

r'nkrx,J

4,-

k-0

is called

: 0,.... n -

1,

lD-discrete Fourier Tlansform (DFT,,). The map-: C,

n*l

\ :: (tln)\a;ikz1,, j :0,...,n *

-*

C", rvith

7,

k:0

is called inverse

discrete Fourier

tansform (DFT;l).

as

it

is rvell knorvn,

l', C" -----* C'

are linear and bijective maps? with

-: -1

Obviously, the implementation of the discrete Fourier Transform leads also to an

implementation of the inverse discrete Fourier Transform. As it is knoln, the computational effort to evaluate the discrete Fourier transform is O(n) when one uses
the idea of "divide and conquer" (fast Fourier transform). The importance of the
discrete Fourier transform consists in the fact that it is the main tool to evaluate
convohrtions.
1B

3.2

The discrete lD-convolutions

Convolution is a binary operation on vectors or matrices.

Definition

C.

3.1-

(Linear convolution) Let a:: (as,..,an-r),b:: (b0,..,o-r)

Define

min{L,n-l)

(o * b);

::

a*bt-r,, I :

0, .., 2n

- 2.

k:nraz(0,1-n*1)

a+b

is called the linear convolutioa

Note that o * is an (2n

oJ a,b.

1)-tuple over C.

Notation:
Hadamard product: Let z, y tie complex n-tuples.Then r O y is the cornponent'rvise
producl vecLor.
Padding: Let r C', then (rl0;) e qn+l ir the vector rvith I zeros at the end.
linear convolutiou can be computed by using the discrete Fourier transform:

Theorem 3.2 Let a,b e C.

(a x bl01)

Then

FT;

lD FT2*(a10,)

D FT2"(b10")).

Definition 3.3 (Cyclic convolution) Let a,b e C.

We set

n-1

(a8b)7,: I

a1,bg--x1 mod

n, l:0,. ..,fr-1,

,t:0

and ue call a & b the cyclic

convolution of a,b.

Theorem 3.4 Let a.b C".

a

Then

& b : D FT;i|D FT,(a) o

FT,(b)1

The following lemma connects cyclic convolution and linear convolution.

Lemma 3.5 Let a,b e

C.

Then
(a * bl01)

(a10")
19

(bI0").

For a vector J

::

(rr,.., ,n-r) e C', we define

/ *n

rn-L

t,,
\J"- '

xn*2

lr.

,,\

''n /

The rnatrix Z(r) is called the circulant matrix of r.

Lemma 3.6 Let a,b e C.

Then

a8b7 bBa: Z(h)a-Z(a)b.

Theorem 3.7 Let h e C". ZUt) has e,illenuecto,rs
g".....!Jn't , a- -- (1.oT....,rf("-")
and e'i,genualues
n-l

40,..,

3.3

)r-r

: Lhr rf^
k:0

Discrete 2D-convolution

Definition 3.8 Let f

:(f(i,,j)) C",,,h:(h{i,j))e C'".his

min(i,tr-7')

(f

I r,

'r h)(i,

i) ::
k:mat

(0,i.

al*ernet.

mt(j.n-L)

- n*

1) L:mat (0,i * n* r)

tft't)h(i
f (k,l)h(i -- k,',ij - l),i, j :0,

.. ., 2n - 2 ;

n.*1

(f a

h)(t,,

These are

j) :: I /(*, t)h((i, k,I:O

k) -oa ,,(i

- /) *oa n),i,i -0,"'rfr-1'

the 2D discrete linear and cyclic convolution,

respectively.

Remark 3.9 These can be written as a product of 1D conuolutions:

Let a,,b e C". Remember the Gauss kernel
h(, , y)

erp[- (r2 +

a2) I Qo2

)l

h'

(r)h' (y) where

Let
aob

:-

(or bt)*, e c,,,n, k,l

20

o,

h'

(u)

..,n

::

erp(-u2 I Qo2))

be the

dyadic product of the uectors a,b. I'hen

n-1

(a I h'). (b e ') : [I

A:0

n-l

"o

h'(i.

-,t)],

ll,@*b)h(i'

II

bt

h'U

* t)lj

l:0

- k, j - t))u,i :

(a o b)

KI

On the left side is the dyadic product of 1D corvolutions, the riglrt side is a 2D
convolution.

3.4

The associated C-Operators

Now u,e are prepared to define the C-operator by 2D-convolution which u,e warrt to
use in s5-stern(2.1):

Linear convolution: C(tu) :: k * ut

Cyclic convolutior; (l(ur) :: k g w.

Here k

: (ki)

lRn'n is a kernel; see Section 2.1.

Using Theorem 3.7 (circulant matrix), we can calculate the eigenmatrices and eigenvalues of the cyclic C-operator:

Lemma 3.7O

We hu'ue:

C(y* . a")
fnr all mjn
Proof

:0,

...,ly'

: \*

)'n(y* o y")

C(g*.y") : (y*oy")&k

(a^ A

k,).

(y" A k,)

: (Z(k') y*) c (Z(k') y")

: (A* U*) . (\" A")
: )_ , (y*.g")
2t

r
For readabilitl,, su1

rn: n:7

t : (l

N-1

: L

N-1

*;,,'11L nir')
t:0

rr,, erp(N-r2r

i.(p

+ l))

P"L

and

y' .

yt : (rr ,')r,,
erplt{-L2r i(k + t)l

where a,r is the n-th root of unity. These are exactly the expressions obtained by
synergetics in [1].

Lemma 3.7L Suppose tlta,t 1 is an e,igenualu,e of C . Ttten

lrl s

t'

Proof:
Letubeaneigenvector: C)('u)-^tu,ul0.Letlrr,,l:ma-x{lu;3llz,

Then lu1,,l ) 0 and rve have

lrllr,l :

j:1,...,,^,r}.

lc*(r)l

I k(t*i,r*i)uril
i:L,i:t
k(t

i:\,i:l

* i,r * j)lu*|,

k(t-i.r
i.:l,j:l

-.7)

:1.
I

Eigenvalues of the cyclic C-operator

Eigenvalues of the linear rnap k E, . : ]R.N,N

IRN,N,

(the maximum is the entry k(0,0) .

We shorv the four largest eigenvalues for different
Mar:Las.

N.

k is the shifted Gauss kernel

The calculations are done with

N:5

?i:

1.000 4.762 A.762 0.762

22

l/:9
.^/

7i

1.000 0.9183 0.9183 0.9183

7i

1.000 0.9697 0.9697 0.9697

15

Eigenvalues of the linear C-operator

Eigenvalues of the map c

* . : IR.N,N

---_+ IR.N,N,

c is the centered Gauss kernel.

Again, rve show the four largest eigenr.alues for clifferent l/ , calculated rvith M.qrLns. The other eigenvalues are close, too"

l/:5
.

-i

0.9092 0.7896 0.7896 0.6857

dr

0.9661 0.9176 0.9176 0.8716

di

0.9866 0.9668 0.9668

N:g
A/:

15

0.9,175

The results above suggest

Conjecture
Let 7N

(rfl,".,7nry,)

l-(N), the vector space of all

bounded, real sequences.

Then

JIL('N-dN):o'
rvhich rve deline as follows:

ve >01,41 NV,A/ >

MVi:7,...,N2

lryr_,rl

<..

(This could be called uniform discrete convergence .) It would be formidable

to prove this result. The main difficulty consists iu the fact that the operator C is
not symmetric in general. Therefore the powerful tool of Raleigh-quotients cannot
be applied.

All this holds,for o

0.6 in the Gauss kernel. Let

is very narrow. Then for all N, ri we have

Ji:6i:1.0

23

o :0.1 , so that the Gauss kernel

Chapter 4

Stability via Linearization

4.L

Linearization

Now we want to calculate the jacobian Dtr(P) of ttre righthand side f in (2.1) ',vhose
eigenvalues may be used to decide whether a stationarl, point is I'stable'r.

Notation:
Z :: X e Y is the pointu,ise product of X, Y.
Let F(P) : 0 . We linearize F in the neighborhood of the stationary point P. As
in [17] . lVe use

B:IRN'N_-IRN'N,
Br,(W)

.^/,^/
:: (112N)(Lr7 + utj),t,r' :7,.
i:L

.i:r
Using this mapping the righthand-side in (2.1)
F(P)

aI" + P o C(p)

..

,N, tf W : (.0)

can be fbrmulated in a compact form:

* p e B@n+ pe C(p)) :

0.

LetVIRN,N.Then

F(v + P)
: an+ (v+ p) o (cv + cp) * (v +p) o (ar + B((v + p) e (cv + cp)))
: v o cP + P a cv - (y + P) o B(an+v a cv

+vocP+Pocv+PacP)

V oCP + P

oCV

-V o B(C{L+ p aCp) * p o B(V aCp

.1

+ p aCV)

Cancelling terms due to F(P) : 0 and neglecting terms which are not linear
obtain for the jacobian of F in P

in V

u,e

DF(P)(v)
: V aCP + P oCV
: v a cP + P ct cv

4.2

-V oB(a1 + p oCp)-pe B(V a p + p aCV)

- av *v o B(P a cP) - p a B(v ecp) - p a B(p a cv)

Stability of stationary points

Let

B,(r) :: {u e R.N,N : lr - ul ( r}, r }

fl,

rvhere l.l is any matrix norm.

Definition 4.1
Ve

The stationary po,int

P is called

stable i,ff

> Old > OVr B6(P)(O(t,r) e B,(P) for att r > 0),

where Q is the flow a,ssociated to tlte sohtt'ion with starti,ng poi,nt

point P i,s callerl, o,ttractiue i,ff

ld > OVr B,,(PX,!*o(l,r)

Tlr'e stat'ionary poi.nt

ds

called asymptoticully stable

P)

iff P

r . The stat,ionar.y

i,s

stable and, attract,iue.

A characterization of the property "asymptotically stable,, is given in

Theorem 4.2 Let P

be a stationary

point.

Then follow,ing assertions are equ,iua-

lent:

a) P is asymptotically stabl,e.
b)

i,s attract,iue.

c) Re() < 0 for all ei,genualues ), of DF(P)

Proof:

See

[17].

A property which we need in the discussion of the simulation results is given in

Definition 4.3 A stat'ionary point P

'imag'inary,

po s,itiu

e and

neg at,iu

e'ig enu

i,s called hyperboli,c i,ff i.t has nonzero, nonalues on,ly.

25

Chapter

Numerical simulation
5.1

Computational details of the simulation

All computations

are performed

in N,fATLAB.

Our equation (2.1) together rvith an initial condition is an initial value problern
(r\-P):
(5.1)
u' : F(tu), u(0) : uoi

l/

is the size of the weight matrix

ur.

Let tu be a global sohil,ion. For ttre numerical approximation of this solution \re
the forrvarcl Euler rnethod with step size , :
,!n*1

w" + h,F(tun) , w'

u(0)

(5 2)

Thus, the approximation of the solution are given by the sequence of values u0 , u1
Notice that each ruk may be considered as matrix in IRN,N.

In the follorving, M is the number of iterations.

approximation of the solution at the "fina} time" T

use

..

Therefore tr,If is r.alue of the

:: Mh.

The linear convolution is reaiized b1, using packages in NIe:rLas. Cyclic convolution
is programmed straight{bnvardll, according to the definition.
We shall use pixels for the entries of a matrix trr: the brighter the pixel, the larger
the entry of ir.

diag stands for the rnain, diagx for the diagonal going from left bottom to right
top.

26

Frorn the above, a rather large number of parameters steers each simulation:

l/
a
h
l,t

size of the dynamical systeut

size of the grolvth rate
stepsize in the Euler method
number -of approximating valnes

i,'.:,; ;""*::il:ffi:
o
?uo
q

"width" of the gaussian

initial value
constant in the homotopy

5.2 A combination of cyclic and linear convolution

For combining the cyclic C-operator C" and the linear one C7, u,e introduce a homotopy h :
Encl :: End(RN'N), h: I x Bn62 ---- End

(q,C.,Cr) r_--* 0-dC"*qCt,

u,here

I ::

10,11 and

End are the endomorphisms of IRN'N .

Consider the dynamics fbr

q-0:

cyclic

case;q:1: linear case, N:5,

u,here C. is the cyclic and C7 the linear

convolution.

Conjecture

3:

that is, only for g : 0 the system shows the cyclic behavior.
If q is only slightly greater than zero, the system switches abruptly into the linear
mode. That is suggested by the follo'iving series of test-calculations:

h:
q: l}-a,\0-7,e:

l}-a, h,[ : 106,a

10*10,r1(o)

: r * --

0.1

> *M a: diag-(4.75)

(the "zeros'r outside the diag- are about 0.05).

Note horv close rve are to 0, yet still linear behaviour.

q:10"2,tu(o) :1-

-*

>

tuL{

xn

Finally the cyclic case; of coursel numerically 10-12 is equal to zero.

27

14

246a101214

Figure 5.1: The weight matrix wM as a pixel pattern at the end of an iteration.
left:linear,right:cyclic, a : 0.95

5.3

Linear versus cyclic dynamics

Starting point u(0) : diag(1g-'), h : lA-2, M - 10n, I/

These will be the parameters, unless otherwise specified.

linear

case:

1b.

0.95.

o.e5 o.eb o.eb \

.01
l o.es
a^t-l:I
1.01 ". :
l.ot
o.cs
o.e5
o.e5
o.si I
\
f

o.os

o.cs

I .o

For a comparison betu,een the linear and cyclic case) see Figure 5.1.

a
For the linear case and tr(O)
wM for

0.93.

diag(10*2), we see an abrupt change in the matrix

0.93, 0.923,0.92

For this transition, see Figure 5.2.

0.5.

Lin,ear case:
1!Lr

:(

0.7377

i3;?'

13.8602

l-,-

0
2B

248101214

2468101214

re 5.2:
Figure
5.2: linear: The transition of wM for a
r

10

Figure 5.3'. wtut for a

29

0.g3 (left) and

12

0.92

1t

0.g28 (right)

Cyclic

case:

n.amt o.13oe o
f o'lsos
\
1.3'8s74
o
:l
,.rn,
"-l:::l
0 ... rs.rrzn)
\0
I

Tiny differences occur at the boundaries, because the trvo

(C-operators!).

cases

differ mainly there

0.3, 0.2,0.1

\\'e see the diagonal matrix, for both cases, the diagonal numbers slightly increasing
up to 14.78 i
wM x diag(14.78,.. .,14.78)
;

see Figure 5.4.

2466101214

Figure 5.4:1: Linear or cyclic, shown toM

A typical orbit is shown in Figure 5.5 where a

0.1 is chosen anclzuy2(t),wz,lt),,wz,z(t)

are being plotted.

The values (1.5,7, 1.5) indicate a hyperbolic point.

5.4

Cyclic dynamics with different starting points

We simulate the cyclic dynamics, but now with starting point u(0) :: (Aj|) where
A':0 for all i,, j *t,r and Ali: N. All these initial values are loose perutation
matrices.

Let N

:5,a:0.1

and tir(O;

i,1. Then

,rAr

diag(4.7,
30

..

,4.7)

1.5

1.5

la

Figure

5.5: A typical orbit.Linear or cyclic, a :

0.1., plotted

al',vays

wn(t) , wzz(t) , wz(t')

That demonstrates the strong self-organization capabilities of the system: if rve

have one non-zero entry only in the diagonal, the system moves to the diagonal.
N{oreover, the final values wLt ate always permutation matrices.
Let N:5,d:0.1 and'u(O) :41,2. Then uM is the matrix shown in Figure ??.
Let N : 5, o : 0.1 and tl(O) : 41,3 . Then tlM is the matrix shown in Figure ??
and the orbit in Figure ??.
There is a sharp h;,,perbolic point at iteration step n
Ttre corresponding matrix is plotted in Figure 5.9.

Let

N:5,o:0.1

Let,A/

:5,a:0.1

Finally, let

and

47000 rvhere

,4r1

100000.

tl(0):,42'1. Then wM is the matrix shown in Figure 5.10.

and to(O)

ly':5,o:0.1

:43,3.

Then wM is depicted in Figure 5.11

and u,,(O) : A4,4.We show the orbit in Figure 5.12

There is indication for an hyperbolic point at about (2.2,2.2,2.2).

Other initial matrices of the form At' show similar results.
Before reaching the endpoint, all orbits rivisitil an hyperbolic point:
lrere tu12 : 1lU21 : rD22 :2.2 at iteration step n : 7500.

31

0.5
1

'1.5

2.5
3

-]

1.5

Figure 5.6: tr(0)

2.5

3.5

A1,2, this the final

4.5

matrix tuM.

2
2.5
J

4
4.5
5

1_5 2

Figure 5.7:

w(0):

2.5

A1,3,

3.5

4.5

this the final matrix

32

u.'M

5.5

0.8

0.6

0,4

0.2

0
2.5

Figure 5.8: Here is the orbit with u(0)

A1'3

0.5

2
?.5
3
3.5
4
4.5

5.5
0

Figure 5.9: This is the matrix associated to the poiut

,,

Jd

n:47000 in the last orbit.

0.5
1

1.5

2
2.5
J
3.5
4
4.5
5
5.5
0

1.5

Figure 5.10:

2.s

T.u(O)

3.5

4.5

A2,1, shown trM.

0.5

1.5
2
2.5
3
3.5

1.5

2.5

Figure 5.t1: .u(0)

--

34

3.5

4.5

A',t,shown urrl

?.5
2

1.5

0.5

0
5

Figure 5.12: The orbit rvith

w(0):

tr+'t.

5.5 A different C-operator: the Sobel-filter

For a change, rve finish u,ith an un-biological kernel, called Sobel filter, u,hich
usecl in irnage processing for enhancing edges:

/t

is

1\

sn:{o o ol

\-r *2 -t)

We choose ly' : 5. Therefore the matrix So is enlarged by filling up 'lvith zeros in

such a way that So is located in ttre center.
Consider

ur(0): diag(10-2)o:0.1.

The resultiug matrix wM is drawn in Figure

5.13:

The rate of convergence is lorver than rvith the Gauss filter for the same other parameters. uLM is stable,r,vith four complex eigenvalues; notice that the Sobel-filter
contains negative entries.

5.6

Sensitive behavior with respect to

trO

One aspect of complex systems is the sensitivity with respect to the initial matrix.
This is being illustrated by the nexL two examples. Set -l:5,a:0.1 .
Linear case,

tuo:f

+10-2
35

0.5
1

1.5
2
?.5
3

4
4.5
5
5.5

AA

1.5

2.5

3.5

Figure 5.13: The Sobel-filter, shown

.M:

4.5

lr

'tD"'

diag(4.75)

The orbit is shown in 5.14.

The cyclic case gives the same resuit.
Now we
Linear,

try

a slightly different perturbation

of

1[.

tuo:1[+i0-2

,nM

with orbit 5.15

diag.(4.75)

The cyclic case gives a broken di,ag"

5.7

So

tiny differences produce rroppositerr results.

Eigenvalues of the linearized system

The eigenvalues of the linearization in a stationary point are used to decide whether
the stationarl, point is stable.
36

10
Figure 5.14: First perturbation of I.Note that there is hardly any hyperbolic point.

1.4
1.2
.t

0.8
0.6
0.4
0.2
0
2.5

00

Figure 5.15: Second perturbation of X.Note that there is a probable hyperbolic

point.

J/

Cyclic case :

0.1, and tu(0) : I.

The eigenvalues ; of the linearization
table:

Let N

5,0

:1[

DF(wM), utilI

0.48 0.48 0.48 0.48

0.20 0.20 0.20 0.20
0.20 0.20 0.06 0.06

are gir,-en in the follou,ing

0.20
0.20
0.06

0.06 -0.22 *0"22 -0.22 -0.22

-0.40 -0.40 *0.40 -0.40 -1.i0
Let

y, 'i:

1,...,1/2, be the eigenvalues of the C-operator.

Result according to the simulation:

1. All eigenvalues have multiplicity 4 or B (except 1).
2. \\,'e have l - h - a (except at the boundary of the matrix )
These properties are in agreement wittr [1]. 'Ihe point u,M seems to be hyperbolic

Linear case

l/ :5,a:

0.1, and ur(O) : I.

Tlre eigenvalues ); of the linearization DF(ru^r'), lDM
follorving table:

Let

-1.18 -1.19 -1.23 -1.27

**+**
*+***
*****
*
*

*-2.30

Result of the simulation:

1. All eigenvalues have algebraic multiplicity
,a. /\z
\. J ^.
7- lt

u.

3. The point torr

seems

to be stable.

stable, too.

3B

1.

diag-(,X.76)

-1.2{)

*2.37

arer gir,en

in the

Conclusion
complex self-organization properties of the Hussler-von
der Malsburg equations are retained when using linear convolution instead of the
original cyclic convolution - which is biologically more realistic. From many starting
points the dynamics reaches one of the 2 diagonal weight matrices. The difference
lies in the algebraic multiplicities of the linearized eigenvalues. This phenomenon
fits exactly in the theory of circulant matrices, which describe the cyclic convolution.

We have seen that the

Moreover the dynamics is complex in the sense that tiny changes in the initial
values can produce very different final statioaary states.

Another result is that the biological hypercube K1,, is an attractor for the system.
Also we proved the existence of a stationary solution for the linear convolution,
through the Poincare-Miranda theorem.
We gave also a conjecture about the eigenvalues of the C-operators as l[ goes to
oo. An interesting future investigation wold be the introduction of a model which
describes the system when N goes to oo.

39

?
Bibliography
[1] C. von der Malsburg A. F. Hussler. Development of retinotopic projections:
an analytic treatment. J. Theoret. Neurob'ia., 2:47-73, 1983.
[2] D.J. Willshaw C. von der Malsburg. How to label nerve cells so that they can
interconnect in an ordered fashion. Proc. Natl. Acad. Sci,. USA,74:5L76*5L78,
1977.

[3] D.J. Willshaw C. von der Malsburg. Differential equations for the development
of topological nerve fibre projections. SIAM-AMS Proc.,13:39-47, 1981.

t4l C. von der Nlalsburg. The correlation theory of brain function. MPI Jor Biophysi,cal Chemi,stry, Internal Report

1-2:

1-80, 1981.

[5] C.von der Malsburg J.Zht. Maplets for correspondence-based object recognition. N eural N etw orks, 1 7(B-9) 131 1*1326, 200 4.
:

[6] C.von der Malsburg J. Zfui. Associative memory of connectivity patterns.

N eurocomputi,ng,

69

:1305-1308, 2006.

[7] H. Haken. Synerget'ics,Introd,uct'ion. Springer,Berlin,

1978.

[Bl H. Haken. Generalized ginsburg-landau equations for phase transition-like phenomena in lasers,non-linear optics, hydrodynamics and chemical reactions. Z.
Physi,k B, 21:105*174, 7975.

[9] H. Haken. Higher order corrections to generalized ginsburg-landau equations

of non-equilibrium systems. Z. Physi,k 8,22:69*72,1975.
[10] J. Zhu. Analysis of map formation in visual perception. FIAS, pages 1-14,
2007.

[11] M. Guessmann. Selbstorganisation zwischen Mannigfaltigkeiten euklidischer

und nichteuklidischer Geometrie durch Kooperation und'Kompetition. D'issertat'ion, Uni, Stuttgart, 20A6.
[12] E.Kunz. Ei,nfhru.ng i,n die algebraische Geometrie. Vieweg, Braunschweig,
1997.

[13] F. Lorenz. L'ineare Algebra

L BI Verlag, Mannheim,

[14] S. Lang. Algebra. Addison-Wesley, Reading, 1984.

40

1984.

[lS] G.Tnervs.

Ph,ysiologie des Menschen. Springer, Berlin, 1977.

1982.
[fO1 iV. de 1\Ielo. The Geometry of Dynam,ical Systezns. Springer, Berlin,

[17] J. Baumeister. Differerrtialgleichungen. Script, 2006.

[18] Teschl. I)ynamisc]re Systeme. Scri'pt, 20A5.
[19] W. \&'alter. Gewhnl'iche Di'fferenti,alglei,chungen. Springer, Berlin, 2000.
[20]

W. Kulpa. The Poincare-1\{iranda theorem. The american mathemat'i,c:al

monthly, 104(6): 5a5-550, 1997.

[21] J. Baumeister. Itrverse Probleme. Scri4tt,2007-

4t