3URFHHGLQJVRIWKHWK0HGLWHUUDQHDQ&RQIHUHQFHRQ 7

&RQWURO $XWRPDWLRQ-XO\$WKHQV*UHHFH

Training ANFIS Structure with Modified PSO

Algorithm
V.Seydi Ghomsheh *, M. Aliyari Shoorehdeli **, M. Teshnehlab **
* Computer Department of Islamic Azad university , Kermanshah, Iran
And Computer Department of Science and research branch
Islamic Azad university of Tehran, Iran
vahidseydi@gmail.com
** K. N. Toosi university technology
Tehran, Iran

Abstract- This paper introduces a new approach
for training the adaptive network based fuzzy
inference system (ANFIS). The previous works
emphasized on gradient base method or least
square (LS) based method. In this study we apply
one of the swarm intelligent branches, named
particle swarm optimization (PSO) with some
modification in it to the training of all parameters
of ANFIS structure. These modifications are
inspired by natural evolutions. Finally the
method is applied to the identification of
nonlinear dynamical system and is compared
with basic PSO and showed quite satisfactory
results.
Keywords: Particle Swarm Optimization, TSK
System, Fuzzy Systems, ANFIS, Swarm Intelligent,
Identification, Neuro- Fuzzy.
I. INTRODUCTION
T he complexity and the dynamics of some
problems, such as prediction of chaotic
systems and adapting to complex plant, require
sophisticated methods and tools for building an
intelligent system. Using fuzzy systems as
approximators, identifier and predictor, is
reliable approach for this purpose [1, 2]. The
combination of fuzzy logic with architectural
design of neural network led to creation of
neuro- fuzzy systems which benefit from feed
forward calculation of output and back-
propagation learning capability of neural
networks, while keeping interpretability of a
fuzzy system [3]. The TSK [4, 5] is a fuzzy
system with crisp functions in consequent, which
perceived proper for complex applications [6]. It
has been proved that with convenient number of
rules, a TSK system could approximate every
plant [7]. The TSK systems are widely used in
the form of a neuro- fuzzy system called ANFIS
[8]. Because of crisp consequent functions,
ANFIS uses a simple form of scaling implicitly.
This adaptive network has good ability and
performance in system identification, prediction
and control and has been applied in many
different systems. The ANFIS has the advantage
of good applicability as it can be interpreted as
local linearization modeling and conventional
linear techniques for state estimation and control
are directly applicable.
The ANFIS composes the ability of neural
network and fuzzy system. The training and
updating of ANFIS parameters is one of the main
problems. The most of the training methods are
based on gradient and calculation of gradient in
each step is very difficult and chain rule must be
used also may causes local minimum. Here, we
try to propose a method which can update all
parameters easier and faster than the gradient
method. In the gradient method convergence of
parameters is very slow and depends on initial
value of parameters and finding the best learning
rate is very difficult. But, in this new method so
called PSO, we do not need the learning rate.
The rest of the paper is organized as follows: In
Section II, we review ANFIS. In Section III we
discuss PSO method. An overview of the
proposed method and applied this method to
nonlinear identification is presented in Section
IV. Finally, Section V presents our conclusions.
II. THE CONCEPT OF ANFIS
A. ANFIS Structure
Here, type-3 ANFIS topology and learning
method which use for this neuro- fuzzy network
is presented. Both Neural Network and Fuzzy
Logic [9] are model-free estimators and share the
common ability to deal with the uncertainties
and noise. Both of them encode the information
in a parallel and distribute architecture in a
numerical framework. Hence , it is possible to
3URFHHGLQJVRIWKHWK0HGLWHUUDQHDQ&RQIHUHQFHRQ
&RQWURO $XWRPDWLRQ-XO\$WKHQV*UHHFH
convert fuzzy logic architecture to a neural
network and vice-versa. This makes it possible to
combine the advantages of neural network and
fuzzy logic. A network obtained this way could
use excellent training algorithms that neural
networks have at their disposal to obtain the
parameters that would not have been possible in
fuzzy logic architecture. Moreover, the network
obtained this way would not remain a black box,
since this network would have fuzzy logic
capabilities to interpret in terms of linguistic
variables [10].
The ANFIS is composed of two approaches
neural network and fuzzy. If we compose these
two intelligent approaches, it will be achieve
good reasoning in quality and quantity. In other
words we have fuzzy reasoning and network
calculation.
ANFISs network organizes two parts like
fuzzy systems. The first part is the antecedent
part and the second part is the conclusion part,
which are connected to each other by rules in
network form. If ANFIS in network structure is
shown, that is demonstrated in five layers. It can
be described as a multi-layered neural network as
shown in Fig. (1). Where, the first layer executes
a fuzzification process, the second layer executes
the fuzzy AND of the antecedent part of the
fuzzy rules, the third layer normalizes the
membership functions (MFs), the fourth layer
executes the consequent part of the fuzzy rules,
and finally the last layer computes the output of
fuzzy system by summing up the outputs of layer
fourth. Here for ANFIS structure (fig. (1)) two
inputs and two labels for each input are
considered. The feed forward equations of
ANFIS are as follows:
wi = Ai ( x) Bi ( y ), i = 1,2. (1)
wi (2)
wi = , i = 1,2.
w1 + w 2
f1 = p1 x + q1 y + r1 z
f 2 = p 2 x + q 2 y + r2 z (3)
w1 f 1 + w2 f 2
f = = w1 f1 + w2 f 2
w1 + w2
In order to model complex nonlinear systems,
the ANFIS model carries out input space
partitioning that splits the input space into many
local regions from which simple local models
(linear functions or even adjustable coefficients)
are employed. The ANFIS uses fuzzy MFs for
splitting each input dimension; the input space is
7
covered by MFs with overlapping that means
several local regions can be activated
simultaneously by a single input. As simple local
models are adopted in ANFIS model, the ANFIS
approximation ability will depend on the
resolution of the input space partitioning, which
is determined by the number of MFs in ANFIS
and the number of layers. Usually MFs are used
as bell-shaped with maximum equal to 1 and
minimum equal to 0 such as:
(4)
(5)
Where {ai , bi , ci } are the parameters of MFs
which are affected in shape of MFs.
Figure (1): The equivalent ANFIS (type-3
ANFIS)
B. Learning Algorithms
The subsequent to the development of ANFIS
approach, a number of methods have been
proposed for learning rules and for obtaining an
optimal set of rules. For example, Mascioli et al
[11] have proposed to merge Min-Max and
ANFIS model to obtain neuro-fuzzy network and
determine optimal set of fuzzy rules. Jang and
Mizutani [12] have presented application of
Lavenberg-Marquardt method, which is
essentially a nonlinear least-squares technique,
for learning in ANFIS network. In another paper,
Jang [13] has presented a scheme for input
selection and [10] used Kohonens map to
training.
Jang [8] is introduced four methods to update
the parameters of ANFIS structure, as listed
below according to their computation
complexities:
1. Gradient decent only: all parameters are
updated by the gradient descent.
2. Gradient decent only and one pass of LSE: the
LSE is applied only once at the very beginning
to get the initial values of the consequent
3URFHHGLQJVRIWKHWK0HGLWHUUDQHDQ&RQIHUHQFHRQ
&RQWURO $XWRPDWLRQ-XO\$WKHQV*UHHFH
parameters and then the gradient decent takes
over to update all parameters.
3. Gradient decent only and LSE: this is the
hybrid learning.
4. Sequential LSE: using extended Kalman filter
to update all parameters.
These methods update antecedent parameters
by using GD or Kalman filtering. These methods
have high complexity. In this paper we
introduced a method which has less complexity
and fast convergence.
III. PARTICLE SWARM OPTIMIZATION
(PSO) ALGORITHMS
A. General PSO Algorithm.
The particle swarm optimization (PSO)
algorithms are a population-based search
algorithms based on the simulation of the social
behavior of birds within a flock [14]. They all
work in the same way, which is, updating the
population of individuals by applying some kind
of operators according to the fitness information
obtained from the environment so that the
individuals of the population can be expected to
move toward better solution areas. In the PSO
each individual flies in the search space with
velocity which is dynamically adjusted according
to its own flying experience and its companion
flying experience, each individual is a point in
the D- dimensional search space[15].
Generally, the PSO has three major
algorithms. The first is the individual best. This
version, each individual compares position to its
own best position, pbest , only. No information
from other particles is used in these type
algorithms.
The second version is the global best. The
social knowledge used to drive the movement of
particles includes the position of the best particle
from the entire swarm. In addition, each particle
uses its history of experience in terms of its own
best solution thus far. In this type the algorithms
is presented as:
1. Initialize the swarm, P (t ) , of particles such
that the position xGi (t ) of each particle
Pi P (t ) is random within the hyperspace,
with t = 0 .
2. Evaluated the performance F of each
particle, using its current position xGi (t ) .
3. Compare the performance of each individual
to its best performance thus far: if
G make converges faster than basic algorithm.
F ( xi (t )) < pbest i then:
7
G
pbest i = F ( xi (t )) (6)
G G
x pbest i = xi (t ) (7)
4. Compare the performance of each individual
to global best particle: if F ( xGi (t )) < gbest then:
G
gbest = F ( xi (t )) (8)
G G
x gbest = x i (t ) (9)
5. Change the velocity vector for each:
G G G G G G
v i (t ) = v i (t 1) + 1 ( x pbest i xi (t )) + 2 ( x gbest xi (t ))
(10)
Where 1 and 2 are random variables. The
second term above is referred to as the cognitive
component, while the last term is the social
component.
6. Move each particle to a new position:
G G G
xi (t ) = xi (t 1) + vi (t ) (11)
t = t +1
7. Go to step 2, and repeat until convergence.
The random variables 1 and 2 are defined
as 1 = r1C1 and 2 = r2 C 2 , with r1 , r2 ~ U (0,1) ,
and C1 and C 2 are positive acceleration constant.
Kennedy has studied the effects of the random
variables 1 and 2 on the particle's trajectories.
He asserted that C1 + C 2 4 guarantees the
stability of PSO [16].
B. Modified PSO Algorithm
In this approach we remove worst particle in
population and replace it by new particle. This is
important that how to determine worst particle
and how to generate new particle for current
population.
The particle is selected with worst local best
value at generation. Then we randomly choose
two particles from the population and use
crossover operator to generate two new particles.
Then select the best one from two newly
generated particles and the selected particle and
replace the worse particle with it.
In other hand we combine GA operator and
PSO algorithm to modify it .This modification
3URFHHGLQJVRIWKHWK0HGLWHUUDQHDQ&RQIHUHQFHRQ 7
&RQWURO $XWRPDWLRQ-XO\$WKHQV*UHHFH

IV. LEARNING BY PSO k

sin( ) 0 < k < 250
25
In this section, the way PSO employed for
1.0 250 k < 500
updating the ANFIS parameters is explained. (14)

The ANFIS has two types of parameters which u ( k ) = 1.0 500 k < 750
need training, the antecedent part parameters and
0.3 sin( k ) + 0.1 sin( k )
the conclusion part parameters. The membership 25 25
functions are assumed Gaussian as in equation k
(5), and their parameters are {ai , bi , ci }, where a i + 0.6 sin( ). 750 k < 1000
10
is the variance of membership functions and c i
is the center of MFs. Also bi is a trainable Fig.2 (a) shows results using the ANFIS and
PSO for identification (solid line: actual system
parameter. The parameters of conclusion part are output, the dotted line: the ANFIS and PSO
trained and here are represented with {pi , qi , ri }. result). Fig.2 (b) Present the MSEs of modified
algorithm and Fig.2 (c) Present the MSEs of
A. How to Apply PSO for Training ANFIS basic algorithm.
parameters The parameters for training the ANFIS with this
There are 3 sets of trainable parameters in PSO algorithm listed in Table I.
antecedent part {ai , bi , ci } , each of these Example 2: Identification of nonlinear systems.
parameters has N genes. Where, N represents the We consider here the problem of identifying a
number of MFs. The conclusion parts parameters nonlinear system which was considered in [18].
( {pi , qi , ri }) also are trained during optimization A brief description is as follows. The system
model is
algorithm. Each chromosome in conclusion part
has ( I + 1) R genes that R is equal to
y (k +1) = f (y (k ), y (k 1)) +u (k ) (15)
Number of rules and I denotes dimension of data
inputs. For example each chromosome in
conclusion part in fig (1) has 6 genes. The fitness Where
is defined as mean square error (MSE).
Parameters are initialized randomly in first step x x ( x + 2.5 )
and then are being updated using PSO f (x 1, x 2 ) = 1 2 1 (16)
algorithms. In each iteration, one of the 1 + x 12 + x 22
parameters set are being updated. i.e. in first
iteration for example ai s are updated then in
second iteration bi are updated and then after Fig.3 (a) shows results using the ANFIS and
updating all parameters again the first parameter PSO for identification (solid line: actual system
update is considered and so on. output, the dotted line: the ANFIS and PSO
result). Fig.3 (b) Present the MSEs of this
B. Nonlinear Function Modeling algorithm and Fig.3 (c) Present the MSEs of
Example 1: Identification of a nonlinear dynamic basic algorithm.
system. In this example, the nonlinear plant with
multiple time-delays is described as [18] The parameters for training the ANFIS neural
network with this PSO algorithm listed in Table
y p ( k + 1) = f ( y p ( k ), y p ( k 1),
II.
(12)
y p ( k 2), u ( k ), u ( k 1))
TABLE I: PARAMETERS FOR EXAMPEL I

Where
NO of input 5
f (x 1, x 2 , x 3 , x 4 , x 5 ) = NO Of Data for train 1000
(13) NO MFs for each inputs 2
x 1 x 2 x 3 x 5 ( x 3 1) + x 4
15
1 + x 22 + x 32 NO of particle for each population

Here, the current output of the plant depends on Epoch for each population 100
three previous outputs and two previous inputs.
Testing input signal used is as the following:
3URFHHGLQJVRIWKHWK0HGLWHUUDQHDQ&RQIHUHQFHRQ 7
&RQWURO $XWRPDWLRQ-XO\$WKHQV*UHHFH

(a) (a)

(b) (b)

MSE=7.0020.e-004 MSE=0.0905

(c)
(c)

MSE=0.197

Figure 2: Using PSO for training parameters in ANFIS
structure for example1. Simulation results for nonlinear
system identification. (a) (dashed line), and actual system
output (solid line). (b) MSEs for modified algorithm.(c)
MSEs for basic algorithm
Figure 3: Using PSO for training parameters in ANFIS
structure for example2.. Simulation results for nonlinear
system identification. (a) (dashed line), and actual system
output (solid line). (b) MSEs for modified algorithm.(c)
MSEs for basic algorithm.
3URFHHGLQJVRIWKHWK0HGLWHUUDQHDQ&RQIHUHQFHRQ
&RQWURO $XWRPDWLRQ-XO\$WKHQV*UHHFH
TABLE II: PARAMETERS FOR EXAMPE II
NO of input 3 Networks, and Soft Computing, Van Nostrand
NO Of Data for train 100
NO MFs for each inputs 2 [10] Manish Kumar, Devendra P. Garg Intelligent Learning
10
NO of particle for each population
Epoch for each population 100
V. CONCLUSIONS
In this paper, we proposed a novel method for
training the parameters of ANFIS structure. In
our novel method we used PSO for updating the
parameters. The PSO which we used has some
differences with the usual PSO.
The simulation results showed the new approach
has better results for complex nonlinear systems
than basic PSO. Since this algorithm is free of
derivation which is very difficult to calculate for
training of antecedent part parameters
complexity of this new approach is less than
other training algorithms like GD and LS. In the
other hand the number of computations required
by each algorithm has shown that PSO requires less
to achieve the same error goal as with the
Backpropagation [19]. Also, the local minimum
problem in GD algorithm for training in this
novel approach is solved. The effectiveness of
the proposed PSO method was shown by
applying it to identification of nonlinear model.
REFERENCES
[1] Mannle M, Identify rule- base TSK fuzzy method, uni
of Karsruhe, 2000.
[2] Mannle M, Richard A, and Dorasm T. A, Rule- based
fuzzy model for nonlinear system identification, uni of
Karsruhe, 1996.
[3] Jang J. R, Sun C, and Mizutani, Neuro- Fuzzy and soft
computing, prentice hall, 1997.
[4] Sugeno M, and Kang G. T, Structure identification of
fuzzy model, fuzzy sets and systems, pp. 15- 33, 1998.
[5] Takagi T, Sugeno M, Fuzzy identification of systems
and its application to modeling and control, IEEE
Transaction on systems, Man & Cybernetics, pp: 116- 1321,
1985.
[6] Alcala R, Casillas J, Cordon O and Herrera F, Learning
TSK rule- based system from approximate ones by mean of
MOGUL methodology Granda uni of spain, Oct 2000.
[7] Mannle M, FTSM: Fast Takagi- Sugeno fuzzy
modeling, uni of Karsruhe, 1999.
7
[8] Jyh-Shing Roger Jang, ANFIS: Adaptive-Network-
Based Fuzzy Inference System, IEEE Trans. Sys.,Man and
Cybernetics., Vol. 23, No. 3, May/June 1993.
[9] 13. Yager, R. R. and Zadeh, L. A., Fuzzy Sets Neural
Reinhold,1994.
of Fuzzy Logic Controllers via Neural Network and Genetic
Algorithm, Proceedings of 2004 JUSFA 2004 Japan USA
Symposium on Flexible Automation Denver, Colorado, July
19-21, 2004.
[11] Mascioli, F.M., Varazi, G.M. and Martinelli, G.,
Constructive Algorithm for Neuro-Fuzzy Networks,
Proceedings of the Sixth IEEE International Conference on
Fuzzy Systems, 1997, Vol. 1, July 1997, pp. 459 464.
[12] Jang, J.-S. R., and Mizutani, E., Levenberg-Marquardt
Method for ANFIS Learning, Biennial Conference of the
North American Fuzzy Information Processing Society, June
1996, pp. 87 91.
[13] Jang, J.-S.R., Input Selection for ANFIS Learning,
Proceedings of the Fifth IEEE International Conference on
Fuzzy Systems, Vol. 2, Sep 1996, pp. 1493 1499.
[14] J. kennedy, RC. Ebenhart,, particle swarm
optimization, Proceedings of the IEEE International
Conference on neural networks, Vol. 4, 1995, pp. 1942
1948.
[15] Andries P.Engelbrecht, Computational Intelligence An
introduction, John Wiley & Sons Ltd 2002.
[16] J. kennedy , the behavior of particle swarm in VW, N
saravan, D Waagen (eds), proceeding of 7th international
conference on evolutionary programming, 1998, pp581-589.
[17] Y Shi, RC Ebenhart, Empirical study of Particle Swarm
Optimization, proceeding of the IEEE Congress on
Evolutionary Computation, Vol3, 1999, pp 1945-1950.
[18] K. S. Narendra and K. Parthasarathy, Identification and
control of dynamical system using neural networks, IEEE
Trans. Neural Networks,vol. 1, pp. 427, Jan. 1990.
[19] Venu G. Gudise and Ganesh K. Venayagamoorthy ,
Comparison of Particle Swarm Optimization and
Backpropagation as Training Algorithms for Neural
Networks , Swarm Intelligence Symposium, 2003. SIS '03.
Proceedings of the 24-26 April 2003 IEEE.