Published in National Conference & workshop on High Performance Computing & Application, BHU, Varanasi, Feb 2009

A Framework for the study of Fault Tolerance of Hardware

Model of Artificial Neural Network
Amit Prakash Singh, Pravin Chandra, C. S. Rai
University School of Information Technology
GGS Indraprastha University, Delhi – 110 403 [India]
Aps.ipu@gmail.com

Abstract
Artificial Neural Network is inherently property of Fault Tolerance. Fault Tolerance
property of ANN has been investigated with reference to hardware model of ANN. We
have proposed a framework for the investigation of fault tolerance property of hardware
model of ANN in this paper. Fault model has been given for the investigation of fault
tolerance property of ANN. It is proposed that various model of ANN should be taken
and apply the fault model to each of them to study the property of fault tolerance.

1. Introduction

An Artificial Neural Network (ANN) is an information-processing paradigm that is

inspired by the biological nervous system, i.e. the human brain. The key element of this
paradigm is the novel structure of the information processing system. It is composed of a
large number of highly interconnected parallel processing elements (neurons) working
together to solve specific problems. An ANN is configured for specific applications, such
as pattern recognition [2] or data classification, through a learning process.
Artificial Neural Network (ANN) has potential of the parallel processing by the
integrated circuit technology. Over one million gates are available by the latest FPGA
(Field Programmable Gate Array) for the implementation of Artificial Neural Network
[1]. The most common architecture in ANNs consists of multi-layer perceptrons trained
using an error backpropagation algorithm.(Fig. 1)
 Published in National Conference & workshop on High Performance Computing & Application, BHU, Varanasi, Feb 2009

In the case of biological neural networks a solution, tolerant to loss of neurons has a high
priority since a graceful degradation of performance is very important to the survival of
the organism.
2. Hardware Model of Artificial Neural Network
Hardware model of Artificial Neural Network has been widely implemented by various
researchers for application like image processing and controller based application. In
recent years, many approaches have been proposed for the implementation of different
types of neural networks, such as Multilayer Perceptron, Boltzmann Machine and other
as hardware devices.
With the widespread usage of the neural hardware as controller of various devices, it has
become imperative to study the behavior of these circuits under various faults, i.e. the
study of their fault tolerance behavior must be undertaken.
3. Characteristics of Fault-Tolerance of Artificial Neural Network
The fault/error tolerances of networks to the following fault/errors are considered:
1) Single weight stuck at zero
2) Single hidden layer node output stuck at zero
3) Single hidden layer node stuck at 1.
3.1 Fault Models
The fault models which we will study are applicable to FFANN’s that are already
trained[29]. The formulation of the FFANN system as a learning system allows us to
segregate the study of fault-tolerance behavior of the FFANN into two separate sets,
namely:
1) Tolerance to faults/errors outside the adaptive machine
(external/environmental faults/errors), and
2) Tolerance to faults/errors in the structure of the adaptive machine
(internal/architectural/structural faults/errors).

The External and internal Faults are defined as follows:

Definition 3.1: Any incorrectness in the input to the adaptive machine is defined as an
external fault.
3.2 External Fault Type 1 (EFTI): This is the error induced because some components
of the input vectors are missing from the input data or, equivalently, we may treat the
missing component as a stuck at zero faults.
It is felt that for the case of dependency of the output on the input component that is
missing, the output error will be very large for the case where the number of inputs to the
FFANN is relatively small. But, for the case where the dimensionality of the input vector
is relatively large, the error induced because of missing components should be
investigated and methods found to mitigate the error induced. This fault has not been
investigated in literature.
3.3 External Fault Type II (EFTII): Input values stuck at a constant value (the
different values that may be considered are – absolute maximum value in the input, the
absolute minimum value of the input, the negative of the absolute maximum or the
minimum value, the simple maximum and the minimum values). ± maximum or
minimum of the corresponding input components. The case of the input component stuck
at zero is not considered because it is equivalent to case (EFTI) considered in the above
section. For this case also, an investigation should be conducted in the case of large input
 Published in National Conference & workshop on High Performance Computing & Application, BHU, Varanasi, Feb 2009

networks only, as for them only a non-catastrophic output may be expected from the
biological analogy.
3.4 External Fault Type III (EFTIII): White noise in the inputs, with the noise variance
for the different components being different, in general. The noise presence may be due
to measurement errors, modeling errors or data representation granularity. This fault/error
should be investigated irrespective of whether the number of inputs is small or large. This
error investigation is also an investigation of the error propagation through the network,
the investigation of this error will allow us to comment on the stability of the FFANN as
a computational mechanism. This fault has been investigated by Input noise introduction
during training, has been recognized to provide a regularization effect, this effect should
be measured to see whether this leads to a more robust network or not.
4. Architectural Faults
To understand what may comprise an internal fault, one has to look at the two types of
FFANN architecture that may be thought of as the extreme end of an architectural
spectrum represents a network in which the layer nodes are connected to adjacent layer
nodes only (Type–I architecture, Fig. 1) while fig. 2 represents a network in which layer
nodes are connected to all subsequent layer nodes (Type–II architecture). There can be
many variants in between these two extremes (like FFANN’s in which nodes of a layer
are connected to only a few nodes of preceding or succeeding layer nodes, etc.).

To enumerate the types of architectural faults / errors that can occur, one may define:
Definition 4.1: A FFANN is a 7-tuple {I,L,M, S,N,O,W}, where I denotes the indexed
set of input nodes, L is an indexed set of layers (ordered left to right, and the the first
layer is taken as the first hidden layer while the the last layer corresponds to the hidden
layer just before the output layer), M denotes the cardinality of the set L, S is an indexed
set of elements Si, i ∈ {1,.........M } which denotes the size of the ith layer of the network, N
is an doubly indexed set whose elements N ij (which denotes the jth node of the ith layer,
(i ∈ {1,........, M } and j ∈ {1,........., Si } ) are a tuple of the form {A, σ} specifying the
input aggregation rule A and the activation rule σ, O is the set of output nodes, while W
is also an indexed set representing the weights of the network.
With this characterization of a FFANN, the study of architectural fault– tolerance reduces
to the investigation of the tolerance to faults of L, M, S, N, O, and W.
5. Classification of Faults
 Published in National Conference & workshop on High Performance Computing & Application, BHU, Varanasi, Feb 2009

Faulty Weights and neurons: The removal of the interconnection weights in a network
and the occurrence of stuck-at faults in neurons and the occurrence of stuck-at fault in
neurons are two types of faults that can serve as a test bed for the robustness of neural
networks. The robustness of a backpropagation trained multilayer network to removing
weights to/from the hidden layer and the influence of redundancy in the form of excess
hidden neurons has been investigated in [4]. The effect of “Stuck-at-0” and “stuck-at-1”
neurons on the solutions found in recurrent optimization networks is investigated in [5].
The faults/errors may be classified as:
5.1 Layer Faults: These are the errors that affect hidden layer(s) at the same time.
Following types of errors are defined:
(a) Layer Stuck at Zero (LSZ):
All nodes in a particular layer produce an output equal to zero.
(b) Layer Stuck at One (LSO):
All nodes in a particular produce an output equal to ±1 (assuming that the tangent
hyperbolic function, f = tanh(x) is used as the activation function), if any other activation
is used, then these two values should be modified to reflect the two limiting values of the
activation function used and moreover these faults are relevant only if at least in each
layer the activation functions are the same).
(c) White Layer Noise (WLN):
This is the presence of additive white noise that is correlated across the layer
outputs. This may happen because of voltage and/or current fluctuation affecting a
complete layer.
Due to the architectural peculiarities of the FFANN architecture, some of these faults are
catastrophic for the Type–I architecture (LSZ and LSO) therefore investigations should
be limited to Type–II architectures only. The third fault (LN) should be investigated for
all types of networks.
5.2 Hidden Node Faults: These are the errors that affect hidden nodes. Following types
of errors are defined:
(a) Node Stuck at Zero (NSZ):
The node output is stuck at zero.
(b) Node Stuck at One (NSO):
Nodes produce an output equal to ±1 (assuming that the tangent hyperbolic
function, f = tanh(x) is used as the activation function), if any other activation is used,
then these two values should be modified to reflect the two limiting values of the
activation function used.
(c) White Noise in Node (WNN):
The presence of additive Gaussian noise in the nodes’ output with zero mean and
fixed standard deviation. Thermal noise (then it can be parameterized by the temperature)
and degradation of circuitry in time can lead to this type of noise in a FFANN hardware
implementation.
5.3 Weight Faults: These are the faults / errors that affect the weights of the network.
Following types of faults / errors are defined:
(a) Weight Stuck at Zero (WSZ):
This fault corresponds to an open fault or connection breakage between two
nodes.
(b) Weight Stuck at Maximum / Minimum (WSMa/WSMi):
 Published in National Conference & workshop on High Performance Computing & Application, BHU, Varanasi, Feb 2009

Weight stuck at a value of ±|W|max , where |W|max is the maximum magnitude

weight in the system. A -ve weight will be pushed to −|W|max while a +ve weight will be
pushed towards +|W|max and vice-versa.
This allows us to model weight faults at substantially large values, which may or may not
lead to node hard faults of the type NSZ or NSO depending on the weight interaction
with the other weights leading to the same node as the faulted weight.
(c) White Noise in Weights (WNW):
The presence of white noise (zero mean Gaussian with finite variance) may be
taken as a reflection of thermal noise or circuit degradation. This noise is different from
node output noise as it is not correlated in weights leading from the same node.
5.4 Output Node Faults: The output node faults can be described in an analogous
manner to the input node faults but only the noise contamination effects are non-
catastrophic and may be considered.
6. Measurement of Faults and Future Works
To measure the effect of faults/errors enumerated above, the following types of
error/fault/parameter sensitivity may be defined:
1. MSE, MAPE and Other Global Measures:
The mean squared error (MSE) and the mean absolute percentage error (MAPE)
should be used to measure the effect of all types of faults if the output of the FFANN
is real. In this case the median of the error may also be reported; while for
classification problems, the percentage of misclassification is suggested as a measure
of fault/errors. A measure will be done to determine the fault-tolerance property on
weights whose fault lead to errors larger than a prescribed value.
2. Experimental Verification:
Fault that implies a more effect on the network will be identified and distributed over
the network, so that effect of the fault will be reduced. Theoretical studies have been
made by various researchers, Experimental verification will be done.
The experimental results of [37] shows that FFANNs are not fault tolerant by default.
Appropriate training is required for the fault tolerant ANN. Further experiments will
be made to validate the definition given in [37] to make Fault tolerant ANN.

7. References
[1]. Naoya Chujo, Susumu Kuroyanagi, Shinji Doki and Shigeru Okuma, “An Iterative
Calculation Method of the Neuron Model for Hardware Implementation”, IEEE 2000
[2]. Seong-Whan lee, Hee-Heon Song, “A new recurrent neural-network architecture for visual
pattern recognition”, IEEE Transaction on Neural Networks, vol.8, pp. 331-340, March
1997
[3]. Zhu Jihan and Peter Sutton, “FPGA Implementation of Neural Networks- A Survey of a
Decade of Progress”, Proceedings of 13th International Conference on Field Programmable
Logic and Applications, (Lisbon, Portugal September 2003), 1062-1066.
[4]. M.D. Emmerson and R. I. Damper, “Determining and Improving the Fault Tolerance of
Multilayer Perceptrons in a Pattern-Recognition Application”, IEEE Transactions on Neural
Networks, Vol. 4, no. 5, pp 788-793, September 1993
[5]. P.W. Protzel, D. L. paumbo and M. K. Arras, “Performance and Fault-Tolerance of Neural
Networks for Optimization”, IEEE Transaction on Neural Networks, vol. 4, no.4 pp. 600-
614, July 1993
 Published in National Conference & workshop on High Performance Computing & Application, BHU, Varanasi, Feb 2009

[6]. J.N.H. Heemskerk, “Overview of Neural Hardware”, Neurocomputer for Brain Style
Processing: Design, Implementation and Application, PhD thesis, chapter 3, Unit of
Experimental and Theoretical Psychology, Leiden Univ., The Netherlands, 1995
[7]. Radek Holota, “Neural Network with MIN/MAX Nodes for Image Recognition and Its
Implementation in Programmable Logic Devices”, In: Proceedings of IEEE International
Conference on Computational Cybernetics, Siófok, Hungary, 2003, ISBN 963-7154-18-3.
[8]. Timothy G.W. Gordon and Peter J. Bentley, “On Evolvable Hardware”, Soft Computing in
Industrial Electronics. Physica-Verlag, Heidelberg, Germany, pp. 279-323.
[9]. Aoyama, T.; Qianyi Wang; Suematsu, R.; Shimizu, R.; Nagashima, U, “Learning
algorithms for a neural network in FPGA”,Proceedings of the 2002 International Joint
Conference on Volume 1, 12-17 May 2002 Page(s):1007 – 1012
[10]. Andres Upegui, Carlos Andres Pena-Reyes, Eduardo Sanchez, “A Functional Spiking
neuron hardware oriented model”, International conference on Artificial & Natural Neural
Networks, Vol. 2686, June 2003, pp 136-143.
[11]. Ponca M and G. Scarbata, “ Implementation of Artificial Neurons Using Programmable
Hardware”, Synopsys User Group Conference-SNUG Europe 2001, March, 2001.
[12]. L. Spaanenburg, R. Alberts, C.H. Slump, and B.J. VanderZwaag, “Natural learning of
neural networks by reconfiguration”, SPIE Proceedings on Bioengineered and Bioinspired
Systems, SPIE Conference on Bioengineered and Bioinspired Systems, Maspalomas, Gran
Canaria, Vol. 5119, pp. 273-284, 2003-05.
[13]. Leslie S. Smith, “Implementing neural models in silicon”, Handbook of Nature-Inspired
and Innovative Computing: Integrating Classical Models with Emerging Technologies, ed
A. Zomaya, Springer US, 2006, pp433-475
[14]. Bernard Girau, “On-chip learning of FPGA-inspired neural nets”, International Joint
Conference on Neural Networks, vol. 1, July 2001, pp222-227
[15]. Carlo H. Sequin and Reed D. Clay, “Fault Tolerance in Artificial Neural Network”,
Technical Report TR-90-031, International Computer Science Institute, UE Berkely, CA.
July 1990
[16]. Nait Charif Hammadi and Hideo Ito, “A Learning Algorithm for Fault Tolerant
Feedforward Neural Networks”, IEICE Trans. Information and Systems, vol. E80-D, no.1,
pp.21-27, 1997.
[17]. Kunihit Yamamori, Susmu Horiguchi, J.H. Kim, Sung-K. Park and Byung H. Ham, “The
Efficient Design of Fault-Tolerant Artificial Neural Networks”, IEEE International
Conference on Neural Networks, Vol. 3, pp. 1487-1491, Dec. 1995.
[18]. Ching-Tai Chiu, Mehrotra K., Mohan C.K. , Rankat S., “Training Techniques to obtain
fault-tolerant neural networks”, Fault-Tolerant Computing, FTCS-24,pp360-369, 1994
[19]. Buh Yun Sher and Weng-Shong Hsieh, “Fault tolerance Training of Feedforward Neural
Networks”, Proceedings of the National Science Council, Republic of China, Vol.23, No. 5,
pp599-608, 1999
[20]. George Bolt, “Investigating Fault Tolerance in Artificial Neural Networks”, Technical
Report YCS-154, University of York, Department of Computer Science, 1991
[21]. Chung-Hsing Chen, Lon-Chan Chu and Daniel G. Saab, “Reconfigurable Fault Tolerant
Neural Network”, International Joint Conference on Neural Networks, Vo. 2, pp. 547-552,
June 1992
[22]. Yutaka Maeda and Masatoshi Wakamura, “Simulataneous Perturbation Learning Rule for
Recurrent Neural Networks and Its FPGA Implementation”, IEEE Transaction on Neural
Networks, Vol. 16, No. 6, November 2005.
[23]. Behnam S. Arad and Ahmed El-Amawy, “On Fault Tolerant Training of Feedforward
Neural Networks”, Neural Networks, Vol. 10, No. 3, pp 539-553, 1997
 Published in National Conference & workshop on High Performance Computing & Application, BHU, Varanasi, Feb 2009

[24]. M.T. Tommiska, “Efficient digital implementation of the sigmoid function for
reprogrammable logic”, IEE Proc. Comput. Digit. Tech. Vol. 150, No. 6, pp. 403-
411,November 2003
[25]. J. Zhu and Peter Sutton, “An FPGA Implementation of Kak’s Instantaneously-Trained,
Fast-Classification Neural Networks”, IEEE International Conference on Field-
Programmable Technology (FPT), pp 126-133, Dec 2003
[26]. Hamed Elsimary, Ahmed Darwish, Samia Mashali and Samir Shaheen, “Performance
Evaluation of a Novel Fault Tolerance Training Algorithm”, IEEE World Congress on
Computational Intelligence, pp.856-861, vol.2, July 1994.
[27]. Kunihito Yamomori, Susumu Horiguchi, J.H. Kim, Sung-K Park and Byung H. Ham, “The
Efficient Design of Fault-Tolerant Artificial Neural Networks”, IEEE International
Conference on Neural Networks, Vol. 3, pp. 1487-1491, Dec. 1995.
[28]. Fabian Vargas, Djones Lettnin, Diogo Brum and Darcio Prestes, “A New Learning
Approach to Design Fault Tolerant ANNs: Finally a zero HW-SW Overhead”, Proceedings
of the 11th Asian Test Symposium, 2002
[29]. D.S. Phatak and I. Koren, “Complete and Partial Fault Tolerance of Feedforward Neural
Nets” IEEE Transaction on Neural Networks, vol. 6, issue 2, pp. 446-456, March, 1995.
[30]. Nazeih M. Botros and M. Abdul-Azia, “Hardware Implementation of an Artificial Neural
Network Using Field Programmable Gate Arrays (FPGA’s)”, IEEE Transaction on
Industrial Electronics, vol. 41, issue 6, pp. 665-667, Dec 1994.
[31]. Dhananjay S. Phatak and Elko Tchernev, “Synthesis of Fault Tolerant Neural Networks”,
International Joint Conference on Neural Network, Vol. 2, pp. 1475-1480, May 2002
[32]. Vincenzo Piuri, “Analysis of Fault Tolerance in Artificial Neural Networks”, Journal of
Parallel and Distributed Computing, pp. 18-48, 2001
[33]. Dhananjay S. Phatak, “Fault Tolerant Artificial Neural Networks”, Proceedings of the 5th
Dual Use Technologies and Applications Conference, May, 1995.
[34]. Behnam S. Arad and Ahmed El-Amawy, “On Fault Tolerant Training of Feedforward
Neural Networks”, Neural Networks, Vol. 10, No. 3, pp. 539-553, 1997.
[35]. James M. Avery, “Neural Network Development Using VHDL”, IEEE Seminar on ASIC,
pp. P24/1-4, Sept 1989.
[36]. A. Muthuramalingam, S. Mhiavathi and E. Srinivasan, “Neural Network Implementation
Using FPGA: Issues and Application”, International Journal of Information Technology,
Vol. 4, No. 2, 2007
[37]. Pravin Chandra and Yogesh Singh, “Feedforward Sigmoidal Networks-Equicontinuity and
Fault-Tolerance Properties”, IEEE Transaction on Neural Networks, Vol. 15, No. 6,
November 2004
[38]. Pravin Chandra and Yogesh Singh, “Fault Tolerance of Feedforward Artificial Neural
Networks - A Framework of Study”, Proceedings of the International Joint Conference on
Neural Networks, Vol.1, pp.20-24, July 2003.