Square Wave Artificial Neuron (SWAN)

David Ellis
Computer Science Department
Eastern Washington University
Cheney, Washington, USA
mdlilaccottage@gmail.com
Kosuke Imamura
Computer Science Department
Eastern Washington University
Cheney, Washington, USA
kimamura@ewu.edu


Abstract----We developed a square wave based artificial
neuron to take advantage of inexpensive and readily available
Field Programmable Gate Array technologies. While
conventional neurons require computationally intensive floating
point arithmetic to determine the output, our artificial neuron
converts inputs into square waves and the time that these waves
require to produce a predetermined bit pattern is used to
determine the output value. The behavior of this neuron was
successfully simulated by software to demonstrate that the
Square Wave Artificial Neuron is software implementable.
Keywords--- FPGA, neural network, neuron, learning
I. INTRODUCTION
Neuron models are the basic building blocks of Neural
Networks (NN). The first NN theory was developed in 1873
by Alexander Bain [1]. This theory was based on brain tissue
and the fiber like interconnection between the individual
neurons forming the brain. Artificial Neural Networks (ANNs)
were developed based on this biological NN theory. The
purpose was to apply mathematical models to the functions of
the neurons as a way to understand the function of biological
brain structures. It is from this origin that Artificial Neurons
(AN) derive their terminology and mathematical complexity.
Our Square Wave Artificial Neuron (SWAN) is based on
binary logic and uses no floating point computation. Thus, this
neuron is implementable on a digital device without CPU.
Conventional neurons require computationally intensive
floating point arithmetic to determine the output [2].
Typically, a neuron consists of two functions: activation
function and output function. The activation function
processes the inputs generally as a summation function of
input signals. The output function receives the results of the
activation function and determines the neurons output value.
A neuron can be described with Equation 1 and 2 [2].
Equation 1: A = input i * weight i
Equation 2: output = F (A)
The proposed neuron has the same two function structure.
However, the SWAN activation function converts inputs into
square waves of frequencies which are determined by weights.
The output function is given a reference pattern and the time
that the input waves take to produce the given pattern
determines the output. This neuron was implemented on a
Field Programmable Gate Array (FPGA) and simulated by
software to prove its viability.
The next section reviews the literature on neurons. Section
III describes the proposed SWAN model. Section IV discusses
the implementation on an FPGA. Section V and VI are the
testing and conclusion respectively.
II. LITERATURE REVIEW
As stated previously, neuron models are the basic building
blocks of Neural Networks (NN) and have basic characteristics
in common. They provide for multiple inputs, have some type
of transfer function that is composed of an activation function
and have an output function.
The McColloch and Pitts model, Figure 1, was developed
in the 1940s and defined these basic characteristics. The inputs
to the neuron can have values either 0 or 1. The weights are
fractional values between 0 and 1. An activation value is
calculated using Equation 1 and the output function is a simple
step function. A fixed threshold value is used with the value
determined by the activation function to set the output to a
value of 0 or 1 [3].




Figure 1 McCulloch and Pitts Neuron
In 1958, Frank Rosenblatt developed the Perceptron
diagrammed in Figure 2. The Perceptron builds on the same
concepts as the McColloch and Pitts model. However, it uses
input and output values that range from 1 to -1, where the
negative values provide the inhibit function. The activation
function is essentially the same as in the McColloch and Pitts
model [3]. The output function tends to use either a linear or
sigmoid function along with a threshold value [4].
Beginning around 1969, Spike Neural models, Figure 3,
were being developed [5][6]. These models are far more
complex and attempt to emulate the biological neurons as
closely as possible. Early models such as those developed by
Hodgkin and Huxley were based in analog circuitry [7]. Later
models have been implemented in software or a combination of
ISBN: 978-1-4799-3724-0/14/$31.00 2014 IEEE 42
software and Field Programmable Gate Arrays (FPGA)
hardware [8].




Figure 2 Perceptron





Figure 3 Spike Neuron
Variations of these three AN models have been used to
implement neural networks with FPGAs. There are several
considerations when implementing these models in FPGAs.
First, the typical neuron model determines an activation value
based on the summation of the input and weight product
(Equation 1). This calculation is dependent on floating point
operations and there are Arithmetic Logic Unit (ALU) designs
and programming methods that can be used with current
FPGAs to support this level of calculation [9].
Calculating the activation function tends to be one of the
more difficult operations for FPGAs. The difficulty is not in
the calculating of the function but doing so at a reasonable
cost in both space and speed [9].
III. SQUARE WAVE ARTIFICIAL NEURON
The Square Wave Artificial Neuron (SWAN) converts each
input bit into a square wave the frequency of which is
determined by a given weight. A set of these square waves are
sampled to create a binary pattern. The sampled binary pattern
is compared with a given reference pattern and if they are
identical, the time it took since the sampling start time is
recorded. For example in Figure 4, the first sampled pattern at
time0 is 1011, the second at time1 is 0011, and the third at time
2 is 1010. At the fourth sampling cycle at time3, the sampled
pattern has become equal to the reference pattern, 0110. The
time3 is recorded and is referred to as the logical event time.
The output is 1, if the event time is smaller than a given
Threshold Time, and 0 otherwise.
The SWAN's output is a pulse with a width less than or
equal to time MAX.. The time is a logical time and thus
simply a system clock based counter. Once time MAX is
reached, the time cycles back to time0 and the next sampling
starts. The square wave sampling window is given by:
Equation 3: Sampling Window = MAX * (system clock /
clock divider).
The inputs to the SWAN are single bit values. The number
of inputs to the model can be from one to some number n. Each
of these inputs has an associated weight. The value of the
input and the value of its weight are used to generate an
oscillating square wave (Graph 1). The frequency of the input
bit's square wave is given by:
Equation 4: Input Frequency = System Clock Frequency
/ 2*weight
where: weight > 0
The input frequency encodes each input into a square wave
by the weight associated with the input. A set of input
frequencies creates various bit patterns when sampled at time N.












Figure 4 Square Wave Artificial Neuron
To establish an activation function, a Reference Pattern
(REF) is compared with the bit pattern created by sampling of
the square waves. The REF is randomly generated during the
training process. The logical event time is when the sampled
pattern has become equal to the reference pattern (Graph 1).








Graph 1 Generated Input Frequency Square Waves, REF, and the event time.
The sampled pattern is 0110 at the dotted line.
To provide the output function, a Threshold Time (t
TH
) is
assigned; time0 < t
TH
,< timeMAX. The neurons output is
determined by:
ISBN: 978-1-4799-3724-0/14/$31.00 2014 IEEE 43
if event time < t
TH
, then output = 1
if event time t
TH
, then output = 0
IV. IMPLEMENTATION
The hardware used to implement and test the SWAN was
the Terasic DE2-115 development board [10]. The software
was Alters' Quartus II version 13.1 Web Version [11]. We
used Verilog to implement our neuron.
Figure 5 is a block diagram of the timer module to
generate time 0 to time MAX. In our experiment, MAX is set
to 255 or 8 bit. The counter is reset and the reset pulse is
generated, when the count reached a value of 255.





Figure 5 Logical Time Module (time0 - time MAX)
Next, the Trip Weight Module (TwM) provides the Input
Frequency generator (Figure 6). The TwM is designed to
operate on a single one bit input, and the weight is an eight bit
wide value to match the size of the Logical Timer. Four of
these module are used to create the four input SWAN.






Figure 6 Trip Weight Module
An internal eight bit counter is driven by the system clock
and compared with the weight. When the two are equal the
tripOut variable is inverted using the controlled inverter. The
trip counter is reset and the comparison cycle repeated,
generating a square wave output via tripOut. The input pulse
tracker is used to sum the in signals phase with the tripOut.
Next is the Transform Event module (TeM) (Figure 7).
The TeM is used to detect when the input is transformed to
match the reference value. This module, like the Trip Weight
Module (TwM), acts on a single bit value and four of these are
used to create the four input SWAN.
The stored reference value is compared with the signal
applied to the in signal using a simple XNOR operation. The
in signal is from the TwM, and the use of the simple XNOR
provides dynamic monitoring of the in signal. When a match
is found, the TeM outputs a logic high pulse indicating the
match. This output pulse is equal to twice the system clock
width and is referred to as the foundFlag.




Figure 7 Transform Event Module
The last module is the Output Module (Figure 8). The
Output Module is used to generate the final output of the
neuron. This output is provided in two forms. The first form is
the output of a logic 1 or 0, which is also provided as a pulse.
The second output provides the event time in terms of seconds.
The logical event time is the counter value, which is captured
when the found-signal had changed to a logic high state. The
output of the axon is set or reset based on the logical event time
and a stored threshold value. If the logical event time is smaller
than the threshold value the axon output is a logical 1.
Otherwise, the axon output is logic 0.







Figure 8 Output Module
The SWAN can linearly scale as the number of input bits
by adding TwM/TeM pairs in parallel. For our experiment, we
designed a 4-input system as shown in Figure 9.













Figure 9 4 input SWAN
ISBN: 978-1-4799-3724-0/14/$31.00 2014 IEEE 44
V. TEST AND RESULTS
Our first experiment was to configure the SWAN based 4
bit system (Figure 9) to perform the XOR function. We used
two of the four input bits. The weights and reference are
randomly generated per training criterion. The event times
from the SWAN should satisfy the following criterion to
establish the threshold time.
Criterion: The maximum event time for the input data
set with an output value 1 must be less than four logical time
units of the minimum event time for the input data with an
output value 0.
This criterion is to separate the data into two
categories by the threshold time, output 1 and output 0. If the
above criterion is not met, a new set of random weights and
reference value will be generated and the training process is
repeated. When the criterion is met, the threshold time (t
TH
) is
calculated by the following formula:
Equation 5: t
TH
= ET1 + (ET0 ET1)/2
ET0: the minimum event time to output 1.
ET1: the maximum event time to output 0.
The SWAN became an XOR gate for in0 and in1 inputs
when the weights for in0 and in1 were 00001110 and the
reference value were 0001. And, the threshold logical time
was 10001100. There could be other choices of weights that
would also create an XOR function. The in2 and in3 inputs
had no effect on the axon output. This is because setting the
weight to 00000000 for an input inhibits the square wave,
preventing it from affecting the axon output.
We performed the second test keeping the weights and
threshold values the same and changing the reference value.
The SWAN became an XNOR gate for in0 and in1 inputs
when the reference value were 0011. And, the threshold
logical time was 10001100.
VI. CONCLUSION AND FUTUR WORK
Our experiment shows that the Square Wave Artificial
Neuron (SWAN), a neuron which has a time based transfer
function, is capable of creating effectively a reconfigurable
decoder by feeding training data. At the same time, any
reconfigurable hardware only with such capability is subject to
the limitation of combinatorial logic circuit evolution [12].
Thus, a challenge facing with the SWAN is how to
approximate continuous space. To do so, we are planning to
increase the number of bits for each input. Essentially, we
would extend our single bit input SWAN (Figure 9) to n bit
inputs to process one input data and duplicate that module m
times to process m n-bit input data. Also, the output will have
a multi-bit value. The expression (input i * weight i) in
Equation 1 can be approximated by assigning lower frequency
to more significant bits and higher frequency to less
significant bits. The SWANs transfer function is based on
linear time. Since the SWAN does not use back propagation,
the transfer function does not have to be sigmoid or even be
differentiable.
The next challenge is how fast we can find proper
threshold time and reference patterns. Current SWAN uses a
random algorithm to generated weight and reference values to
the inputs to produce the corresponding square waves. While
this will work efficiently when the search space is relatively
small, we need to a better algorithm for the weight and
reference assignment. We are considering machine learning
heuristics such as genetic algorithms and evolution strategies.
To conclude, we showed that the SWAN is a viable neuron
which does not rely on floating point arithmetic execution on a
CPU based computer. Instead, it converts inputs to square
waves and samples these waves to set the logical event time.
The SWAN was implemented on a readily available FPGA,
taking a full advantage of reconfigurable digital hardware.
REFERENCES
[1] A. Wilks, N. Wade, "Bain on Neural Networks", Brain and Cognition
33, 295-305, 1997..
[2] W. Duch, N. Jankowski, Survey of Neural Transfer Functions,
ftp://ftp.icsi.berkeley.edu/pub/ai/jagota/vol2_6.pdf.
[3] A. Jain, J. Mao, "Artificial Neural Networks: A Tutorial", Computer,
v.29 n.3, p.31-44, March 1996.
[4] Y. Liao, "Neural Networks in Hardware A Survey", tech. rep., Santa
Cruz, Ca, USA, 2001.
[5] D. Kunkle, C. Merrigan, "Pulsed Neural Networks and their
Application", Rochester Institute of Technology, Rochester, New York,
February 2002.
[6] E. Izhikevich, "Simple Model of Spiking Neurons", IEEE Transactions
on Neural Networks, VOL. 14 NO. 6, November 2003.
[7] E. Lewis, "Using Electronic Circuits to Model Simple Neuroelectric
Interactions", Proceedings of the IEEE, Vol. 56, No. 6, June 1968.
[8] L. Caron, F. Mailhot, J. Roust, "FPGA implementation of a spiking
neural network for pattern matching", Circuits and Systems (ISCAS),
2011.
[9] A. Omondi, J. Rajapakse, "FPGA implementation of Neural Networks",
2006; Springer-Verlag,
[10] "DE2-115 Users manual", Teasic Inc., 2010, www.terasic.com
[11] "Quartus II Handbook Version 13.0", Altera Inc., 101 Innovation Drive,
San Jose, CA, 20 www.altera.com.
[12] K. Imamura, J. A. Foster, A. W. Krings, "The Test Vector Problem and
Limitations to Evolving Digital Circuits", Proceedings of the 2nd
NASA/DoD workshop on Evolvable Hardware, IEEE Computer
Society, July 2000.

ISBN: 978-1-4799-3724-0/14/$31.00 2014 IEEE 45