4.3 Stochastic Models
4.3.2 Centralised Stochastic Models
In a centralised stochastic neuron model based on the general architecture of figure 4.1, synapses and dendrite structure are only responsible for input current integration and all the non-linear dynamics of the neuron is centralised in the soma unit. In such a model, each synapse unit can add some pulses to a bitstream according to its synaptic weight. The soma unit then generates the updated membrane potential signal in response to the arriving stochastic pulses and sends a copy conveying the updated potential to all the synapse units. Therefore, soma unit should contain all the parametric non-linear stochastic operators that can be evolved to change the behaviour of the neuron.
Syn
Soma
Syn
Syn
Syn
Syn
Syn
Pre
-syn
a
p
tic
in
p
u
ts
Axon
W
Soma
Axon
RND
Universal Stochastic Function Generatorm
Stochastic to Binary convertern
Spike Generation reset ut ut+Δt aut2+(1+b)ut+cFigure 4.3: Left: General design of a centralised stochastic model. Right: Internal structure of the soma and synapse units. RND represents a source of random bitstreams with specific probabilities.
To investigate the challenges of centralised stochastic models, here again, a one-dimensional quadratic neuron is designed using stochastic computing techniques. Figure 4.3 shows the general de- sign of a centralised stochastic model with a dendritic ring that conveys the stochastic bitstreams (with probability au2
t+ (1 + b)ut+ c) to synapse units. Every time that a synapse unit (bottom right of figure
4.3) receives a presynaptic input spike it adds the synaptic weight (a short bitstream with probability of wstored in a shift register or generated by a digital to stochastic converter) to this upward stochastic bitstream and passes it to the next synapse unit. The soma unit (top right in figure 4.3) receives the total membrane potential including postsynaptic currents (ut+∆ = au2t+ (1 + b)ut+ c + I) and calculates
the new upward bitstream. The soma unit is designed based on a stochastic-to-digital converter [50], a universal stochastic function generator, a spike generation module and a source of random bitstreams
with constant probabilities (RND in figure 4.3). 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 x f(x) f(x)=x Stochastic f(x)=3.6*x2−1.2*x+0.156 Parameter points
Figure 4.4: Simulation results of a 16x4-bit stochastic function generator with parameters tuned to approximate the func- tion f(x) = 3.6x2− 1.2x + 0.156 (shown in green). The red curve shows the probability of the function generator out-
put bitstream against the input value (x). Intersection points of the blue line (f (x) = x) and the update function determine the resting (x= 0.1) and threshold (x = 0.5) potentials respectively.
The stochastic to digital converter is mainly an up/down n-bit saturating binary counter that counts the ones up and the zeros down in the bitstream and estimates the probability of 1s in the bitstream as a n-bit binary number. This is the binary representation of the current membrane potential. After an action potential that is detected by spike generation module the counter will be reset to a pre-specified value (reset potential) and a spike pulse will be sent to the output. The universal stochastic function generator is essentially a lookup table of 2n values of 2m-bit binary number. It uses the same principle
of digital to stochastic converters presented in [50] and used in [391, 14]. However, in the function generator each n-bit value of the membrane potential will address a register in the lookup table that contain the corresponding value of the function. The other address lines of the lookup table are connected to m random bitstreams with relative probabilities proportional to the weights of the binary digits in the register so that the total probability of these m bitstreams add up to one. It is therefore possible to tune the parameters in the lookup table to generate any arbitrary function of the input. Figure 4.4 shows the simulation results of an example stochastic function generator with a 16x4-bit lookup table with parameters tuned to approximate a quadratic function that can be used in a quadratic neuron model. The red curve shows the probability of the function generator output bitstream versus the input binary value. The green curve is the reference quadratic function. Intersection points of the blue line with
these curves about x = 0.1 and x = 0.5 determine the resting and threshold potentials respectively. The probability of the bitstream is calculated by counting 10,000 bits. The simulation demonstrates that very close approximations of such functions can be achieved by tuning a set of parameters. Parameters loosely work as control points that pull the function curve up and down at different values of x.
Although this is a very simple and general model it allows many different extensions and improve- ments. Each synapse unit has access to both the presynaptic spike and the current partial and total membrane potential (as upward and downward stochastic bitstreams) to be used for adding synaptic plasticity mechanism for learning, short-term plasticity, etc. It is also possible to add a second variable and equation for the slow dynamics of the soma.
This simple model can be implemented using Virtex-5 LUTs, shift registers, and flip-flops. The n- bit converter can be implemented using n flip-flops, a shift register (1 6LUT), and n/2 6LUTs virtex-5 primitives. The function generator can be implemented with 2m+n−66LUTs configured together as a
2m+n-bit RAM. The random bitstreams with constant probabilities can be generated globally and used
in all modules. The total hardware needed for the soma unit can be estimated to need 2m+n−6+ n/2
6LUTs and n + 1 flip-flops.
To calculate the noise in the system we start with the quantisation error of the binary representation in the stochastic to digital converter. The SNR (Signal-to-Noise Ratio) of a n-bit binary representation is equal to:
SN RdB = 20 log10(2n) ≈ 6.02 · n dB (4.8)
Therefore, for achieving the 42dB SNR of the biological pyramidal neuron, such a centralised stochastic neuron model needs at least n = 7 bits. Even with a minimum of m = 2, necessary to achieve the needed accuracy for the function generator, a soma unit at least needs 12.5 6LUT primitives and eight flip-flops. That is roughly equal to four Virtex-5 slices. It would be also possible to compactly implement a synapse units and a very simple flexible learning mechanism in a minimum of half a Virtex-5 SLICEM. Another source of noise in such a system is the inherent statistical variation in counting stochastic bitstreams [50]. The coefficient of variation of a Bernoulli bitstream with probability p is:
CV = σ/µ =r 1 − p
np (4.9)
where n is the number of counted bits [50]. Then, SNR can be calculated as:
SN RdB = 20 log10(1/CV ) = 20 log10(µ/σ) = 10 log10(
np
1 − p) dB (4.10)
This value depends on the probability of the bitstream as well. Counting n = 128 bits that is equal to the capacity of the 7-bit counter of the stochastic to digital converter gives a maximum of about 21dB for p = 0.5and 11.5dB for p = 0.1. This is a tighter bottleneck on the accuracy of the system than the noise due to the digital quantisation. This means to achieve 42dB SNR of the biological neuron, even only at p = 0.5, the soma needs to count over 16000 (214) bits. This was also confirmed through simulations
that showed for an acceptable noise level in signal estimation, well over 8000 stochastic bits must be counted.
This affects both the speed and hardware efficiency of the model since counting 214 bits needs at
least 214clock cycles and a 14-bit counter. To mitigate the impact of increasing accuracy on the hardware
efficiency, it is possible to increase only the number of bits in the counter and use the higher order bits for the same small function generator. However, even accepting a higher noise level cannot significantly improve the speed as noise level is of the order of O(1/√n)where n is length of the counted bitstream. Stochastic models have the advantage of hardware efficiency at the cost of lower speed, thus possi- bility of realising a large-scale network on FPGA. They are slower than their deterministic counterparts. However, very simple and low latency circuits allow them to achieve higher clock frequencies that may slightly mitigate this problem. Stochastic models are more immune to noise and when they achieve the required accuracy, they are more robust than their deterministic counterparts. Their reliance on sin- gle wire signals can minimise the FPGA routing resources needed for intracellular communications. Stochastic neuron models are also biologically more plausible than deterministic models as they can better model the internal noise and uncertainty of the biological neurons. Preliminary experiments and simulations showed that it is also possible to evolve different stochastic functions using LUTs in the FPGA giving a rather continuous range for parametric flexibility needed for evolvability of the neu- ron types and synaptic plasticity. Distributed models appear to have a better convergence speed than centralised stochastic models as they exploit many more PEs. With the right design, the inherent redun- dancy, modularity, and parallelism in distributed stochastic models can also lead to fault-tolerance. They can also model the distribution of the ion channels in space better than centralised models. This may prove useful in evolving local nonlinearities in the dendrite trees. The difference between centralised and distributed stochastic models shows a trade-off between hardware compactness and performance.
Although stochastic models can lead to very bio-plausible, simple, compact, robust, flexible, and evolvable neuron models, the fundamental trade-off between their reliability and efficiency may not lead to very efficient bio-plausible designs on FPGAs compared to deterministic models. Therefore, deterministic models are also investigated in the following section.