The recurrent neural networks considered in this section can be described as follows: Consider a fully connected recurrent network of K LIF neurons, where K is the number of LIF neurons in the network. The recurrent network receives inputs from a series of pools of spiking input neurons, I1, I2, ..., Im. Each of the
spiking input pools of neurons consists of M input neurons, where, in all cases considered here, M = 1.
The recurrent network is divided into m sub-groups, with each sub-group con- sisting of k neurons. In addition to receiving input connections from all other recurrent network neurons, the neurons of each sub-group also receive input from
Basic Network Structure and Setup 187 one of the input groups I1, I2, ..., Im. Therefore, the characteristic that defines
what group a network neuron will belong to is the input neuron from which it receives a connection. The number of neurons in the recurrent network, K, is therefore equal to the product of m and k. A simple rendering of one such setup can be seen in figure 7-1.
1 2 1 I1 2 3 3 4 4 Input Stimuli, m=4 Recurrent Network, k=2 I4 I3 I2
Figure 7-1: A rendering of the experimental setup used for the investigation into ST DP + N learning throughout this chapter. The recurrent connections between the neurons of the recurrent network are not shown in this illustration. The number on each of the neurons shown within the recurrent network, represent the sub-group to which the neuron belongs. The sub-group — usually referred to as simply group — to which a network neuron belongs is determined by the input neuron from which it receives an input. Network neurons receiving an input from input neuron 1 belong to group1, neurons receiving an input from input neuron
2 belong to group2 and so on. There are as many groups of network neurons as
there are input neurons. All groups contain equal numbers of network neurons. In this figure, only 4 groups and 4 inputs are shown, m = 4, and there are 2 network neurons per group, k = 2.
Basic Network Structure and Setup 188 network receives inputs from all other neurons within the recurrent network.
7.2.1
Network Generation
Consider a recurrent network, N , of LIF neurons with full connectivity. The input connection vector to an individual LIF neuron n ∈ N is defined as Cn =
c1
n, c2n, ..., cK−1+Mn where, K is equal to the number of neurons in N , while M
is the number of neurons in a single input pool. In the case of the following experiments M = 1.
Each LIF neuron in the network N is an excitatory neuron — no inhibitory neurons are used. This is done because the parameters given for inhibitory neu- rons within CSIM, based upon study of biological inhibitory neurons (Thomson et al, 2002), produce inhibitory neurons that respond more quickly to stimuli than excitatory neurons. This faster reaction time means that an inhibitory neu- ron receiving a stimulus from one of the input neurons, I1, I2, ..., Im, will have its
synaptic weight modified by an amount that is larger than any modification value for an excitatory neuron in the same situation — see the form of the excitatory segment of the learning window in chapter 5. Therefore, with repeated iterations, the inhibitory neuron dominates and consequently has the effect of dampening down any activity that may have otherwise occurred.
This may suggest that inhibitory neurons have a specialised role within biologi- cal neural networks and that they probably should not be treated with the same learning rules as excitatory neurons as is done in Pfister et al (2006), and Legen- stein et al (2005).
7.2.2
Why full Connectivity?
The use of full connectivity of the recurrent neurons is due, in part, to the small size of the sub-groups used throughout this chapter. These groups were discussed and introduced previously in section 7.2 and are also explained in figure 7-1.
Choosing Parameters 189 This small group size, combined with the fact that the neurons for each sub- group are selected randomly from the whole recurrent network at the initial network generation means that, in order to be sure that a neuron within the network receives enough connections from the relevant sub-group, full, or near full connectivity must be used.
The need for full connectivity can also be thought of as somewhat realistic, by virtue of the fact that within biological neural networks, local connectivity be- tween the same number of neurons considered here would typically be very high. If one considers that a single biological neuron could receive circa 10, 000 synaptic connections, then one could arrive at the conclusion that enabling full connectiv- ity in a network of only 200 neurons would be a reasonable implementation. ST DP + N learning discussed in section 5 is applied initially to every single synapse within the recurrent network. This means that, with each neuron receiv- ing hundreds of connections there will be tens of thousands of synapses — in the largest networks considered — to be modified with each iteration of the stimuli through the network and the resulting firing activity.
Computationally, this is quite an intensive task for the resources immediately available. As a consequence the majority of experiments detailed here use net- works consisting of only 200 recurrent neurons — for the purposes of obtaining multiple runs in order to demonstrate some form of statistical significance and consistency.