Multiple Pattern Learning Results and Analysis

Figure 6-7 shows a plot of the mean values of d(goal, actual), the distance between the desired goal spike train for the target neuron and its actual output spike train after training for the different training techniques just described and for cases of 1, 2, 3 and 4 I/O associations, over 20 runs in each case. For those instances in figure 6-7 that show the result for multiple patterns, d(goal, actual) is given for each pattern. The idea is to train a readout neuron on multiple I/O associations and observe the accuracy of the output of the trained neuron for each pattern it is required to learn, as the number of patterns increases.

Along with each measure of mean distance, the 95% confidence interval is also provided so that significant differences between the results of using different train- ing techniques and/or different numbers of I/O associations can be clearly seen.

Multiple Pattern Training Techniques 169

Benchmark(1) Ser(2) Alt(2) Super(2) Alt(3) Super(3) Alt(4) Super(4) 1.4 1.6 1.8 2 2.2 2.4 2.6 2.8 3 3.2 3.4

d(goal,actual) for different training methods, with 500 inputs

Training Method(Num of Patterns)

d(goal,actual)

Pattern 1 Pattern 2 Pattern 3 Pattern 4

Figure 6-7: A plot of the mean values of d(goal, actual) for varying number of patterns and training methods. Averaged over 20 runs in each scenario.

For a given training method and number of I/O associations, we can say with 95% confidence level that the given confidence interval will contain the value of d(goal, actual) obtained from a single run. The difference between any two mean values of d(goal, actual) can be considered significant (at the 95% level) if their confidence intervals do not overlap.

The distances are calculated using the metric described previously in this chapter. The metric acts as an objective measure of how much of the goal pattern the neuron has learnt, or how well the neuron has learnt a particular goal pattern. In the experiments that follow, there are many examples of the readout neuron learning some patterns well, and some not so well, but in all cases the learning is

Multiple Pattern Training Techniques 170 imperfect. The metric is ideal as a measure of learning in such cases as it allows the amount of learning a neuron has undergone to be quantified, and as a result comparisons can be made between how much a neuron has learnt of two or more patterns.

It should be noted that in addition to changing the learning rate and the norm to maintain the relative strength of the training throughout changes in network size, another check is also implemented. It was found in preliminary testing that as the number of patterns, on which a neuron was trained, was increased, the number of spikes contained in the resulting output spike trains for each pattern of the trained neuron, also increased. The increase was such that it was possible that the observed poor performance — high d(goal, actual) values — could have been attributed to too many spikes in the outputs, something which could perhaps be remedied by reducing the norm of the input weight vector to maintain the average number of output spikes close to 5, over all of the patterns on which the neuron was trained. As in figure 6-7, it can be seen that the greater the number of patterns a neuron is required to learn, the less well it is able to learn them, even with the output maintained at the goal of 5 spikes.

For each of the training technique/number of I/O associations shown, a total of 20 networks were randomly generated, and the relevant distances calculated for each. The average was then recorded and plotted in figure 6-7.

Figure 6-8 shows the number of spikes present in the actual output of the target neuron after training for each of the combinations of training technique/number of I/O associations used. It shows that the average number of spikes in the output spike trains of the neuron, over all patterns learned in each instance, is kept close to 5.

It is thought that the need to change the norm as the number of patterns increases arise because, as more patterns are required to be learnt, more synaptic weight are strengthened. This results in a greater synaptic input to the neuron and therefore a lower norm must used to maintain spiking frequency.

Multiple Pattern Training Techniques 171

Benchmark(1)Serial(2) Alt(2) Super(2) Alt(3) Super(3) Alt(4) Super(4) 0 1 2 3 4 5 6 7 8 9

Output freq w.r.t training method and num of patterns − 500 inputs

Training Method(Num of Patterns)

Num of Output Spikes

Figure 6-8: The average number of spikes contained in the output spike train of the output neuron over the 20 runs for each training method-number of patterns combination used. Also shown are the 95% confidence intervals for each training method.

and a precise output spike train produced by the ST DP + N learning rule when the 500 input synapses to a single neuron, are trained over 50 repetitions, on a single I/O pattern and averaged over 20 unique, randomly generated I/O pat- terns. This is the benchmark distance result, against which all other results shall be compared.

It can be seen that some entries in figure 6-7 have up to 4 differently coloured bars. Each of these bars represent a distance d(goal, actual) for one of the goal spike trains on which the target neuron has been trained.

Increasing Number of Inputs 172 The first instance of training the target neuron to learn two patterns uses the serial method, discussed above. The serial technique was mentioned here for completeness. In reality, it is not a viable training technique due to the fact that, as the number of training repetitions increases, the previous learnt patterns get effectively erased. Therefore, all other results that can be seen in figure 6-7 for number of I/O associations 2, 3 and 4 are obtained using only the super and alternate techniques.

The next two results both also represent attempts to train a neuron to learn 2 precise I/O associations, using the alternate and the superposition techniques. It can be seen that both techniques are capable of producing weight vectors that allow for the weight vector of the target neuron to store two I/O associations that are not quite of comparable accuracy to the benchmark single pattern. As the number of I/O associations increase to 3 and 4, it can be seen that the results for d(goal, actual) for each of the output spike trains of the target neuron, increase so much that they are no longer comparable to the benchmark — their confidence intervals no longer overlap. It can also be seen that the alternate technique consistently outperforms the superimposed technique. Additionally, it would appear that increasing the number of patterns a single neuron is required to know at any time, results in a decrease in the accuracy and the integrity of all of the patterns learnt.