Simulating the BAM

The disadvantage to this approach is that it consumes twice as much mem- ory as does the single-matrix implementation. There is not much that we can do to solve this problem other than reverting to the single-matrix model. Even a implementation will not solve the problem, as it will require ap- proximately three times the memory of the single-matrix model. Thus, in terms of memory consumption, the single-matrix model is the most efficient imple- mentation. However, as we have already seen, there are performance issues that must be considered when we use the single matrix. We therefore choose to implement the double matrix, because run-time performance, especially in a large network application, must be good enough to prevent long periods of dead time while the human operator waits for the computer to arrive at a solution.

The remainder of the network is completely compatible with our generic network data structures. For the BAM, we begin by defining a network with two layers:

record BAM =

X : "layer; to first layer

Y : to second layer

end record;

As before, we now consider the implementation of the layers themselves. In the case of the BAM, a layer structure is simply a record used to contain pointers to the outputs and arrays. Such a record is defined by the structure

record LAYER =

: to node outputs

WEIGHTS : to

end record;

Notice that we have specified integer values for the outputs and weights in the network. This is a benefit derived from the binary nature of the network, and from the fact that the individual connection weights are given by the dot product between two integer vectors, resulting in an integer value. We use inte- gers in this model, since most computers can process integer values much faster than they can floating-point values. Hence, the performance improvement of the simulator for large BAM applications justifies the use of integers.

We now define the three arrays needed to store the node outputs, the connection weights, and the intermediate These arrays will be sized dynamically to conform to the desired BAM network structure. In the case of the outputs arrays, one will contain x integer values, whereas the other must be sized to contain y integers. The weight_ptr array will contain a memory pointer for each PE on the layer; that is, x pointers will be required to locate the connection arrays for each node on the x layer, and y pointers for the connections to the y layer.

Conversely, each of the arrays must be sized to accommodate an integer value for each connection to the from the input layer. Thus, each

weights array on the x layer will contain y values, whereas the weights arrays on the y layer will each contain x values. The complete BAM data structure is illustrated in Figure 4.15.

4.4.3 BAM Initialization Algorithms

As we have noted earlier, the BAM is different from most of the other ANS networks discussed in this text, in that it is not trained; rather, it is initialized. it is initialized from the set of training vectors that it will be required to recall. To develop this algorithm, we use the formula used previously to generate the weight matrix for the BAM, given by Eq. (4.6), and repeated here

outputs weights

Figure The data structures for the BAM simulator are shown. Notice the difference in the implementation of the connection arrays in this model and in the single-matrix model described earlier.

4.4 Simulating the BAM 163

for reference:

We can translate this general formula into one that can be used to determine any specific connection weight, given L training pairs to be encoded in the BAM. This new equation, which will form the basis of the routine that we will use to initialize the connection weights in the BAM simulation, is given by

= \i[c] (4.36)

where the variables r and c denote the row and column position of the weight value of interest. We assume that, for purposes of computer simulation, each of the training vectors x and y are one-dimensional arrays of length C and R, respectively. We also presume that the calculation will be performed only to determine the weights for the connections from layer x to layer y. Once the values for these connections are determined, the connections from y to x are simply the transpose of this weight matrix.

Using this equation, we can now write a routine to determine any weight value for the BAM. The following algorithm presumes that all the training pairs to be encoded are contained in two external, two-dimensional matrices named XT and YT. These arrays will contain the patterns to be encoded in the BAM, organized as L instances of either x or y vectors. Thus, the dimensions of the XT and YT initialization matrices are L x C and L x R respectively.

function weight

return integer;

and return weight value for position var i : integer; : [] sum : integer; begin sum = x = XT; y = YT; iteration array x y

for i = 1 to L do all training

sum = sum + *

end do; return end

the

The weight function allows us to compute the value to be associated with any particular connection. We will now extend that basic function into a general routine to initialize all the weights arrays for all the input connections

to the PEs in layer y. This algorithm uses two functions, called and that return the number of rows and columns in a given matrix. The implementation of these two algorithms is left to the reader as an exercise.

procedure initialize

all input connections to a layer, var connects: units: L,R,C : i, integer; begin units = L = rows_in R = cols_in C = cols in into {size-of of training of Y of X

for i = 1 to do all units on

connects = pointer to weight

for j = 1 to do

all connections to connects[j] weight (R, C, L, XT, YT)

end do; end do; end procedure;

We indicated earlier that the connections from layer y to layer x could be initialized by use of the transpose of the weight matrix computed for the inputs to layer y. We could develop another routine to copy data from one set of arrays to another, but inspection of the initialize algorithm just described indicates that another algorithm will not be needed. By substituting the x layer record for the y, and swapping the order of the two input training arrays, we can use to initialize the input connections to the x layer as well, and we will have reduced the amount of code needed to implement the simulator. On the other hand, the transpose operation is a relatively easy algorithm to write, and, since it involves only copying data from one array to another, it is also extremely fast. We therefore leave to you the choice of which of these two approaches to use to complete the BAM initialization.

4.4.4 BAM Signal Propagation

Now that we have created and initialized the BAM, we need only to implement an algorithm to perform the signal propagation through the network. Here again, we would like this routine to be general enough to propagate signals to either layer of the BAM. We will therefore design the routine such that the direction

4.4 Simulating the BAM 165

of signal propagation will be determined by the order of the input arguments to the routine. For simplicity, we also assume that the layer outputs arrays have been initialized to contain the patterns to be propagated.

Before we proceed, however, note that the desired algorithm generality implies that this routine will not be sufficient to implement completely the iterated signal-propagation function needed to allow the BAM to stabilize. This iteration must be performed by a higher-level routine. We will therefore design the unidirectional propagation routine as a function that the number of patterns changed in the receiving layer, so that the iterated propagation routine can easily determine when to stop.

With these concerns in mind, we can now design the unidirectional signal- propagation routine. Such a routine will take this general form:

function propagate return integer;

signals from layer X to layer var changes, i, integer;

ins, outs : connects : sum : integer; of begin outs = start of Y changes = 0;

for i = 1 to length (outs) do all output

ins = X connects= sum for j = 1 to do all sum = sum + * end do; if (sum < 0) negative then sum = -1 -1 as

else if (sum > 0) positive

then sum = 1 1 as

else sum = use old

if (sum != unit

then changes = changes + 1;

= sum; new

end

return of

To complete the BAM simulator, we will need a top-level routine to per- form the bidirectional signal propagation. We will use the propagate routine described previously to perform the signal propagation between layers, and we will iterate until no units change state on two successive passes, as that will indicate that the BAM has stabilized. Here again, we assume that the input vectors have been initialized by an external process prior to calling recall.

procedure recall

signals in the BAM until

var delta : integer; many units

begin

delta = 100; non-zero

while (delta != 0) two successive

do

delta to

delta = delta + propagate delta = delta + propagate end

end procedure;

Programming Exercises

4.1. Define the pseudocode algorithms for the functions and as described in the text.

4.2. Implement the BAM simulator described in Section 4.4, adding a routine to