5: Perform the BAM energy calculation on the second example from Section 4.2.3.

Proof of the BAM Energy Theorem. In this section, we prove the first part of the BAM energy theorem. We present this proof because it is both clever and easy to understand. The proof is not essential to your understanding of the remaining material, so you may skip it if you wish.

We begin with Eq. (4.14), which is reproduced here:

According to the theorem, any change in x or y must result in a decrease in the value of E. For simplicity, we first consider a change in a single component of

specifically

We can rewrite Eq. showing the term with explicitly:

Now, make the change The new energy value is

I

The BAM and the Hopfield Memory

Since only has changed, the second terms on the right sides of Eqs. and are identical. In that case we can write the change in energy as

A — I — — — — — ) 7 T •

For convenience, we recall the state-change equations that determine the new value of

k 0

There are two possible changes of to consider. Suppose = +1, and it changes to — 1; in this case, — > 0. But, according to the procedure for calculating this transition can occur only if < Therefore, the value of AE is the product of one factor that is greater than zero and one that is less than zero. The result is that AE < 0.

The second possibility is that = and — Then, < 0, but this transition occurs only if Again, product of one factor less than zero and one greater than zero. In both cases where changes, AE decreases. Note that, for the case where does not change, both factors in the equation for AE are zero, so the energy does not change unless one of the vectors changes.

Equation can be extended to cover the situation where more than one component of the y vector changes. If we write — - then the equation that replaces Eq. for the general case where any or all components of y can change is

AE = - E) (4.18)

This equation is a sum of m terms, one for each possible which are either negative or zero depending on whether or not y; changed. Thus, in the general case, E must decrease if y changes.

| Exercise 4.6: Prove part 2 of the BAM energy theorem. | Exercise 4.7: Prove part 3 of the BAM energy theorem.

Exercise 4.8: Suppose we have an autoassociative BAM whose weight matrix is calculated according to

+ + • • • +

where the are not necessarily Show that the weight matrix can be written as

4.3 The Hopfield Memory 141

where a is a constant, I is the identity matrix, and S is identical to the weight matrix, but with zeros on the diagonal. For an arbitrary input vector, x, show that the value of the BAM energy function is

where is a constant. From this result, deduce that the change in energy, AE, during a state change, is independent of the diagonal elements on the weight matrix.

4.3 THE HOPFIELD MEMORY

In this section we describe two versions of an ANS, which we call the Hopfield We shall show that you can consider the Hopfield memory as a derivative of the BAM, although we doubt that that was the way the Hopfield memory originated. The two versions are the discrete Hopfield memory, and the continuous Hopfield memory, depending on whether the unit outputs are a discrete or a continuous function of the inputs respectively.

4.3.1 Discrete Hopfield Memory

In the discussion of the previous sections, we defined an autoassociative BAM as one which stored and recalled a set of vectors XL }• The prescription for determining the weights was to calculate the correlation matrix:

=

Figure 4.4 illustrates a BAM that performs this autoassociative function. We pointed out in the previous section that the weight matrix for an au- toassociative memory is square and symmetric, which means, for example, that the weights and are equal. Since each of the two layers has the same number of units, and the connection weight from the nth unit on layer 1 to the nth unit on layer 2 is the same as the connection weight from the nth unit on layer 2 back to the nth unit on layer 1, it is possible to reduce the autoassocia- tive BAM structure to one having only a single layer. Figure 4.5 illustrates this structure. A somewhat different rendering appears in Figure 4.6. The figure shows a fully connected network, without the feedback from each unit to itself. The major difference between the architecture of Figure 4.5 and that of Figure 4.6 is the existence of the external input signals, This addition modifies the calculation of the net input to a given unit by the inclusion of the term. In this case,

x layer

x layer

Figure 4.4 The autoassociative BAM architecture has an equal number of units on each layer. Note that we have omitted the feedback terms to each unit.

x layer

Figure 4.5 The autoassociative BAM can be reduced to a single-layer

structure. Notice that, when the reduction is carried out, the feedback connections to each unit reappear.

4.3 The Hopfield Memory 143

Figure 4.6 This figure shows the Hopfield-memory architecture without

the feedback connections to each unit. Eliminating these connections explicitly forces the weight matrix to have zeros on the diagonal. We have also added external input signals,

to each unit.

Moreover, we can allow the threshold condition on the output value to take on a value other than zero. Then, instead of Eq. we would have

+ 1 >

— <

(4.20)

where is the threshold condition for the unit.

A final point concerns the use of binary (0, +1) vectors rather than bipolar vectors. original papers used binary (0, 1) vectors, whereas we have used the bipolar form up to now. To facilitate the reading of the original papers, we want to be able to shift to binary vectors, All that is required is that we modify a few equations slightly. To calculate the weight matrix, we use

= - f)(2v, - - + + - - (4.21)

where 1 is the vector with all as components. The expression - F) converts the binary vector, into the equivalent bipolar vector, so the weight matrix is the same as that calculated with the original bipolar vectors. The second change occurs when we apply the threshold: If < Ui then = 0 rather than

The energy equation is modified by the addition of the and terms. Furthermore, we now explicitly define the diagonal elements of the weight matrix to be zero. The energy equation for the network is

The factor of did not appear in the energy equation of the BAM. In the BAM, both forward and backward passes contributed equally to the total energy of the system. In the Hopfield memory, there is only a single layer, hence, half the energy that there is in the BAM.

Exercise 4.9: Beginning with Eq. (4.22), show that the Hopfield network will