M=900
-
M=950 - -X-
6
5
4
3
O)2
1
0
0.25
0.3
0.35
0.4
0.2
0.45
0.5
alphaFigure 4.9: Four superim posed contour plots of th e functional linear th re sh olds (> 85% recall quality) for different levels of m em ory loading (M = 500, 700,900, and 950. The linear thresholds are defined by th e slope ( a and offset (7). T he netw ork configuration is th e sam e as th a t of figure 4.8.
In a sim ilar m anner to th a t described previously, th e contour lines corre sp o n d in g to an > 85% quality level were ex tra cte d from th e sim ulation resu lts, an d are p lo tte d in Figure 4.9. Note th a t th e sh ap e of th e contour regions has been well p red icted by th e theory, and th a t th e fu n ctio n al region a t high load ing is in clu d ed in th e functional regions at lower m em ory loading. As before, th is im plies th a t a single value of th e slope a n d offset for th e linear th resh o ld works in d ep en d en t of loading.
4.4
D iscu ssio n
In th is s tu d y th e table-based search m etho d is used to evaluate th e perfor m ance of all possible progressive recall strategies in a sim ple form of asso ciative memory. In th is ty p e of biologically plausible associative m em ory th e b est perform ance is achieved w ith sparse coding of inform ation and w ith sim ple sy n ap tic interactions th a t do not require a wide range of weight values or weights th a t change of sign. T he results p resen ted in th is chapter, which are consistent w ith those previously rep o rted by A m ari (1989) an d Treves and Rolls (1991), show th a t th ere is an increase in inform ation capacity as th e en coding of sto red events becomes more sparse. T h e m em ory capacity (in bits) resu ltin g from th e b est linear threshold stra teg y is twice as high w ith W = 100 as it is w ith W = 200.
T h e basic finding in th is ch ap ter is th a t th e o p tim al thresholding sequence for progressive recall in th is type of auto-associative m em ory is a linear function of th e netw ork activation. Unlike rep o rted in G ibson an d R obinson (1992) or B e n n e tt et al. (1994), th e recall capability of th e netw ork depends strongly on th e choice of th e offset value in th e linear function, 7 . In p a rtic u la r, it is
im p o rta n t to have 7 greater th a n zero for netw orks th a t are n ear th e capacity
lim it.
In m ost of th e present analysis th e seed p a tte rn contained only correct cells. We also applied th e table-based search technique to a recall sequence in which th e seed p a tte rn included spurious cells. In th e exam ple cases presented here th e resu ltin g optim al th resh o ld sequence was also a linear function, w ith ap p ro x im ately th e same slope and intercept, as th a t found w ith an u n c o rru p ted seed p a tte rn . However, if th e num ber of spurious cells in th e seed p a tte rn is increcLsed fu rth e r th e n th e to ta l activation in th e netw ork will resu lt in a global
th resh o ld th a t will de-activate all of the p rin cip al cells in th e network.
T h e thresholding strateg y found using th e table-b ased search differs from earlier progressive recall strategies (G ardner-M edw in 1976; Lansner and Eke- berg 1985), in th a t it allows th e activation of spurious cells. In th e th resh o ld sequences found using th is m ethod th ere is a co n sisten t period, during th e early states of th e recall, in which m any spurious cells are active. O th er suggested thresh o ld in g strategies use cell-specific features (e.g. w inner-take-all in terac tion; selection of th e cell w ith th e single highest activation; knowledge of th e un-w eighted as well as th e weighted activation in p u t) in order to increase th e storage capacity of th e network. In th is c h ap te r we have shown th a t su b sta n tia l cap acity increase can be achieved using a form of n eu ral in teractio n which does n o t require th is additional complexity.
T h e resu lts obtained in this stu d y are based on th e use of th e theo ry de veloped by G ibson and Robinson (1992). T he pred ictio ns calculated using th e th e o ry m atch well to th e results found from sim ulations. T his m atch betw een th eo ry an d sim ulation makes it possible to efficiently analyse variations in th e recall process. However, th e theory needs to b e ex ten d ed so th a t a sim ilar type of analysis can be perform ed on variations of th e associative m em ory netw orks th a t are in terestin g either for an expected hig h er storage capacity or more biologically realistic properties. For exam ple, th ese d a ta have been analysed using exam ple netw orks th a t have fifty p ercen t connectivity. If significantly lower (biologically reasonable) connectivity levels are used th e activation of th e p rin cip al cell can no longer be described using a G aussian d istrib u tio n (B uckingham 1991) and th e model is no longer a good in d icato r of th e perfor m ance expected from th e sim ulations. F u rth erm o re, th e m odel required th a t th e ex trin sic seed p a tte rn is only briefly p resen ted (d u rin g th e first tim e step
). If th is p a tte rn is presented th ro u g h o ut th e recall th e n th e storage capacity should be higher. It would also be useful to ex ten d th e th eo ry to cope w ith asynchronous recall, which m ight increase th e storage capacity.
It is difficult to com pare th e perform ance of different m ethods for con s tru c tin g associative memories. However, p erh ap s one m easure is the num ber of b its of inform ation th a t can be stored in each synapse. T he num ber of synapses in th e netw ork is clearly one of th e m ost biologically expensive fea tu res, an d a t least from th e silicon circuit p o in t of view it is more expensive to have a large num ber of different states in each synapse. Using th is m easure th e associative m em ory described here com pares favourably w ith a Hopfield netw ork (Hopfield 1982) which has been shown to have a m axim um capacity of 0.14 b its p er synapse. T he sim ulation resu lts re p o rte d here have achieved 0.08 b its p er synapse, using two state synapses in ste ad of th e large range of s ta te s required to encode real-valued synapses.
In th e p resen t theoretical and sim ulation-based evaluation of exam ple n e t works, we found th a t th e constants (a an d 7) th a t define th e best thresholding
s tra te g y do no t vary w ith th e num ber of p a tte rn s sto red in th e memory. This m akes th e im plem entation of th is type of m em ory m uch sim pler. In a bio logical im p lem en tatio n of th is memory, b o th of these co n stan ts correspond to p ro p erties of th e interneuron. For example, each of th e in p u t weights to th e in te rn e u ro n could be a j N ajid th e baseline activ ity ra te of th e interneuron m ight be 7. It has long been suggested th a t th e CA3 region in th e hip p ocam
pus can b e m odelled using a sparsely and rec u rre n tly connected netw ork of p rin cip al cells (M arr 1971). T his type of m em ory m odel m ay eventually lead to a b e tte r u n d erstan din g of biological n eu ral netw ork function.
N o t e s P a rt of th is chapter has been re p o rte d in H irase an d Recce (1996, in press). I have been responsible for conducting th e research including th e algorithm s for th e tab le based search, w riting th e te x t, producing th e graphs, an d th e program m ing. The linear th resh o ld id ea was a resu lt of combined effort an d discussions w ith Michael Recce.