Additive Activation Models

As mentioned before, the activation value of the ith neuron can be interpreted as the cell membrane potential, and i t is a function of time, = The activation models are described by an expression for the first derivative of the activation value of a neuron. Thus gives the rate of change of the activation value of the ith neuron of a neural network.

Activation Dynamics Models 45 For the simplest case of a passive decay situation,

where ( is a constant for the membrane and can be interpreted as the passive decay rate. The solution of this equation is given by

In circuit analogy, can be interpreted as membrane conductance, which is inverse of the membrane resistance The initial value of is The steady state value of is given by

= which is also called the resting potential.

The passive decay time constant is altered by the membrane capacitance which can also be viewed as a time scaling parameter. With the passive decay model is given by

and the solution is given by

Without loss of generality, we can assume = 1 throughout the following discussion. If we assume a nonzero resting potential, then the activation model can be expressed by adding a constant to the passive decay term as

-

+

whose solution is given by

The steady state activation value is given by = the resting potential.

Assuming the resting potential to be zero = 0), if there is a constant external excitatory input then the additive activation model is given by

= -

+

where ( is the weight given to The solution of this equation is given by

=

+

--- -

The steady state activation value is given by = which shows that the activation value directly depends on the

46 Activation and Synaptic Dynamics In addition to the external input, if there is an input from the outputs of the units, then the model becomes an additive autoassociative model, and is given by

1

where is the output function of the unit. For inhibitory feedback connections or for inhibitory external input, the equations will be similar to the above except for the signs of the second and third terms in the above equation. The classical neural circuit described by is a special case of the additive autoassociative model, and is given by et al, 19811

where is the resistance between the neurons i and j, and

model assumes a linear output function = thus resulting in a signal which is unbounded. If the output function is strictly an increasing but bounded function, as in Figure 2.2, and the connection weights are symmetric, = then the resulting model is called model [Hopfield, The model belongs to the class of feedback neural network models,' called autoassociative memory, that are globally stable. We will discuss further on this point in a later section.

A network consisting of two layers of processing units, where each unit in one layer (say layer 1) is connected to every unit in the other layer (say layer 2) and vice versa, is called a heteroassociative network. The additive activation model for a heteroassociative network is given by

N

= -

+

+

,

i = 1, 2, M

j = l

where and are the net external inputs to the units i and j, respectively. Note that and could be different for each unit and

Activation Dynamics Models 47 for each layer. In the above equations V = is the matrix of weights from the units in the layer 2 to the units in the layer 1, and

W = is the matrix of weights from the units in the layer 1 the units in the layer 2. These are coupled first order differential equations. Under special conditions, such as the weights in both the directions being identical, W = and the output function being bounded, the resulting hetroassociative model reduces to a bidirectional associative memory Analogous to the autoassociative memory, the bidirectional associative memory can also be proved to be globally stable. Table 2.1 gives a summary of the development of activation dynamics models discussed in this section.

Table 2.1 Summary of Development of Additive Activation Dynamics Models

General form:

= = N

Passive decay term:

where is the membrane conductance and is t h e membrane capacitance

Nonzero resting potential

With external input

where is a positive constant

Additive model:

model:

Hetroassociative model:

48 Activation and Synaptic Dynamics 2.2.3 Shunting Activation Models

Grossberg has proposed a shunting activation model to restrict the range of the activation values to a specified operating range irrespec- tive of the dynamic range of the external inputs [Grossberg, 1982; Grossberg, 19881. We will first consider the saturation model, where the activation value is bounded to an upper limit. For an excitatory external input the shunting activation model is given by

-

+

- (2.14)

The steady state activation value is obtained by setting = 0, and solving for The result is

As the input then That is, the steady state value of saturates at In other words, if the initial value

then for all t. If the input value refers to an intensity of reflected light = where I is the background intensity value, and is the fraction of the background intensity that is input to the ith unit, then the above saturation model is insensitive to for large background intensities.

In order to make the steady state activation value sensitive to reflectance, irrespective of the background intensity, Grossberg suggested an on-centre off-surround shunting activation model by providing inhibitory inputs from other input elements to the ith unit along with the excitatory input from the ith input element to ith unit as shown in Figure 2.3. Throughout the following discussion we

Processing units

Input intensities

Figure 2.3 An on-center and off-surround configuration for shunting activation model.

assume a hard-limiting threshold function for the output function. That is, = 0, for and, = 1, for Assuming

N

= I and = for convenience, the shunting activation model with on-centre off-surround configuration is given by

Activation Dynamics Models 49

The steady state activation value is obtained by setting = 0, and is given by

From this we can see that, even if as I the steady state activation value does not saturate. Instead, will still be sensitive to the input reflection. It can be seen that, since 1, the steady activation value is always less than the saturation limit, for all t.

In order to make a unit insensitive to small positive inputs, may be due to noise, the shunting activation model can be modified to incorporate a lower limit to the activation value. The following is the resulting model:

The steady state activation value is obtained by setting = 0, and is given by

Note that

+

- as I This steady state activation value is negative as long as the input reflectance value to the ith unit,

+

In that case the output signal of the unit will be zero, since we assume that 0, for That is, the ith processing unit will be sensitive to the input only if its input reflectance value is above a threshold. Thus it is possible to make the unit insensitive to random noise input within a specified threshold limit value. The above shunting activation model has therefore an operating range of - for the activation value, since the lowest value for = - which occurs when = 0.

A shunting activation model with excitatory feedback from the same unit and inhibitory feedback from other units is given by

where is the inhibitory component of the external input. Note that, on the right hand side of Eq. (2.20) is replaced by for convenience. The inhibitory sign is taken out of the weights and

50 Activation and Synaptic Dynamics hence The shunting model of (2.20) is a special case of

membrane equations and 19521. Equation (2.20) can be written in the most general form as

= - A,

+

-

where all the constants are positive. The first term' hand side corresponds to the passive decay the second term corresponds to the excitatory term and the third corresponds to the inhibitory term. If we consider the excitatory term (B, - C, x,)

+

it shows the contribution of the excitatory (external and feedback) input in increasing the activation value of the input. If C, = then the contribution of this input reduces to an additive effect, as in the additive activation model. C, then the contribution of the excitatory input reduces to zero when the activation = This can be viewed as shunting effect in an equivalent electrical circuit, and hence the name shunting activation model. If the initial value then the model ensures that for all t thus showing the boundedness of the activation value within an upper limit. This can be proved by the following argument: If then the second term becomes negative, since we assume that = 1, for all and I, 0. Since the contribution due to the inhibitory third term is negative, if the excitatory second term is also negative, then the steady activation value, obtained by setting = will be negative. Thus there is a contradiction, since we started with the assumption that

which is positive. Hence for all t.

Likewise, the inhibitory term

+

[

shows the contribution of the inhibitory (external and feedback) input in decreasing the activation value of the unit. In this case, if = then the contribution of this input reduces to an additive effect, as in the additive activation model. If then the contribution of the inhibitory input reduces to zero when the activation

=

-

.

This can be viewed as a shunting effect in an equivalent electrical circuit. If the initial value -

,

then the model ensures that -

,

for all t This can be proved by the following argument: For

-

the contribution of the inhibitory third term will be positive, since 0, for all x. Since the excitatory second term is always positive, the steady state activation value obtained by setting (t) = is always positive. But we assumed that - which is negative. Thus there is a contradiction. Hence 2 - Table 2.2 gives a summary of

Activation Dynamics Models 51

the development of shunting activation models discussed in this section.

Table 2.2 Summary of Development of Shunting Activation Dynamics Models

Goal: To keep the operating range of activation value to a specified range General form:

Saturation model: To restrict to an upper limit

On-centre configuration: To make it sensitive to changes in the external input

=

+

- -

j # i Setting noise limit:

=

+

- -

+

With excitatory feedback from the same unit and inhibitory feedback from other units:

=

+

(B, -

+

-

+

I

In document ANN by B.Yegnanarayana.pdf (Page 60-67)