We have shown that our structural plasticity mechanism modifies receptive fields mainly in its surround by random formation of new synapses close to existing synapses and removes weak synapses. The mechanism is build upon the principles that the weight strength is re- lated to the synaptic lifetime and that the formation of new synapses takes place in vicinity to active ones. Our model is with and without structural plasticity able to learn various Gabor-like simple-cell receptive fields in V1-L4. However, without structural plasticity the development of the receptive field sizes depends on the limitation of the individual connection matrix of each neuron. Without, each of the four regarded model configu-
5 Intrinsic Plasticity
This chapter introduces the intrinsic plasticity mechanisms. First, we describe why we need this plasticity. We introduce the related computational models and their implementa- tions. Then, we present our neuron model and the implementation of the plasticity mech- anisms. Subsequently, we evaluate the functioning, stability, and effect on the activity distribution. Parts of the chapter have been published in Teichmann and Hamker (2015).
5.1 Introduction
One of the most important criterion of biologically plausible learning principles is their locality, i.e. a neuron has only access to its own state and incoming signals. Common formulations of Hebbian learning use only information local to synapse or neuron. In- deed, such formulations can not ensure that a population of so defined neurons will learn a codebook representing the input’s manifold. By extending such populations by plausible mechanisms of inter neuron interaction, as inhibition, it can be shown that an adequate representation of the input can be learned (e.g. F¨oldi´ak, 1990; Wiltschut and Hamker, 2009). A common plausible principle to learn such connections is anti-Hebbian learning (e.g. F¨oldi´ak, 1990; Wiltschut and Hamker, 2009; Teichmann et al., 2012). However, this principle aims only to reduce correlations between neurons based on their coactivity. It has no objective ensuring an adequate input encoding by a population of neurons. This means that information encoding can be just optimal when the neuron responses are independent, i.e. no information is duplicated, and all information in the input lead to a response (Si- moncelli and Olshausen, 2001). However, Hebbian learning depends on the association between pre- and postsynaptic activity. If a presynaptic neuron shows, by average, more activity, its connection will be more strengthen. Hence, it is very likely for more complex input data, or in higher layers of abstractions, that some neurons will show higher activity for the pattern they encode than others. Which in turn will enforce the imbalance in subse- quent layers (cf. Diehl and Cook, 2015). This means, the information, which neurons with
5 INTRINSIC PLASTICITY
FIGURE 5.1: ”Two fundamentally different mechanisms for the homeostatic regulation of neu- ronal firing. (a) Neuronal activity is determined both by the strength of excitatory and inhibitory synaptic inputs and by the balance of inward and outward voltage-dependent conductances that reg- ulate intrinsic excitability, here illustrated as the relative number of Na (blue) and K (red) channels. Neurons can compensate for reduced sensory drive either by using synaptic mechanisms to modify the balance between excitatory and inhibitory inputs (b) or by using intrinsic mechanisms to modify the balance of inward and outward voltage-dependent currents (c).“ Figure and description taken from Turrigiano (2011).
higher average activities convey, will dominate the information the efferent neurons will learn to encode.
The human neural system has various mechanisms to stabilize its functioning (Turri- giano and Nelson, 2004; Turrigiano, 2011). Beside homo- and hetero-synaptic regulations of the weight development, as synaptic scaling, it has been found that neurons preserve their average firing rate over time, independent from their synaptic strengths (Zhang and Linden, 2003). If a neuron is highly stimulated, or weakly, over a long period of time it can be observed that its sensitivity to this stimulation decreases, or increases, and the neuron returns to its previous activity regime (Fig. 5.1). This means, its intrinsic excitability is adapted, called intrinsic plasticity (Zhang and Linden, 2003; Turrigiano, 2011).
For first computational implementations of intrinsic plasticity it has been speculated that neurons try to approach an exponential firing regime (Stemmler and Koch, 1999; Triesch, 2005a,b). This is because exponential distributions are found efficient for information en- coding, for a fixed average activity in a single neuron case (Simoncelli and Olshausen,
5.1 INTRODUCTION
2001), as well as transmitting information energy efficient (Rolls and Treves, 2011; Levy and Baxter, 1996). Stemmler and Koch (1999) modified a Hodgkin-Huxley model by volt- age dependent conductances. The conductances are increased or decreased by a constant value when the average voltage in the regarded compartment is above or below a certain value. With that, they regulated the mean activity of the neuron. They found that the model changed the activation function so that the responses evoked by a Gaussian input distribu- tion become exponentially distributed. Triesch (2005b) introduced a sigmoidal activation function, where the baseline and slope of the inward current can be configured. These values are adapted during learning to match the mean and the variance of a desired expo- nential distribution. Triesch (2005a) introduced online minimizing the Kullback-Leibler divergence of the neuron activity to an exponential function as alternative approach to determine parameters for the slope and baseline value. Later Savin et al. (2010) demon- strated this approach in a stochastically spiking model, learning independent components from natural scenes. These approaches explicitly modify the neuron’s activation function to achieve exponential firing. However, Triesch (2005a, 2007) have been criticized that, their approach to enforce an exponential distribution, drives the activation function to bio- logically implausible high values and that enforcing sparseness rather than an exponential distribution would be sufficient (Elliott, 2014). Altogether, it is still unclear whether the objective of cortical neurons is an exponential regime or how they achieve it.
We speculate that the aspect of the exponential regime is not an objective of cortical neurons, rather it is a byproduct of the neural circuit developing sparse representations. Instead of enforcing an exponential activity distribution, we hypothesize the intrinsic plas- ticity mechanism aims to stabilize the operating point of the neurons so that the brain is not wasting resources for non responding cells or hyperactive ones. Thus, we aim to control the two most important moments of neural activity, the mean and the variance, by adapting the slope and the threshold of the rectified linear activation function of our model neu- rons (Sec. 5.3). Beside sigmoidal activation functions, rectified linear activation functions have been very common in the past and recently in deep neural networks. There is also biological evidence that this function type is an adequate description for cortical neurons Ringach and Malone (2007). However, this function is mainly linear, except its rectifica- tion. Hence, mathematically it can not transfer any input distribution into an exponential one. We implemented a modifiable rectified linear activation function and the control of its parameters and demonstrate first its effectiveness for stabilizing the neural response prop-
5 INTRINSIC PLASTICITY
erties (Sec. 5.4). Than, we analyze the information encoding in deeper network layers. We compare different parameterizations of the plasticity rule. Finally, we examine if the activities of our model neurons are exponentially distributed and whether this is caused by our intrinsic plasticity mechanism.