fields.
3.6 Stimulation protocol for evaluations
To any input stimuli we present to our network we apply the described preprocessing to obtain the whitened on-off-center structure for our input. We always apply the preprocess- ing with the same parameters, which we found suitable for all used image datasets. After the network training we turn off all plasticities (structural, intrinsic, and synaptic plastic- ity) before we present test stimuli to the network. Further, we reset all network activities and membrane potentials to zero before we present a stimulus. We record the responses 100ms after the stimulus onset (10 times the time constant of the neurons activation func- tion). Any further details about the stimuli and presentation protocols are described in the respective evaluation sections.
4 Structural Plasticity
This chapter introduces the structural plasticity mechanisms. First, we describe the re- lated computational models and their mechanisms. Then, we present our implementation. Subsequently, we evaluate its functioning and stability. An initial version of the presented implementation has been developed together with the exchange student Maxwell Shinn.
4.1 Introduction
One challenge in creating neural models of the visual system is the appropriate definition of the connectivity. The modeler constrains the results with its definition, also for learning models. Using too few connections neurons will not develop appropriate receptive fields. Using too many the model might lose features like retinotopic organization. Further, often the precise knowledge about appropriate connection sizes is lacked, for instance in deeper layers of the cortex or for different neuron types like interneurons. Also within the same population of neurons receptive field sizes can largely differ. Hence, a mechanism at hand refining the connection structure based on the learned weights would be appreciated. Such a mechanism can be found in the human brain by structural plasticity.
In recent years several models of structural plasticity have been developed. These mod- els largely differ in the used methods. However, the most models treat structural changes as random process. Nevertheless, several models link the process of synapse removal di- rectly or indirectly to the strength of the synaptic weight. Zheng et al. (2013) developed a self-organizing recurrent network with a simple form of STDP. A synapse is removed (non random) when the weight becomes, through synaptic plasticity, lower than zero. Simi- larly, in Deger et al. (2018) a synapse is removed when, through STDP, the weight reaches zero, however, two neurons can have multiple contacts so that a connection is finally re- moved when the total weight becomes zero. Fauth et al. (2015b) implemented a two neuron model to investigate the relation between synaptic plasticity and structural plasticity. There synapse removal has been defined as probabilistic process depending on the strength of the
4 STRUCTURALPLASTICITY
synaptic weight. That is, synapses are removed using a sigmoidal probability function with high values for weak weights.
Beside a direct weight strength dependent removal some models indirectly rely on the weight strength. Helias (2008) implemented a network of integrate-and-fire neurons which learns the excitatory connectivity with a plasticity rule accounting for LTP. Structural plas- ticity acts here as counterpart to LTP and removes synapses if the correlation between the pre- and postsynaptic neuron’s activity drops below a threshold. Alternatively, in Deger et al. (2012) a sigmoidal shaped probability function is used for the removal, if the corre- lation is below average. Similarly to Hebbian learning, correlation bases on the coactivity of the pre- and postsynaptic neuron. Thus, basing the synapse removal on the correlation between the neuron’s activity is strongly related to a weight strength based approach. Fur- ther, strong excitatory synaptic connections would necessarily lead to highly correlated neuronal activity.
The process of synapse removal has also been explained by a model using random fluc- tuations in the spine volume (Yasumatsu et al., 2008). It has been shown that the finding can be reproduced that thin spines are much more likely to disappear than thick spines. Another criterion, independent from individual synaptic weights, is the homeostasis of the neuron’s activity. It was used to change the amount of dendritic or axonal contacts a neuron can form (Butz and van Ooyen, 2013; Butz et al., 2014a,b; Gallinaro and Rotter, 2018). If the amount of these contacts drops below the actual number of formed synapses a random synapse is removed. Much simpler, the constant decay of the synapse amount through random removal has been successfully used (Butz et al., 2009a; Fauth et al., 2015a).
As well as for synapse removal constant probabilities are often used for the decision whether a synapse is formed (Deger et al., 2012; Zheng et al., 2013; Deger et al., 2018). Fauth et al. (2015a,b) used a constant probability to form synapses out of a set of potential synapses between two neurons. However, relying on the definition of potential synapses the model brings, implicitly, a spatial restriction where new synapses can be formed. That is, new synapses can just be formed where axons are close to the dendrite of a neuron. Regarding the location of neurons in the cortical tissue and assuming axons and dendrites are growing outgoing from this location the vicinity between these neurons can be taken into account. Butz et al. (2009a) used a Gaussian distance measure within a grid of neurons to determine the formation probability. This approach has been extended by considering the amount of vacant synaptic elements (Butz and van Ooyen, 2013; Butz et al., 2014a,b).
4.1 INTRODUCTION
The amount of these elements is changed based on activity homeostasis. That is, increasing the amount when the neuron is active below a target value and decreasing it if it is active above the target value. Note, that here the synaptic weights are constant. Similarly, but without considering the distance between the neurons, Gallinaro and Rotter (2018) formed randomly after fixed time intervals new synapses when neurons have vacant elements. Surprisingly, the model of Helias (2008) comes with no synapse formation. Each neuron starts there with k excitatory synapses which are removed when the pre- and postsynaptic neuron’s correlation is low. Thus, just connections between highly correlated neurons remain. Indeed, this method seems to be ineffective for any noisy system where for longer time periods correlations can be low. Then this approach would lead to a nearly total wipe out of synapses, where just few remaining connections dominate the postsynaptic activity. It also do not match findings where the majority of found cortical synapses have just a low contribution to the postsynaptic activity (e.g. Cossell et al., 2015).
We assume for our model that structural plasticity is a stochastic process without any guidance. Synapses are created in the vicinity of existing synapses and weak synapses are removed. Moreover, we aim to reduce the influence of the initial connectivity on the learnings of the network. The outcome of the model should be experience driven. Ini- tially, we use retinotop organized connections, which is a desired property of the visual cortex. The structural plasticity should not impair this organization, which would be likely for a stochastic process. We assume that the synapse creation in close range to existing synapses prevents the system from implausible expansions of the receptive fields to spa- tially distant locations. The method should be usable for large connection matrices, where a fully stochastic process would lead to small probabilities for forming synapses close to existing ones. Beyond that, it should be applicable on all types of synapses. We assume that a stochastic process just utilizing the vicinity of the synapses as additional criterion to the weight strength will work for excitatory and inhibitory synapses. We describe the mechanisms and assumptions of our implementation in Sec. 4.2. We apply structural plasticity on all model synapses, similar as we apply synaptic plasticity on all synapses in parallel. Subsequently, we evaluate the changes of the connectivity induced by the mech- anisms (Sec. 4.3). That is, the amount synapses and the receptive field sizes. We compare the development of the receptive fields from different initial conditions. We also control if the retinotop organization of the receptive fields is preserved and the network convergence is not impaired.
4 STRUCTURALPLASTICITY