Multiplication With Neurons

(1)

School of Informatics

Multiplication With Neurons

Author:

Panagiotis E. Nezis

Supervisor:

Dr. Mark van Rossum

Thesis

MSc in Informatics

August 17, 2008

(2)

ἐν οἶδα ὁτι οὐδὲν οἶδα Socrates

“I know that I don’t know”

Καὶ πάσχειν δή τι ὑπὸ τοῦ χρόνου, καθάπερ καὶ λέγειν εἰώθαμεν ὅτι κατατήκει ὁ χρόνος, καὶ γηράσκει πάνθ᾿

ὑπὸ τοῦ χρόνου, καὶ ἐπιλανθάνεται διὰ τὸν χρόνον, ἀλλ᾿ οὐ μεμάθηκεν, οὐδὲ νέον γέγονεν οὐδὲ καλόν· φθορᾶς γὰρ αἴτιος καθ᾿ αὑτὸν μᾶλλον ὁ χρόνος· ἀριθμὸς γὰρ κινήσεως, ἡ δὲ κίνησις ἐξίστησι τὸ ὑπάρχον.

Aristotle Physics, Book IV, Ch. 12 A thing, then, will be affected by time, just as we are accustomed to say that time wastes things away, and that all things grow old through time, and that there is oblivion owing to the lapse of time, but we do not say the same of getting to know or of becoming young or fair.

For time is by its nature the cause rather of decay, since it is the number of change, and change removes what is.

Translated from Greek by R. P. Hardie and R. K. Gaye

(3)

(4)

Abstract

Experimental evidence can be found that supports the existence of multiplicative mechanisms in the nervous system [23]. The exact way multiplication is implemented in neurons is unclear.

However there is a lot of interest about its details, driven by the experimental observations, which imply its existence. In this thesis we used feed forward networks of integrate-and-fire neurons in order to approximate multiplication. The main hypothesis done is that the minimum function can give a multiplicative like response. Networks that implement the minimum function of two inputs were created and tested. The results show that the hypothesis was correct and we successfully managed to approach multiplication in most cases. The limitations and some interesting observations like the importance of spike timing are also described.

i

(5)

(6)

Acknowledgments

I would like to thank my supervisor, Mark van Rossum, for his enthusiasm, encouragement and insight; our discussions were as enjoyable as they were productive.

I am also grateful to all other Professors I had both in the University of Edinburgh and the National Technical University of Athens, for turning me into a scientist.

There is also a number of persons, who may not have been directly involved in this project, but without whom things would have been much harder.

Last but not least, my family receives my deepest gratitude and love for their faith and their support during the current and previous studies.

iii

(7)

(8)

Declaration

I declare that this thesis was composed by myself, that the work contained herein is my own except where explicitly stated otherwise in the text, and that this work has not been submitted for any other degree or professional qualification except as specified.

(Panagiotis Evangelou Nezis)

v

(9)

(10)

Chapter 1 Introduction

The way information is transmitted in the brain, is a central problem in neuroscience. Neurons respond to a certain stimulus by the generation of action potentials which are called spike trains.

Spike trains are stochastic and repeated presentation of the same stimulus don’t cause identical firing patterns. It is believed that information is encoded in the spatiotemporal pattern of these trains of action potentials. There is a debate between those who support that information is embedded in temporal codes and researchers in favor of rate coding.

In rate coding hypothesis, one of the oldest ideas about neural coding [30], information is embedded in the mean firing rates of a population of neurons. On the other hand, temporal coding relies on precise timing of action potentials and inter-spike intervals. Aim of this proposal is to explore how networks of rate-coding neurons can do multiplication of signals.

In the literature experimental evidence can be found that supports the existence of multiplicative mechanisms in the nervous system. Studies have shown that the optomotor control in the fly is controlled by neural circuits performing multiplication [14],[12]. More recent experiments have found a multiplicative like response in auditory neurons of the barn owl’s midbrain [23], [9].

The exact way multiplication is implemented in neurons is unclear. However there is a lot of interest about its details, driven by the experimental observations, which imply its existence.

Koch and Poggio [18] have discussed different biophysical properties, present in single cells, capable of producing multiplicative interactions. In this proposal we are going to use integrate-and-fire neurons which don’t include the nonlinearities Koch and Poggio propose. As a result the main aim is to approximate multiplication, being confined by the limits of these neuronal models.

1.1 Proposal

In this project we are going to use feed-forward networks of integrate-and-fire neurons. The aim of these small population models is not to do exact multiplication since this is not possible, but to approximate it. Synaptic input is inserted in the neurons among with a noisy bias current.

The synapses may be either excitatory or inhibitory.

An excitatory synapse is a synapse in which an action potential in the presynaptic cell increases the probability of an action potential occurring in the postsynaptic cell. A postsynaptic potential is considered inhibitory when the resulting change in membrane voltage makes it more difficult for the cell to fire an action potential, lowering the firing rate of the neuron. They are the opposite of excitatory postsynaptic potentials (EPSPs), which result from the flow of ions like sodium into the cell.

In our case inhibition is implemented through GABAA synapses with a reversal potential equal to the resting one [30]. This is called shunting inhibition and it has been shown to have a subtractive effect to the firing rate in most circumstances (the shunting conductance is independent

1

(13)

of the firing rate) [16], despite its divisive effect in subthreshold amplitudes.

Since the firing rate of a neuron cannot take a negative value, the output will be a rectified copy of the input which is the difference between the excitatory and inhibitory synaptic inputs. The only nonlinearity present in this neuronal model is the rectification. We are going to combine it with excitation and subtractive inhibition in order to approximate multiplication. The minimum function is going to be used to approximate multiplication. Boolean functions like minimum or maximum can easily be implemented using rate coding neurons.

1.2 Layout of the Thesis

The contents of this thesis are structured in such a way that the non-specialist reader is presented initially with all the background knowledge needed. The aim was to make the thesis as self-contained as possible. Readers who are familiarised with the concepts presented in the background chapter, could skip it or read it selectively.

The remainder of this thesis is outlined as follows. Chapter 2 presents all background knowledge needed in order a non-specialist reader to be able to understand the rest of this thesis. The main aim of this chapter is to present the Integrate-and-Fire neuron model, but first the necessary underlying biological concepts are described. We present the anatomy of a neuron, we analyze the electrical properties of neural cells and how action potential are generated before giving the equations that describe the Integrate-and-Fire model. This chapter (or part of it) could be skipped by somebody familiar with this background information.

In Chapter 3 we try to mention the importance of this thesis. Initially we explain abstractly the necessity of a multiplicative operation in perceptive tasks and describe its relation with the Boolean AND operation. Next, we present experimental evidence of multiplicative operations in the neural system. The fact that the mechanisms that implement such multiplicative operations are not well researched, despite there are multiple reports about neural multiplication, made me interested in this thesis. Finally on the same chapter we present some of the models that researchers have proposed.

In Chapter 4 we present our approach to the problem of multiplication like operations in the brain. Initially we show that an Integrate-and-Fire neuron with an excitatory and an inhibitory input acts as a rectifying unit. Next we show that multiplication could be approached with the minimum function, given that we don’t care for the exact multiplication of two firing rates but for a proportional relation. Finally we present two feed forward networks of I&F neurons that implement the minimum function and were used in the simulations.

The results of our research can be seen in Chapter 5. The simple networks proposed in Chapter 4 are able to implement multiplicative like operations however their performance is not the same. We show which of the two networks performs better and try to analyze why this happens. We also “prove” another important fact, that spike timing is important, even when dealing just with rate coding networks. Finally in Chapter 6 we discuss the results of this thesis and propose some things that could be done if time permitted it.

In order to do the simulations we created a SIMULINK library specific for Integrate-and-Fire neurons. The Appendix describes how SIMULINK works, its advantages compared to other approaches, the Library we created and some examples of its usage.

(14)

Chapter 2 Integrate-and-Fire Neuron Models

2.1 Introduction

The nervous system which is responsible for every action we make, has a magnificent structure including billions of neurons connected to each other in an intricate pattern [13]. Neurons are the elementary processing units in the brain and communicate with short electrical pulses known as spikes or action potentials. It is believed that information is transmitted through firing sequences of spikes. Although spiking neurons are the majority of cells present in the cortex, there is also a large number of glia cells that play a supporting role and are responsible for energy supply of the brain. In Figure 2.1 we can see coloured hippocampal neurons and glia cells.

Figure 2.1 Left-Hippocampal neurons (green) and glial cells (red). Right- Hippocam- pal neuron expressing monomeric Green Fluorescent Protein (GFP). (Images taken from http://www.greenspine.ca.)

Biological research has produced detailed knowledge about the biophysical mechanisms underlying neuronal functionality and spike generation. From a modeling perspective, this knowledge can be used in order to construct neuron models, which can be used in computer simulations of neurons and neural networks. These simulations can help us understand how information is encoded into neural signals and how the network connectivity determines the firing activity. A large number of neuron models has been proposed, ranging from complex realistic descriptions of the

3

(15)

Figure 2.2 Diagram of a typical neuron. (Image taken from Wikipedia.)

biophysical mechanisms to simplified models involving a small number of differential equations.

These simplified models may seem unrealistic but are very useful for the study and analysis of large neural systems.

In this chapter we are going to present the Integrate-and-Fire model, one of the most widely used neuron models, which uses just one differential equation to describe the membrane potential of a neuron in terms of the current it receives (injected current and synaptic inputs). This is the model we are going to use for the multiplication networks in this thesis. Before it we will describe some underlying biological concepts like the anatomy of neurons and the electrical properties of the membrane.

2.2 Biological Background

Before describing the Integrate-and-Fire model it would be helpful to give some biological background about neurons and biological cells in general. In this section the anatomy of neurons is described along with the structure of cellular membranes, the operation of ion channels which are responsible for spike generation and finally the synapses and synaptic transmission.

2.2.1 Anatomy of a Neuron

Neurons are electrically excitable cells in the nervous system that process and transmit information. They are the most important units of the brain and of the whole nervous system. There is a wide variety in the shape, size, and electrochemical properties of neurons which can be explained by the diverse functions they perform.

In Figure 2.2 we can see a diagram of the anatomy of a typical neuron. The soma is the central part of the neuron where all the “computational” procedures like spike generation occur.

(16)

Section 2.2 - Biological Background 5 Several branched tendrils are attached to neurons. Each neuron has multiple dendrites which play a critical role in integrating synaptic inputs and in determining the extent to which action potentials are produced by the neuron.

There is just one axon which is a long nerve fiber, which can extend tens, hundreds, or even tens of thousands of times the diameter of the soma in length. In contrast with dendrites, the axon conducts electrical impulses away from the neuron’s cell body acting as a transmission line.

Action potentials almost always begin at the axon hillock (the part of the neuron where the soma and the axon are connected) and travel down the axon.

Finally synapses pass information from a presynaptic cell to a postsynaptic cell. We will see synapses and synaptic transmission in more detail in a following paragraph.

2.2.2 Membrane and Ion Channels

The cell membrane is a selectively permeable lipid bilayer found in all cells. It contains a wide variety of biological molecules, mainly proteins and lipids, which play a significant role for many cellular processes such as ion channel conductance and cell signaling.

In Figure 2.3 we can see an illustration of the membrane. The spherical lipid bilayer is ap- proximately 7 nm thick and is responsible for regulating the movement of materials into and out of cells. Except the phospholipid bilayer, we can see that the membrane includes several membrane proteins which determine the selective permeability of the membrane and passive and active transport mechanisms.

Figure 2.3 Illustration of a cell membrane. We can see the phospholipid bilayer and some of the proteins, lipids and other biological molecules that it contains. Among these proteins we can see an ion channel. (Image taken from Wikipedia.)

(17)

The most important proteins for neural functionality are the ion channels, integral membrane proteins through which ions can cross the membrane. There are plenty such channels, most of them being highly selective and allowing only a single type of ion to pass through them. The phospholipid bilayer is nearly impermeable to ions so these proteins are the elementary units underlying principal functionalities such as spike generation and electrical signaling (within and between neurons).

2.2.3 Synapses

Synapses are specialized junctions responsible for the communication between neurons. There are two main types of synapses, the chemical ones and the electrical synapses which are also known as gap-junctions [6]. Chemical synapses are the most important and most numerous in the nervous system. Despite gap junctions are very important parts of the nervous system (for example they are particularly important in cardiac muscle [25]), in this thesis we will assume that only chemical synapses are present on the dendritic tree. In the following paragraphs we will briefly describe how a synapse works.

In chemical synapses transmission is mediated by a chemical, called neuro-transmitter [31].

Synaptic transmission begins when an action potential reaches the presynaptic axon terminal.

The occurring depolarization of the presynaptic membrane initiates a sequence of events leading to neurotransmitter release and activation of receptors on the postsynaptic membrane. An illustration of a synapse can be seen on Figure 2.4.

Figure 2.4 Illustration of a synapse and synaptic transmission. (Image taken from [21].) Axon terminal contains a “pool” of synaptic vesicles which are little balls filled with neurotransmitter. When the axon terminal is depolarized, voltage-gated calcium (Ca) channels open and calcium ions (Ca²⁺) rush into the axon terminal. Some of these ions bind to synaptic vesicles bringing them closer to the presynaptic membrane, causing a fusion between the vesicle and synaptic membrane and finally the neurotransmitter is released which goes into the extracellular space.

Some of the neurotransmitter molecules bind to special receptor molecules on the postsynaptic membrane. The response of the postsynaptic cell varies, since it depends on the kind of transmitter-receptor combination. For excitatory synapses, the neurotransmitter causes the opening of channels which let through ions of potassium K⁺ and Sodium Na⁺. On the other hand inhibitory synapses activate mainly Chlorine channels Cl⁻.

(18)

Section 2.3 - Electrical Properties of Cells 7

2.3 Electrical Properties of Cells

A neural cell can be modeled using electrical components like resistors, capacitors and voltage sources. The occurring electrical circuits are used for computational simulations and approach sufficiently the behavior of real cells.

2.3.1 Membrane Voltage - Resting Potential

If one measures the intracellular (Vi) and extracellular (Ve) potentials of a neuron, one will observe the existence of a voltage difference (Vm) across its membrane.

Vm(t) = Vi(t) − Ve(t) (2.1)

Different intracellular and extracellular concentrations of ions are responsible for this voltage.

Most of the times Vm is negative (except when a spike occurs).

If the neuron is in rest (the sum of ionic currents flowing it and out of the membrane is zero) then the electrical potential across the membrane is called resting potential Vrest. For a typical neuron Vrest is about −70 mV .

2.3.2 Spike Generation

Figure 2.5 Generation of an action potential.

The uniqueness of neurons is their ability to produce, propagate and decode spike trains.

Before presenting the way a neural cell can be represented electrically it would be interesting to see how a spike is generated.

If we inject the neuron with current the Vm increases. When the membrane potential reaches a certain threshold, enough voltage- gated sodium channels open and the relative ionic permeability favors sodium (Na) over potassium (K). The number of channels that open depends on the injected current and equivalently to the voltage applied at the membrane. This explains why higher potential cause faster spikes. When the cell is in rest there is a negative electrical potential inside it.

The opening of the channels makes the Na⁺ ions flow into the cell, causing a rapid depolarization of the membrane. The flow of positive charged ions inside the cell leads the membrane to a potential close to EN a. After it, the voltage gated sodium channels inactivate and the voltage-gated potassium channels open. As a result K⁺ ions rush out of the cell through the open channels causing the membrane potential to become negative again. Since at this time, there is very little sodium permeability the potential approaches EK causing a hyperpolarization close to the resting potential until the potassium channels close again. This is the process of a spike generation. Sodium channels cannot be activated again until some time has passed (this time is known as the absolute refractory period).

(19)

2.3.3 Membrane Capacitance & Resistance

Capacitance Cm

The neuron membrane, as we have already seen, is an insulating layer consisting mainly of lipids and proteins. However both the intracellular and extracellular solutions contain ions and have conducting properties. So the role of the insulating membrane is “equivalent” to that of a capacitor on an electrical circuit.

The actual membrane capacitance Cm is specified in terms of the specific capacitance per unit area cm, measured in units of Farad per square centimeter (F/cm²). If A is the area of a cell (in cm²) then the actual capacitance Cm (in F ) is given by :

Cm = cm· A. (2.2)

Cm is proportional to membrane area A, so the bigger the neuron the larger its capacitance. Given that the charge distributed on a surface is proportional to the capacitance (Q = CV ), we can see that larger neurons have bigger amounts of ions (charge) distributed across their membranes. A typical value for the specific capacitance cm, which was used in our simulations, is 1 µF/cm².

Resistance Rm

The ion channels allow the ionic current to flow through the cell’s membrane. Since there is a difference between the membrane voltage Vm and the resting voltage Vrest of the cell we can model the current flow through the ionic channels with a simple resistance Rm.

The actual membrane resistance Rmis specified in terms of the specific resistance (or resistivity) rm, measured in units of ohms-square centimeter (Ω · cm²). If A the area of a cell (in cm²) then the actual resistance Rm (measured in Ω) is given by :

Rm = rm

A . (2.3)

We can see that Rm is inversely proportional to membrane area A, so big neurons are more leaky than smaller cells. A typical value for the resistivity rm, which was used in our simulations, is 20 kΩ · cm².

2.3.4 Synaptic Reversal Potential and Conductance

An ionic reversal potential V_syn^rev is associated to every synapse. At this potential there is no net flux of ions through the ionic channel and the membrane potential across it is stabilized to V_syn^rev [17]. For an excitatory synapse the reversal potential is about 0 mV while for an inhibitory one V_syn^rev has a value close to the neuron’s resting potential (−70 mV ).

It has been experimentally observed that spiking activity on the presynaptic cell causes a conductance change in the membrane of the postsynaptic cell. This synaptic conductance gsyn(t) depends on the presence of presynaptic action potentials and changes with time. It increases almost instantly to a maximum value g0 and then subsides exponentially within a time period of 5 ms. This is the synaptic time constant τsyn.

Although ionic channels and synaptic transmission is a highly nonlinear phenomenon, the presence of a synapse in a membrane clatch can be modeled satisfactory with the synaptic conductance gsyn(t) in series with the synapse’s reversal potential V_syn^rev.

(20)

Section 2.4 - The Integrate-and-Fire Model 9

2.3.5 Electrical Structure of Neurons

Using the aforementioned electrical properties of neural cells, we can describe the dynamics of the membrane potential Vm(t) in response to the input current using a single RC circuit. The existence of a chemical synapse can be modeled by adding the synaptic conductance gsyn(t) and the reversal potential V_syn^rev in parallel with the RC circuit.

R

V

I Cm

m

rest

inj V_m

(a) Simple RC circuit

R

V C_m

m

rest

V_m Vrev

gsyn( t )

(b) With synapse

Figure 2.6 Equivalent electrical circuits of a simple neuron (a) and a neuron with a fast chemical synapse (b).

2.4 The Integrate-and-Fire Model

The Integrate-and-Fire (I&F) is a very simple neuron model used widely to simulate and analyse neural systems [3]. Despite its simplicity the I&F model captures key features of real neuron’s behaviour, like the rapid spike generation. The Integrate-and-Fire model emphasizes on the subthreshold membrane voltage properties and doesn’t take into account complex mechanisms responsible for spike generation like the ionic channels. The exclusion of such difficult to model biophysical mechanisms, makes the IF model capable of being analysed mathematically and ideal for simulations including large numbers of neurons. Other neuron models like the Hodgkin-Huxley model [15] although they capture in a better way the biological mechanisms, are too complex to be used in computational simulations of larger networks. For example the Hodgkin-Huxley model describes both the subthreshold and the spiking behavior of membrane potential but is using four coupled differential equations.

In 1907 Lapicque [19] introduced the I&F model which is a passive circuit consisting of a resistor and a capacitor in parallel, which represent the leakage and capacitance of the membrane.

In this simple model the capacitor is charged until a certain voltage threshold is reached. At this point a spike occurs (the capacitor discharges) and the voltage is reset to a specific value (Vreset).

There are two basic versions of the Integrate-and-Fire model which are described below.

2.4.1 Nonleaky Integrate-and-Fire Neuron

The nonleaky (or perfect) I&F model includes only a single capacitance C which is charged until a fixed and stationary voltage threshold Vthr is reached.

This model doesn’t take into account the membrane resistance and as a result the leaking current which makes it unphysiological. However it is very simple to be described mathematically.

Assuming an input current I(t) the differential equation governing the voltage is :

(21)

CdV (t)

dt = I(t) (2.4)

When Vth is reached at time ti a spike δ(t − ti) is triggered and voltage is reset to Vreset. For tref seconds following the spike generation any input is shunted to ground making another spike during the absolute refractory period impossible [17].

2.4.2 Leaky Integrate-and-Fire Neuron

In the more general leaky model, the summed contributions to the membrane potential decay with a characteristic time constant τm which is called the membrane time constant. Again when the membrane voltage Vm reaches a fixed threshold Vthr an action potential is initiated. After the spiking the voltage is reset to a resting value Vrest and the neuron is inactivated for a brief time corresponding to the absolute refractory period.

The model is described by the following differential equation:

Cm

dVm(t)

dt = Ileak(t) + Inoise(t) + Iin(t) (2.5) where Ileak(t) the current due to the passive leak of the membrane, Inoise(t) the current due to noise (0 for non noisy neurons) and Iin(t) the input current (injected through an electrode Iinj(t) and/or through synaptic input Isyn(t)). So there are two components for Iin(t):

Iin(t) = Iinj(t) + Isyn(t). (2.6) The leaking current is given by the equation :

Ileak(t) = − 1 Rm

[Vm(t) − Vrest] = −Cm

τm

[Vm(t) − Vrest] (2.7) where τm = RmCm the passive membrane time constant depending solely on membrane’s capacitance Cm and leak resistance Rm. For our simulations we used a membrane time constant of τm = 20 ms.

2.4.3 Synaptic Input

Although the study of neuron’s response to injected current pulses and noise is interesting from an experimental perspective, it is not realistic. In a real cell the main source of “input current” is synaptic input.

Each neuron is synaptically connected to multiple other neurons through its dendrites. When an external stimulus is presented to an organism (for example a visual stimulus) some cells activate and the generated spike trains propagate through the axons of the activated neurons, acting as inputs to the cells connected on them.

Assuming a presynaptic spike at time tspike, the postsynaptic current Isyn(t) applied on the neuron at time t can be given by the following exponential equation, describing an AMPA synapse

Isyn(t) = g(t) V_syn^rev − Vm(t)

(2.8) where the synaptic conductance g(t) is given by

g(t) = g0e⁻^t−tspike^τsyn . (2.9)

In the previous equations, V_syn^rev is the synapse’s reversal potential, g0 the maximum synaptic conductance and τsyn the synapse’s time constant.

(22)

Chapter 3 Multiplication in the Nervous System

3.1 Introduction

In the literature experimental evidence can be found that supports the existence of multiplicative mechanisms in the nervous system. Studies have shown that the optomotor control in the fly is controlled by neural circuits performing multiplication [12], [14]. More recent experiments have found a multiplicative like response in auditory neurons of the barn owl ’s midbrain [23] .

The exact way multiplication is implemented in neurons is unclear. However there is a lot of interest about its details, driven by the experimental observations, which imply its existence.

Koch and Poggio [18] have discussed different biophysical properties, present in single cells, capable of producing multiplicative interactions. Also in the literature some other neuronal models implementing multiplicative operations can be found (for example [27]).

In this chapter we will initially try to show why multiplication is important, and how it could play central role in decision making and perceptive tasks. Following we present biological evidence of multiplicative operation in the neural system and in the end we describe some of the models that can be found in literature.

3.2 Importance of Multiplication

The simplest neuron models operate under a regime of thresholding: if the sum of all inputs, excitatory and inhibitory (inhibitory synapses have a negative weight while excitatory a positive one) exceeds a certain threshold then the neuron is active, otherwise there is no spike generation.

This binary threshold function is the only nonlinearity present in the model. In artificial neural networks sigmoid functions are used to give a smoother input-output relationship.

The threshold function may be the dominant nonlinearity present in neurons but it is not the only one. As we will see on the next section literature is full of experimental evidence that supports the presence of multiplicative operations in the nervous system. Given that multiplication is the simplest possible nonlinearity, neuronal networks implementing multiplicative interactions can process information [18].

Below we will try to show how powerful this simple operation is and we will highlight its connection with the logical AND operation. We will also see how important multiplication is for decision making tasks.

3.2.1 Function Approximation

The Weierstrass approximation theorem states that every continuous function defined on an interval [a, b] can be uniformly approximated as closely as desired by a polynomial function. More

11

(23)

formally the theorem has the following statement:

Theorem Suppose f is a continuous complex-valued function defined on the real interval [a, b].

For every ǫ > 0, there exists a polynomial function p over C such that for all x in [a, b], we have

|f (x) − p(x)| < ǫ, or equivalently, the supremum norm ||f − p|| < ǫ.

If f is real-valued, the polynomial function can be taken over R.

The only nonlinear operation present in the construction of a polynomial is multiplication.

As a result if neural networks are capable of doing multiplicative-like operators then they could approximate under weak conditions all smooth input-output transductions [18].

A polynomial can be expressed as the sum of a set of monominals. A monominal of order k can be modeled with a multiplicative neural unit which has k inputs.

P (x) = a1+ b1x1+ b2x2+ c1x²₁+ c2x1x2+ .... (3.1)

3.2.2 Relationship Between Operators

In order to understand the importance of multiplication we should first understand that multiplication is in fact a close relative of another far more fundamental operation, the logical AND (∧) operation. In Boolean algebra x1 ∧ . . . ∧ xi∧ . . . xn is true only if xi is true for all i. If there exists some xi which is false then the whole expression is false. This “behavior” is similar to the multiplication with zero in classical algebra: x · 0 = 0, ∀x ∈ R. More strictly, the behavior of the

∧ operator is similar to the minimum function.

On the other hand, the second more common Boolean operation OR (∨) can be parallelized with addition or more strictly with the maximum function. On the following table we illustrate these relations, in a truth-like table form.

Relationship Between Operators

x y x ∧ y x × y min(x, y) x ∨ y x + y max(x, y)

0 0 0 0 0 0 0 0

0 1 0 0 0 1 1 1

1 0 0 0 0 1 1 1

1 1 1 1 1 1 2 1

3.2.3 Multiplication and Decision Making

The parallelism between multiplication and the logical AND operation could explain the importance of multiplicative neural mechanisms from a decision making perspective. Logical AND (∧) operations are fundamental in such tasks and generally in the organization of perception. We will try to demonstrate this importance with a simplified example.

Imagine a bird whose diet includes an edible red flower. In order to discriminate this flower from another similar one our bird detects its characteristic odour. So it eats only these red flowers which also have the desired smell. This is nothing more than an AND based perceptive task.

Assume that there are two regions in the bird’s neural system, one responsible for recognising the red color and the other for recognising the desired odour. The outputs of these neural regions are combined in a third region which decides if a specific flower is edible or not. If the two outputs were just added, then a very strong output of the “color detector” would produce a stronger output signal than a modest output of both detectors. That could lead the bird to eat a poisonous flower.

If on the other hand the outputs of the two detectors are multiplied then the performance would be better. The absence of one feature (color or odour) would suppress the output and prevent the

(24)

Section 3.3 - Biological Evidence of Multiplication 13 bird from classifying the flower as edible. If on the other hand both features are present but weak then the multiplicative operation would lead to a supra-linear enhancement of the output signal.

Through this intuitive example we showed that perceptive tasks which include ∧ operations can modeled better using multiplication than simple addition. However it is not known to what extent multiplicative like mechanisms are present in the neural system. In the next section we do a literature research, presenting evidence of such multiplicative behaviors. However for binary signals, when imposing a threshold, the difference between the AND operation and addition is minor.

3.3 Biological Evidence of Multiplication

Multiplicative operations are thought to be important in sensory processing. Despite the research on this topic is limited, there is significant experimental evidence that reinforces the ideas for multiplicative biophysical mechanisms. The most interesting clue of multiplicative properties of neurons can be found in the auditory system. There is also evidence that multiplication is carried out in the nervous system for motion perception tasks [18]. In the following sections we will present these clues, trying to underline the importance of multiplication.

3.3.1 Barn Owl’s Auditory System

Barn owls are able to use their very accurate directional hearing to strike prey in complete darkness. This impressive capability is based on a very complex auditory system, barn owls have, which among other specializations includes asymmetric external ears.

As a consequence of this asymmetry, the owl’s auditory system computes both interaural time (ITD) and level (ILD) differences in order to create a two dimensional map of auditory space [22].

Interaural level differences (ILDs) vary with elevation, allowing barn owls to use ILDs, in order to localize sounds in the vertical plane. Similarly interaural time differences (ITDs) are used for localization in the horizontal plane.

Neuronal sensitivity to these binaural cues first appears in the owls brainstem, with separate nuclei responsible for processing ILDs and ITDs. Both ITDs and ILDs information are merged in space-specific neurons that respond maximally to sounds coming from a particular direction in space. The parallel pathways that process this information merge in a region known as the external nucleus of the inferior colliculus (ICx), eventually leading to the construction of a neural map of auditory space (see Figure 3.1).

The research of Pena and Konishi [22], suggests that the space-specific neurons in the barn owl ICx tune at the location of an auditory stimulus by multiplying postsynaptic potentials tuned to ITD and ILD. So the subthreshold responses of these neurons to ITD-ILD pairs have a multiplicative rather than an additive behavior.

Owls were anesthetized and postsynaptic potentials generated by ICx neurons in response to different combinations of ITDs and ILDs, were recorded with the help of intracellular electrode recordings. Acoustic stimuli were digitally synthesized with a personal computer and delivered to both ears by calibrated earphone assemblies giving rise to the various ITD-ILD pairs [23]. The researchers discovered that a model based on the product of the ITD and ILD inputs could account for more of the observed responses. An additive model was also tested but it was not efficient and could not reconstruct the original data matrix as well as the multiplicative model. In Figure 3.2 we can see the success of the multiplicative model in reconstructing the measures membrane potential for different ITD-ILD pairs.

(25)

Figure 3.1 Space-specific neuron in barn owl ’s auditory system, that respond maximally to sounds coming from a particular direction in space. (A) A drawing of an ICx neuron and its axon projecting to the optic tectum (OT ). (B) The same neuron labeled with neurobiotin. (C) Postsynaptic potentials in response to different ITD-ILD pairs. Dotted lines indicate the mean resting potential. (D) Spiking responses of the same neuron to different ITD-ILD pairs. The large peak is the excitatory center and the flat area around it is the inhibitory surround [compare (C) and (D)]. Negative (-)ITD and negative (-)ILD mean, respectively, sound in ipsilateral ear leading and louder. (Figure and caption taken from [22].)

3.3.2 The Lobula Giant Movement Detector, LGMD, of Locusts

Gabbiani et al. [11] mention that there is evidence for the existence of a multiplicative operation in the processing of looming stimuli. They experimented with a neuron in the locust visual system (the LGMD neuron) that responds well to objects looming on a collision course towards the animal. Multiplication could be used for the computation of an angular threshold that could prevent collision with looming objects.

The firing rate of the LGMD neuron was monitored by recording the action potentials of its post-synaptic target neuron (DCMD). The insect was presented with black squares or disks on a bright background, which simulated approaching. The monitored firing rate a firing rate initially increased, until a peak and finally it decayed as the approach ended.

(26)

Section 3.3 - Biological Evidence of Multiplication 15 If we denote by l the looming object half-size, by t the time to collision and by v its approach velocity, then the angular size is given by [10] :

θ(t) = 2arctan l vt

(3.2)

In a beautiful analysis the researchers suggest that “the angular threshold might be the image- based retinal variable used to trigger escape responses in the face of an impending collision. Indeed, a leg flexion (presumably in preparation for an escape jump) has been shown to follow the peak LGMD firing rate with a fixed delay” [11].

The researchers tried to figure out how the the angular threshold is calculated by the insect’s nervous system. They tried different models which were based on the size of the forthcoming object and the velocity, that could describe the recorded responses of the LGMD. One input was excitatory and the other one inhibitory. By using selective activation and inactivation of pre and postsynaptic inhibition, they found out that postsynaptic inhibition played a very important role, suggesting that multiplication is implemented within the neuron itself [10]. Experimental and theoretical results are consistent with multiplication being implemented by subtraction of two logarithmic terms followed by exponentiation via active membrane conductances, according toa × ¹_b = exp(ln(a) − ln(b)). In Figure 3.3 we can see some of their results.

Figure 3.2 Multiplicative combination of ILD and ITD inputs. (A) Raw data matrix. (B) Reconstruc- tion of the matrix from the computed left and right singular vectors and the first singular value. Addition of V0 [DC offset (blue area)] that minimizes the second singular value almost restores the original matrix.

(C) ITD curve. (D) ILD curve. (E) Computed left singular vector. (F) Computed right singular vector.

(Figure and caption taken from [22].)

(27)

Figure 3.3 Transformation between membrane potential (Vm) and firing rate at the spike initiation zone. (a) Approaching stimulus (top), recordings from the DCMD (middle, extracellular) and from the LGMD (bottom, intracellular) close to its spike initiation zone (b, inset). Orange trace is membrane potential after median filtering (V_m). Inset, bracketed portion of V_m and (V_m) expanded 3 times. (b) Top panel presents median filtered membrane potential (orange line is same trace as in a; 5 repetitions).

Bottom 5 traces were recorded after TTX application to the axon (inset). (c) Mean traces in control and TTX (from b) were fitted with a third-order polynomial (black) and used to compute the mean temporal difference (352 ms) in membrane potential over the response rising phase. (d) Fit of mean instantaneous firing rate, hgi, as a function of mean, median filtered membrane potential (mean ± s.d.; solid and dotted black lines) with linear, third-power and exponential models. (Figure and caption taken from [11].)

3.3.3 Other Evidence

Multiplicative evidence can also be found in the optomotor control of the fly [12], [14]. The fly’s optomotor response to transient stimuli was studied under open loop conditions. The stimuli used were moving edges and stripes. A comparison of the fly’s responses to these stimuli led the researchers to the result that progressive moving patterns elicit stronger responses (from front to back) than regressive moving ones (from back to front). The existence of such deviations in fly’s response to different moving edges suggests the evidence of nonlinearities in insect’s perceiving system.

(28)

Section 3.4 - Existing Models 17 Research done by Reichardt [24] has also suggested that the optomotor response of insects to moving stimuli is mediated by a correlation like operation which can be seen abstractly seen as a form of multiplication. Similar experimental observations exist for pigeons as well [28].

Finally Andersen [1] reviews some research papers, which indicate that “sensory signals from many modalities, converge in the posterior parietal cortex in order to code the spatial locations of goals for movement. These signals are combined using a specific gain mechanism. ”

3.4 Existing Models

In the literature there are some papers which propose models for multiplicative neural operations. Most of these models are single cell’s specific biophysical mechanisms which could give rise to a multiplicative-like operation. However we should note that the research in this field is limited, despite the importance of understanding how multiplicative-like operations are neurally implemented. In our proposal we won’t concern with single cell models, but with small feed forward networks of Integrate-and-Fire neurons. On the following paragraphs we will briefly explain some of the multiplicative models found in literature.

3.4.1 Multiplication via Silent Inhibition

Silent inhibition in some special cases can give rise to a multiplicative behavior. We have seen in the previous chapter that synaptic current is given by:

Isyn(t) = gsyn(t) V_syn^rev− Vm(t) . (3.3) If we suppose that the synaptic input changes slowly [18] then we can assume that the synaptic conductance gsyn(t) changes slowly with time. As a result there will be a stationary current and gsyn will be the constant synaptic input. If Rsyn the synaptic resistance then using Ohm’s law, V = R · I we take the following equation for the membrane voltage:

Vm = gsynRsynV_syn^rev

1 + RsynV_syn^rev. (3.4)

If the synaptic reversal is close to the resting potential of the cell (shunting inhibition) then the action of this synapse to Vm remains invisible.

From the previous equation we can take a multiplicative relation if we assume that the product of the synaptic resistance and synaptic conductance is small, gsynRsyn ≪ 1 :

Vm ≈ gsynRsynV_syn^rev. (3.5)

If we also have an excitatory synaptic input with an associated conductance change ge and a reversal potential V_exc^rev then using Taylor expansion we take [18]:

Vm ≈ V_exc^revRsyn ge− g²_e− gegsynRsyn+ . . .

(3.6) which includes quadratic contributions from the excitatory synaptic terms and higher order terms from combinations of the excitatory and inhibitory inputs.

3.4.2 Spike Coincidence Detector

Srinivasan and Bernard [27] used an input spike coincidence detector in order to model multiplication like responses. The main aim of the authors was not to model exact multiplication

(29)

but to describe a scheme by which a neuron can produce a response which is proportional to the product of the input signals that it receives from two other neurons.

They investigated a neuronal model in which the neuron produces a spike only if it receives two spikes, from the two external neurons, that are coincident in time or nearly so. In Figure 3.4 we can see how such a neuron operates.

Figure 3.4 Neuron C receives input from two neurons A,B. Cell C fires a spike only if two input action potentials arrive within a ∆ ms. Only in this case the voltage membrane reaches the threshold. As a result the output firing rate of neuron C is proportional to the firing rates of A,B. (Figure taken from [27].)

In order to model coincidence detection, the proposed neuron spikes when its membrane voltage Vm is above a certain threshold Vthr. The presence of only one presynaptic spike cannot cause enough EPSP to discharge the cell, but if two spikes arrive within ∆ ms then the voltage threshold is reached and an output spike is generated. If Vmax the maximum membrane potential the neuron can reach from a single input spike, then there is an exponential decaying relation between membrane voltage and time,

V (t) = Vmaxe⁻^τ^t (3.7)

where τ the neuron’s time constant. The authors make the assumption that Vmax< Vthr < 2Vmax

so a single spike cannot initiate a postsynaptic action potential. If we have an input spike at time t0 there should be another spike in an interval of ∆ ms before (or after) t0 ([t0 − ∆, t0 + ∆]) in order to have the postsynaptic action potential. Given two spikes at t and t + ∆ then the neuron will fire an action potential and as a result ∆ can easily be determined by the equation:

Vmaxe⁻^∆^τ + Vmax = Vthr. (3.8)

The authors assumed statistical independence of the two input firing rates (a natural assumption in most cases, for example when the stimuli causing activation of the two presynaptic cells

(30)

Section 3.4 - Existing Models 19 are independent) and showed that the output firing rate is proportional to the product of the two input firing frequencies [27]:

fout = 2∆fAfB (3.9)

(31)

(32)

Chapter 4 Multiplication with Networks of I&F Neurons

4.1 Introduction

In the previous chapter we presented evidence of multiplicative behavior in neural cells. We also argued for the importance of this simple nonlinear operation. Despite its simplicity it is unclear how biological neural networks implement multiplication. Also the research done in this field is limited and the models found in bibliography (we presented some of them in the previous chapter), are complex single cell biophysical mechanisms.

We try to approach multiplication using very simple networks of Integrate-and-Fire neurons and a combination of excitatory and inhibitory synapses. In this chapter we are going to present the underlying theory and the proposed models. We also analyze in depth the main idea behind this dissertation which is the usage of the minimum function for implementing a neural multiplicative operator.

4.2 Aim of the Thesis

The aim of this thesis is to find feed-forward networks of Integrate-and-Fire neurons which do multiplication of the input firing rates. The problem can be defined as follows:

Problem

Given two firing rates ρ1, ρ2 [in Hz], find a network of Integrate-and-Fire neurons whose output spike train has a firing rate ρout where

ρout = ρ1 · ρ2 (4.1)

In the next sections we will see that the exact multiplication is not possible so we will try to approximate it. Before presenting the proposed networks we will give the definitions for firing rates and rate coding.

4.3 Firing Rates and Rate Coding

The way information is transmitted in the brain, is a central problem in neuroscience. Neurons respond to a certain stimulus by the generation of action potentials which are called spike trains.

Spike trains are stochastic and repeated presentation of the same stimulus don’t cause identical firing patterns. It is believed that information is encoded in the spatiotemporal pattern of these

21

(33)

trains of action potentials. There is a debate between those who support that information is embedded in temporal codes and researchers in favor of rate coding.

In rate coding hypothesis, one of the oldest ideas about neural coding [30], information is embedded in the mean firing rates of a population of neurons. On the other hand, temporal coding relies on precise timing of action potentials and inter-spike intervals.

4.3.1 Firing Rates

Suppose that we record the output of an integrate-and-fire neuron, for a specific time interval of duration T . In total n spikes are observed which occur at times ti, i = 1, . . . , n. Then the neural response r(t) can be represented as a sum of Dirac functions:

r(t) =

n

X

i=1

δ(t − ti) (4.2)

The specific timing of each action potential is useful only if we use temporal coding. In this thesis we study the multiplication of firing rates, so the times ti are not useful. Due to their stochastic nature, neural responses can be characterized by firing rates instead of specific spike trains [5].

Figure 4.1 Firing rates approximated by different procedures. (A) A spike train from a neuron in the inferotemporal cortex of a monkey recorded while that animal watched a video on a monitor under free viewing conditions. (B) Discrete time firing rate obtained by binning time and counting spikes for

∆t = 100 ms. (C) Approximate firing rate determined by sliding a rectangular window function along the spike train with ∆t = 100 ms. (D) Approximate firing rate computed using a Gaussian window function with σt = 100 ms. (E) Approximate firing rate using the window function w(τ ) = α²τ exp(−ατ )

+, where 1/α = 100 ms. (Figure and caption taken from [5].)

If there is low variability in the spiking activity then the firing rate can be accurately approached by the spike count rate which is nothing more than the frequency if the n action potentials during a time T .

(34)

Section 4.4 - Excitation vs. Inhibition 23

ρ = n T = 1

T Z T

0

r(t)dt (4.3)

Of course if there is variability in the frequency of spikes this approximation is not sufficient and a time-dependent firing rate should be used. The firing rate at time t in this case, can be defined as the time of spikes between t and t + ∆t, where ∆t a small time interval [5]. So mathematically we can express ρ(t) as:

ρ(t) = 1

∆t

Z t+∆t t

hr(t)idt (4.4)

where the trial averaged neural response hr(t)i is the sum of spike occurrences over the interval [t, t + ∆t] for K trials, divided by the number of trials K.

4.3.2 Rate Coding

According to the rate coding hypothesis, information is encoded in the mean firing rate of a population of neurons and not the exact timing of the action potentials. It has been observed that different stimuli cause spike trains of different frequencies on the same neurons. Generally as the intensity of a stimulus increases so does the frequency or rate of action potentials. These experimental observations have lead to the formation of rate coding hypothesis.

Theunissen [29] defines rate coding as “a coding scheme in which there is significant correlation between the relevant stimulus parameter and the mean number of spikes in the elicited response within the encoding window and no additional correlation between the stimulus parameter and any higher-order moments (or higher-order principal components) of the elicited spike pattern”.

Unlike rate coding [32], temporal coding relies on precise timing of action potentials or inter- spike intervals. Combined with traditional rate coding models, temporal coding can provide additional information with the same rate. There is strong evidence that temporal codes are used for cortical information transmission, especially after the discovery of spike timing dependent plasticity [26].

Theunissen [29] defines temporal coding as “a coding scheme in which there is significant additional correlation between the relevant stimulus parameter and any moments of the elicited spike pattern having higher order than the mean”.

4.4 Excitation vs. Inhibition

We have seen in Chapter 2 that the main difference between an excitatory and an inhibitory synapse is the reversal synaptic potential. We are going to analyze on this section how the type of the synapse affects the response of the cell for an input spike train.

The synaptic current Isyn(t) is given by the following equation and depends on the synaptic conductance g(t) and the difference between the reversal potential V_syn^rev and the membrane voltage Vm:

Isyn(t) = g(t) V_syn^rev − Vm(t)

(4.5) where g(t) changes with time as we have already seen. Let a time moment t0 where g(t0) = gt0 = 50 pS. For this time moment we plot the synaptic current for different values of the reversal potential and the membrane voltage.

For an inhibitory synapse the reversal synaptic potential is close to the resting potential of the neuron Vrest, which in most cases is about −70 mV . On the other hand the reversal potential for an excitatory synapse is close to 0 mV . In Figure 4.2 we can see that as V_syn^rev increases, so does Isyn. For values of the reversal potential close to the ones observed on inhibitory synapses we

(35)

−100

−80

−60

−40

−20 0

20

−70

−65

−60

−55

−50

−3

−2

−1 0 1 2 3 4

x 10⁻¹²

Reversal Potential Membrane Voltage

Synaptic Current

Figure 4.2 Plot of the synaptic current Isyn(t) for different values of the reversal synaptic potential V_syn^rev and the membrane voltage Vm. The synaptic conductance was assumed constant to 50 pS.

have negative synaptic current, while a reversal potential of 0 mV can produce synaptic current of about 3 pA.

The positive synaptic current for an excitatory synapse has the following effect : an action potential in the presynaptic cell increases the probability of an action potential occurring in the postsynaptic cell. On the other hand the minor synaptic currents caused by inhibitory synapses, result in inconsiderable changes in the membrane voltage Vm and as a result to low firing possibili- ties. Of course the existence of neural noise can cause the firing of a spike even if there is no input current. In the case of both excitatory and inhibitory inputs on a neuron, the negative inhibitory current acts subtractively to the positive current of excitation. In Figure 4.3 we can see how the firing sequence of a neuron depends on the kind of the synapse, for the same spike train input.

S y n a p t i c I n p u t

C e l l R e s p o n s e

I & F

Excitatory

Inhibitory

Figure 4.3 Output spike train of an I&F neuron after stimulation with the sequence of action potential presented on the left. We can see how the output depends on the kind of the synapse.

4.4.1 Subtractive Effects of Inhibitory Synapses

A synapse is called inhibitory when its reversal potential is less than the threshold for action potential generation. If it is close to the resting potential of the cell then this is called shunting inhibition and has a divisive effect on subthreshold EPSP amplitudes [7].

However this divisive effect is not observed in firing rates. Holt and Koch [16] have shown that there is a subtractive effect on the firing rate. This happens because the current that passes

(36)

Section 4.5 - Rectification 25 through the shunting conductance is independent of the firing rate. The voltage at the shunting site cannot take a larger value than the spiking threshold and as a result the inhibitory synaptic current is limited for different firing rates. Under these circumstances a linear subtractive operation is implemented.

We simulated an Integrate-and-Fire neuron which had an excitatory and an inhibitory synapse.

The reversal potential of the inhibitory one was close to the resting potential of the neuron making it shunting. On the following diagram we can see that there is indeed a subtractive effect on firing rates. The firing rate of the spike train used as input to the excitatory synapse is 80 Hz while the inhibitory firing rates change from 0 to 80 Hz. The firing rates of the output spike train are plotted, among with the actual difference of the two input firing rates ρexc− ρinh.

0 10 20 30 40 50 60 70 80

0 20 40 60 80 100 120

Mean firing rate of the inhibitory input [in Hz]

Output firing rate [in Hz]

Figure 4.4 A noisy I&F neuron with two synaptic inputs (one excitatory and one inhibitory) was stimulated and the output firing rates were recorded. The firing rate of the excitatory input was kept constant to 80 Hz while we were increasing the firing rate of the inhibitory input from 0 to 80 Hz with a step of 5 Hz. With black circles we see the recorded output firing rates, while the red triangles are the difference between the excitatory and inhibitory firing rates ρ_exc− ρ_inh. We can see that there is a subtractive effect as expected.

4.5 Rectification

Let an Integrate and Fire neuron (I&F) which is stimulated with excitatory presynaptic action potential whose average firing rate is ρêxc_in and with inhibitory ones having a firing rate ρînh_in . The output spike train depends on both the excitatory and inhibitory inputs, and its firing rate is ρout. If ρînh_in = 0 Hz then output firing rate will be close to the firing rate of the spike train arriving at the excitatory synapse. On the other hand the absence of excitatory input, or the presence only of inhibitory presynaptic spikes, will deter the neuron from spiking.

(37)

If we have both excitatory and inhibitory synapses then as we have seen the inhibition has a subtractive effect. on firing rates. Since the firing rate of a neuron cannot take a negative value the output will be a rectified copy of the input:

ρout= max 0, ρêxc_in − ρînh_in = ρêxc_in − ρînh_in

+ (4.6)

where [·]+ stands for rectification. Since in our model we care only about firing rates (and not membrane voltage dynamics), we should note that rectification will be the only present nonlinearity in the approximation of multiplication.

A noisy integrate and fire neuron which has two synaptic inputs, one excitatory and one inhibitory, was simulated for different values of ρêxc_in and ρînh_in . The output firing rate ρout in all cases was close to the rectified difference of the two inputs ρêxc_in − ρînh_in as suggested by theory. The expected output firing rates and the recorded ones are presented on the following figure for both simple rectification and power-law Nonlinearity.

0 20

40 60

80 100

0 20 40 60 80 100

Excitatory firing rates [in Hz]

Inhibitory firing rates [in Hz]

Output firing rates [in Hz]

(a) Recorded Firing Rates

0 20

40 60

80 100

0 20 40 60 80 100

0 10 20 30 40 50 60 70 80

(b) Expected Firing Rates - Simple

0 20

40 60

80 100

0 20 40 60 80 100

0 100 200 300 400 500 600

(c) Expected Firing Rates - Power Law

Figure 4.5 An I&F neuron with two synaptic inputs was simulated for different input firing rates in order to examine if there is rectifying behavior. Both the excitatory and inhibitory inputs ρêxc_in , ρînh_in were gradually increased from 5 to 90 Hz with a step of 5 Hz. (a) Recorded output firing rates. (b) Expected input-output relation according to the equation ρ_out =ρêxc_in − ρînh_in

+, (c) Expected input-output relation according to the equation ρout=ρ^exc_in − ρ^inh_in 1.45

+ .

Multiplication With Neurons

School of Informatics