Supercomputer simulations of detailed and abstract neural network models

(1)

STOCKHOLMBRAININSTITUTE

Supercomputer simulations of

detailed and abstract neural

network models

Anders Lansner, Computational Biology, CSC

KTH and Stockholms University

(2)

2013-02-15

(3)

STOCKHOLMBRAININSTITUTE

}

V1 model

} 3.5x3.5 mm

} 0.67 degree visual angle } 479232 neurons } 43M synapses

}

Layer 4

} SS4, SB, LB, n = 258048

}

Layer 2/3

} PYR, LB, DBC, n = 221184

}

10 s simulated time

}

2 h 20 min wall time

}

Many runs to tune model

}

A BrainScaleS demo

} + I-F version

} Æ Neuromorphic HW

2013-02-15

Celsium-Linné symposium Uppsala

Lissom model (Sirosh and Miikkulainen 1994) Measured L4 orientation map

(4)

STOCKHOLMBRAININSTITUTE

And some example output ...

2013-02-15

(5)

STOCKHOLMBRAININSTITUTE

Brain simulation

A multi-scale approach to understand ...

Graham Johnson Medical Media, Boulder, Colorado

The most The most important important ” ”machinemachine””!! 2013-02-15

(6)

STOCKHOLMBRAININSTITUTE

Aims of brain simulation

}

Understanding

}

Help understand the brain by quantitative

modeling

}

Connecting neurons and synapses with

} Macroscopic measurements of brain activity } Cognitive phenomena and behaviour

}

The healthy and diseased brain, personalized

medicine

}

Mimicing = brain-like computing

}

Technological applications, AI type

} Artificial brains, bee, mouse ... human

}

Brain implants, prostheses to help the disabled

2013-02-15

(7)

STOCKHOLMBRAININSTITUTE

Why supercomputers?

}

Many neurons and synapses

in reality, network is key to

function

}

Highly sub-sampled network

models distort results

} 1000 neurons on PC slow

}

Æ

Complex multi-networks

} Need >10K spiking neurons for uniform network component

}

Coupled ODE:s + event based

communication parallelizes

very well!

}

What supercomputer

capacity?

2013-02-15

C Elegans Mouse Rat Cat Macaque Human 0 5 10 15 Animal species lo g (n u m b e r) neurons synapses density -6 -4 -2 0 log (de n s ity )

(8)

STOCKHOLMBRAININSTITUTE

Capacity demands for brain

simulation

2013-02-15

1980 1990 2000 2010 2020 2030 0 2 4 6 8 10 12

100 ops/synapse/ms

100 B/synapse

Tera Peta Exa year lo g( G iga ) GF GB 1980 1990 2000 2010 2020 2030 0 2 4 6 8 10 12

100 ops/synapse/ms

100 B/synapse

Tera Peta Exa year lo g( G iga ) GF GB

Neuromorphic

HW ...

(9)

STOCKHOLMBRAININSTITUTE

Memory recall and oscillations

}

Cortex simulations

} Blue Brain } IBM

} NEST groups

}

Typically randomly connected networks, dynamics, no

learning, no specific function

}

Cortical (active) memory function

}

Starting from Hebbs cell assemblies, theory of

attractor neural networks

}

Biolocal plausibility?Can real neuronal networks do

this?

} Our model of piece of mammalian cortex } Basic function, robust associative memory } Correlated oscillatory dynamics (γβΘ)

2013-02-15

(10)

STOCKHOLMBRAININSTITUTE

Donald Hebb

’

s brain

theory

Bliss and L

Bliss and Löömo, 1973mo, 1973 Levy and Steward, 1978

Levy and Steward, 1978

LTP

LTP/LTD, STDP, .../LTD, STDP, ...

Hebb D O, 1949: The Organization of Behavior

•

Cell assembly = mental object

•

Gestalt perception

•

• Perceptual completionPerceptual completion

•

• Perceptual rivalryPerceptual rivalry

•

• Milner P: Lateral inhibitionMilner P: Lateral inhibition

•

• FigureFigure--background segmentationbackground segmentation

•

After activity

≈

500 ms

•

• Persistent, sustainedPersistent, sustained

•

• Fatigue = Adaptation, synaptic Fatigue = Adaptation, synaptic

depression

depression

•

Generalized to association chains

•

• TimeTime--asymmetric synaptic Wasymmetric synaptic W

(11)

STOCKHOLMBRAININSTITUTE

A Mathematical instanciation of

Hebb’s cell assembly theory

•

Hopfield network 1982

•

Recurrently connected

•

Neocortex L2/3, hippocampal CA3,

olfactory cortex

•

Sparse connectivity and activity

•

Human cortical connectivity (10

-6

)

•

Activity (<1%)

•

Modular

• Kanter, I. (1988). "Potts-glass models of

neural networks." Physical Rev A 37(7): 2739-2742

•

Extensively studied

•

Simulations, e.g. memory properties

•

Theoretical analysis

• Efficient content-addressable memory!

(12)

STOCKHOLMBRAININSTITUTE

Single cell modelling

} Equivalent electrical circuit

} Hodgkin-Huxley formalism

} Na, K, KCa, Ca-channels } CaAP and CaNMDA pools } …

} Quantitative fit possible

} Of electrical signaling

} Connectionist/Mesoscopic } Integrate-and-fire

} Hodgkin-Huxley

} Single/multiple compartment

(13)

STOCKHOLMBRAININSTITUTE

Synaptic transmission and

plasticity

} Conduction delay

} Synaptic conductance, kinetics, PSP-shape

} Reversal potential

} Glutamate (AMPA&NMDA) } GABA_A

} Quantitative fit possible

} Synaptic plasticity

} Fast: Facilitation, Depression } Not yet LTP/LTD

2013-02-15

(14)

STOCKHOLMBRAININSTITUTE

What simulation tools?

}

(P)NEURON

}

NEST

}

PyNN

}

MUSIC

} ”MUlti-SImulation Coodination” } INCF-KTH 2013-02-15

(15)

STOCKHOLMBRAININSTITUTE

Conceptual model of Neocortex

} Hypercolumns are grouped into cortical areas of

various sizes

} Human neocortex has about 110 cortical areas (Kaas,

1987)

} Human V1 has ~40000 hypercolumns

} Memory attractors embedded in W

} On top of cortical mircocircuit

2013-02-15

Cortical areas

Hypercolumns

(16)

STOCKHOLMBRAININSTITUTE

≈

4x4 mm

•

330000

neurons

•

161 million

synapses

100 hypercolumns

Spontaneous activity

2013-02-15

(17)

STOCKHOLMBRAININSTITUTE

8 rack BG/L simulation

October 2006

•

22x22 mm cortical patch

• 22 million cells, 11 billion synapses

•

SPLIT simulator by KTH

• Hammarlund and Ekeberg, 1998

•

8K nodes, co-processor mode

• used 360 MB memory/node

•

Setup time = 6927 s

•

Simulation time = 1 s in 5942 s

•

77 % estimated speedup

• Linear speedup to 4K nodes

2013-02-15

Djurfeldt M, Lundqvist M, Johansson C, Rehn M, Ekeberg Ö, and Lansner A (2008): Brain-scale simulation of the neocortex on the Blue Gene/L supercomputer. IBM J R&D 52:31-41

(18)

STOCKHOLMBRAININSTITUTE A B 5 Hz C 50 mV 1 s D

Lundqvist M, Rehn M, Djurfeldt M and Lansner A (2006). Attractor dynamics in a modular network model of the neocortex. Network: Computation in Neural Systems: 17, 253-276

Spontaneous

“resting activity”

Tsodyks,

Grinvald et al.

1999

Stimulus resets!

}

2000+ neurons

}

250000+ synapses

}

5 s = 600 s on PC

2013-02-15

(19)

STOCKHOLMBRAININSTITUTE

Attractor rivalry and second

best match

0 1 2 3 4 5 6 7

seconds

0 1 2 3 4 5 6 7

seconds

(20)

STOCKHOLMBRAININSTITUTE

Random long-range W?

Cortical pairwise connection statistics obeyed in both cases

2013-02-15

Trained W

3.5 4 4.5 5 5.5 6 6.5 7 7.5

Permuted W

(21)

STOCKHOLMBRAININSTITUTE

Coupled oscillations

} Rasterplot from pyramidal cells of

the network

} Spontaneous switching between

memory states + non-coding ground-state attractor

} Alpha-Beta activity corresponds to

the periods of ground state

} Gamma nested on theta peaks

during active recall

} Palva JM, Palva S, Kaila K. (2005).

Phase synchrony among neuronal oscillations in the human cortex. J Neurosci 25: 3962-3972

} + Long-range synchrony/ coherence

} E.g. inter-hemispheric

} Conduction speed distribution data

} + Fine structure in spike patterns

2013-02-15

(22)

STOCKHOLMBRAININSTITUTE

Spontaneous attractor

”hopping”

}

”

Theta-Coupled Periodic Replay in Working Memory

”

} Fuentemilla et al. Curr Biol 2010, 275 channel MEG

Celsium-Linné symposium Uppsala 2013-02-15

Synthetic LFP F req ue n c y ( H z) 10 20 30 40 50 60 2 3 4 5 6 7 8 Time (seconds)

(23)

STOCKHOLMBRAININSTITUTE

Is ”attentional blink” a by-

product of cortical attractors

Silverstein, D. and A. Lansner Front Comput Neurosci 2011

}

RSVP of e.g. Letters

} Two target letters. T1 & T2 } 100 ms spacing

} T2 missed if too close

}

Attractors stored for

each item

} T1,T2 depolarized } +1 mV } Distractors hyperpolarized } - 1 mV

}

After-activity 300-500

ms, masking T2?

(24)

STOCKHOLMBRAININSTITUTE

Attentional blink results

} Powerful attentional modulation of

target patterns by ± 1 mV

} Lacking ”lag-1 sparing”

} Benzodiazepine modulates GABA

(amplitude & time constant)

} Boucard et al. 2000, Psychopharmacology 152: 249-255

Simulation Experiment

(25)

STOCKHOLMBRAININSTITUTE

}

Can be implemented with biophysically detailed neurons

and synapses

} Recurrent excitation, cellular adaptation, synaptic depression

critical

}

Basic perceptual and memory functions

} Psychophysical reaction/processing times, 100 ms } Perceptual completion and rivalry

} Oscillatory dynamics: theta, alpha, beta, and gamma (nested) } Stimulus sensitivity, highest in ground state

} Cognitive

functions/phenomena

} Working memory, Attentional blink, Stimulus detection } +On-line learning of sequences, reward learning

}

Structured, ”trained” W - How many memories?

}

+ sparesly conneced cortical area size network

} Theoretically ≈106 random attractors

} Time-consuming to verify by simulations ...

}

+ on-line LTP/LTD type synaptic plasticity

} Spiking Hebbian-Bayesian learning rule (BCPNN) now in NEST

Hebb’s theory - summary

(26)

STOCKHOLMBRAININSTITUTE

From brain models to

brain-like computing

}

Abstract, modular, spiking

associative memory (MPI code)

} Minicolumns as units, hypercolumns } Replicates well dynamics of detailed

model

} Extreme scaling workshop, Jülich Feb 2011

} Full Jugene, IBM BG/P, 294912 cores } 30 million spiking units, 300 billion

connections

} Close to real-time learning and recall

}

”L2/3 capability”

Æ

”L4

capability”

} Self-organization, competitive learning – recursively!

}

Machine learning evaluation

} MNIST, 95% on test set

2013-02-15

(27)

STOCKHOLMBRAININSTITUTE

Current challenges & trends

}

Computing power

} New Peta-Exascale supercomputers } Special purpose hardware

} SpiNNaker – ARM core based, 106 _{cores 2013}

} FACETS-BrainScales, analog VLSI } IBM neurochip

}

Software tools

} 3rd generation brain simulators

} INCF: MUSIC, NineML, Connection Set Algebra (CSA) } Synthetic M/EEG, Bold, ...

} Interaction, Visualization tools

}

Experimental data

} Connectome critical

} New methods – e.g. Brainbow } But can we measure W?

2013-02-15

(28)

STOCKHOLMBRAININSTITUTE

Conclusions

}

Brain simulation fits very well on supercomputers!

} Will help understand how the brain works, we can only

speculate how soon we will see results } Major impact on health, education, etc

}

Spiking attractor network models can replicate the

brain’s memory functions and network dynamics

}

We will in 10 years have hardware capable of

implementing ”artificial brains”

} Real-time or faster

}

This will be a technology for truly intelligent IT

} Brains for autonomous systems, robots and virtual agents } Running on compact, low-power, stochastic, adaptive HW } A new branch of IT industry, an opportunity for Europe } Human Brain Project funding will make a difference!

(29)

STOCKHOLMBRAININSTITUTE

Acknowledgements

2013-02-15

JUGENE FZJ

294912 cores

•

Model development and analysis

•

Mikael Lundqvist, PhD student

•

David Silverstein, PhD student

•

Pradeep Krishamurthy, Phd student

•

Data analysis and modelling

•

Pawel Hermann, postdoc

•

Henrik Lindén, postdoc

•

Parallel simulation tools

•

Mikael Djurfeldt, postdoc

(30)

STOCKHOLMBRAININSTITUTE

Thanks for your attention!

2013-02-15

(31)

STOCKHOLMBRAININSTITUTE

From conceptual and abstract

models to biophysics

•

Computational units?

•

Neuron

•

Minicolumn

•

Local sub-network

•

Species differences?

•

Larger functional

modules?

•

Hyper/Macrocolumns

•

How general?

Hubel and Wiesel

icecube V1 model

Peters and Sethares 1997 Yoshimura and Callaway 2005 2013-02-15

(32)

STOCKHOLMBRAININSTITUTE

A layer 2/3 cortex model

Microcircuit layout

} Minicolumns/local sub-networks with

} 30 pyramidal cells, connected 25%

} 2 dendritic targeting, vertically projecting inhibitory interneurons

} RSNP, e.g. Double bouquet

} Hypercolumns (soft WTA modules) with

} Pool of Basket cells

} Martinotti cells, with facilitating synapses from pyramidal cells } Large models: 100 minicolumns, 200 basket per hypercolumn

} Currently rudimentary layers 4 and 5

2013-02-15

1_{17% 2.5 mV} 230% 0.30 mV 70% -1.5 mV 70% 1.2 mV 70% 2.5 mV 25% 2.4 mV

(33)

STOCKHOLMBRAININSTITUTE

Bistability, ground state and

active coding states

}

Plot formats

}

Raw

}

Grouped

}

Bistable

}

Ground state

}

Many active

(coding) states

}

Oscillations

Celsium-Linné symposium Uppsala 2013-02-15

0 1 2 3 4 5 6 7

(34)

STOCKHOLMBRAININSTITUTE

Spiking activity

in ground and active state

•

Ground state –

diffuse

•

Foreground/Active state –

focused

•

V

_m

is oscillatory

• Foreground neurons lead

• Race condition, Fries et al. TINS 2007

•

≈

same number of spikes in ground

and active states

Backgound spikes

Foregound spikes

(35)

STOCKHOLMBRAININSTITUTE

2013-02-15

Targets =

MG

K

IRC

V

P

Y

O

NU

J

T

V

X

G

H

C K DL

S

Z

F

N

B

T

T1

T2

T1

T2

Perceptual/Cognitive

phenomena?