Conclusion

In this project we demonstrated how Bayesian inference can be implemented with spiking neurons on an analog neuromorphic substrate. The implementation was able to cope with the challenges inherent to analog hardware such as the fixed-pattern noise, imprecise parameters, limited control of the parameters and limited bandwidth between the chip and the host computer; while it still benefited from the 104speed-up of the hardware. The entire setup was realized on the hard- ware including the stochasticity-providing random network, hence we realized a self-contained sampling machine. External communication was only used for representing input data and acquiring response spikes from parts of the network, such as the label neurons. Note, that such a system is also a plausible model for sampling-based inference in the cortex [Jordan et al., 2019, Dold et al., 2019] when considering the deterministic behavior of neurons in vivo [Mainen and Sejnowski, 1995, Reinagel and Reid, 2002, Toups et al., 2012, Masquelier, 2013].

We demonstrated sampling from arbitrary Boltzmann distributions (section 3.4.1); and we showed that the framework can be applied to form hierarchical structures to learn from standard datasets and to solve discriminative and generative tasks (section 3.4.2). As previously shown [Probst et al., 2015], the framework can be extended to sample from arbitrary distributions over binary variables. We stress that in the case of hierarchical networks, the model solves the classification and

3. Bayesian inference on BSS-1

the pattern completion task simultaneously by sampling from the conditional distribution with its dynamics. Both, when learning a target distribution and in inference tasks, the model could make use of the acceleration of the hardware reaching a net speedup of 100 to 210 compared to biological real-time.

We view our project as a contribution to the rapidly expanding (although not yet competitive) field of biologically inspired physical model systems. We demon- strated the feasibility of the approach to tackle machine learning problems, and (implicitly) to study biological phenomena. Our results implicitly show the ro- bustness of the application and its ability to harness the advantages of the un- derlying substrate, in this case the speed-up. The introduced architecture scales by design to neuromorphic hardware with more available neuronal and synaptic resources. The underlying Boltzmann Machine can be mapped to other problems where a Bayesian approach is beneficial, such as prediction of temporal sequences [Sutskever and Hinton, 2007], solving constraint satisfaction problems [Jonke et al., 2016, Fonseca Guerra and Furber, 2017], movement planning [Taylor and Hinton, 2009, Alemi et al., 2015], quantum many-body problems [Carleo and Troyer, 2017, Czischek et al., 2018, Carleo et al., 2019b, Melko et al., 2019] and simulation of solid state systems [Edwards and Anderson, 1975]. Since the publication of this project, sampling with LIF neurons has been successfully implemented on a prototype chip of the BSS-2 system [Billaudelle et al., 2019b, Stradmann, 2019].

4 Pilot study on the BrainScaleS-2

prototype chip — demonstrating

advantages of neuromorphic

computation

The content of this chapter was published in Wunderlich et al. [2019] in close collaboration mainly, but not exclusively, with Timo Wunderlich. Here, we follow the publication but we give a more detailed description of the project for the sake of clarity and completeness.

Neuromorphic devices represent an attempt to create novel non-Turing comput- ers relying strongly on inspiration from neural networks in biology [Mead, 1990]. This approach aims at capturing the advantages of the mammalian nervous system to create fast and robust hardware with low-power and/or energy consumption (section 2.3). This endeavor abstracts away details of the brain’s neural networks while trying to keep only the central aspects which provide the desired advantages. The holy grail of neuromorphic engineering, the correct level of abstraction and the most important aspects of biological neural networks, remains elusive; but several approaches appeared in search of it (section 2.3.2). A common central approach is that models of neurons, which are considered the basic computational units of the brain, are emulated using specialized digital and/or analog systems (section 2.3.2). Although the idea of neuromorphic engineering reaches back to the 1980s, publications quantifying the performance of neuromorphic hardware on practical tasks are scarce [Merolla et al., 2014, Davies, 2019, Rhodes et al., 2019].

In this project, we demonstrate and quantify the advantages of neuromorphic computation in terms of speed, energy consumption and robustness in a pilot- study. To this end, we use the example of learning a simplified version of the Pong video game with reward-modulated spike-time-dependent plasticity (R- STDP)-based reinforcement learning on the BrainScaleS-2 system High Input- Count Analog Neural Network Digital Learning System v2 (HICANN-DLSv2) neuromorphic prototype chip.

BSS-2 is a neuromorphic hardware system consisting of CMOS-based ASIC [Friedmann et al., 2017, Aamir et al., 2018] implementing physical models of neurons and synapses in mixed-signal circuits. Several features make BSS-2 unique among neuromorphic approaches (section 2.3): 1) the emulation runs

4. Demonstrating advantages

with an acceleration of 103 compared to biological real-time (compare to BSS-1

in section 3.1) due to the supra-threshold working point of the analog circuits, 2) an on-chip processor enables the flexible implementation of a broad range of plasticity rules and in our study the environment simulation, finally 3) built- in correlation sensors in the synapses and digital spike counters serve as easily accessible observables for plasticity calculations.

In reinforcement learning, a behaving agent interacts with its environment and aims at optimizing its parameters in a way to maximize the obtained reward from the environment (Sutton and Barto [2018] and section 2.1.3). It is a relevant learning paradigm for neuromorphic systems for applications where autonomous systems are required, e.g. in robotics. In recent years, reinforcement learning systems have reached remarkable, often superhuman results in playing board and video games [Mnih et al., 2013, Silver et al., 2017, Vinyals et al., 2019]. Unfortu- nately, mapping these state-of-the art solutions to neuromorphic hardware is a far-from-obvious task, because they often use mechanisms that are not directly found in neuromorphic systems, e.g. non-local learning rules or tree search. Hence, for more compatible models of reinforcement learning we turn to computational neuroscience. Since the discovery that the neuromodulator dopamine corresponds to the reward-prediction error in the brain [Schultz et al., 1997, Niv, 2009], new mechanistic models, the so-called three factor learning rules, have been developed for neural reinforcement learning [Frémaux et al., 2013, Frémaux and Gerstner, 2015]. In these models the plasticity rule depends on a Hebbian term contain- ing pre- and post-synaptic contributions, and on a third global neuromodulator, which behaves similarly to dopamine in the brain. These learning rules are good candidates for implementation on neuromorphic hardware because they roughly respect the constraints of the substrate. The R-STDP plasticity rule is a simple member of this family of learning rules.

Our experiments take place fully autonomously on the BSS-2 chip using the on-chip plasticity processing unit both for plasticity calculations and for environ- ment simulation. Measurements of time of emulation and energy consumption are compared to a reference implementation using the Nest simulator [Peyser et al., 2017] on an off-the-shelf CPU. The neuromorphic chip achieved an order of mag- nitude faster emulation and three orders of magnitude lower energy consumption than the simulator on CPU. The learning converged on the neuromorphic chip within seconds. Further, we show that the learning is robust against imprecision of calibration of the neuron parameters and compensates against fixed-pattern noise on the analog parts of the chip. Finally, meta-parameters optimized for one BSS-2 chip can be transferred without significant performance loss to another BSS-2 chip.

4.1 Materials and methods

4.1 Materials and methods