Bayesian models, uncertainty and sensorimotor control

One fundamental assumption of the central hypothesis of this thesis is that the brain uses a hierarchical generative model to predict the sensory consequences of movements and updates this based on precision-weighted prediction errors. There is a plethora of evidence from the motor control literature to suggest that forward models are

implemented in the sensorimotor system (Bastian, 2006; Blakemore and Sirigu, 2003;

Miall et al., 1993; Paulin, 1993; Wolpert and Miall, 1996). For example, it has been shown that during a reaching movement, saccades move to a position in advance of the current hand position, which suggests that we predict the future sensory consequences of our motor commands (Ariff et al., 2002). Moreover, there is evidence demonstrating that the brain integrates estimates of uncertainty into these models in a Bayesian manner in the sensorimotor system. In one example participants were trained to reach towards a target shifted to the left of the true target location with no visual feedback. Giving visual

feedback midway through the movement shifted the end point towards the real target location. Increasing the uncertainty (visual blur) in this feedback reduced the influence of the sensory data on the final reaching position. Increasing the uncertainty in the sensory evidence, increased participants’ reliance on their prior estimation of where the target was in line with Bayesian statistics (Körding and Wolpert, 2004). There are multiple different frameworks of how Bayesian models control movement and how they may be

31

implemented in the brain. In particular, active inference offers an alternative approach to sensorimotor control then the current leading framework, Optimal Control Theory (OCT).

There is a lack of empirical evidence to support the hypotheses produced by the active inference framework, however it offers an alternative outlook from which novel predictions and experimental paradigms can be tested.

1.3.1. Active inference vs Optimal Control

OCT is a popular theory describing how we control movements. This theory tries to address a key problem in motor control: how does the brain select the most optimal action out of several possible movement trajectories? (Franklin and Wolpert, 2011; Wolpert and Ghahramani, 2000). Within this Bayesian framework, a forward model estimates the current state of the body by combining predicted and current sensory input. This state estimation is then used to update optimal control functions, which rank possible actions based on cost functions aiming to minimise musculature noise whilst ensuring optimal achievement on the task at hand. Motor commands are generated via an inverse model, which reduces a future cost. An efference copy of these commands is sent to the forward model to predict changes in hidden states using sensory predictions. A number of studies have shown how that this model can accurately explain behaviour (Harris and Wolpert, 1998; Haruno and Wolpert, 2005; Todorov, 2004; Todorov and Jordan, 2002). However, this theory lacks a specific understanding of how each component could map onto neurophysiological connections in the brain; active inference tries to do this.

There are also some fundamental differences between OCT and active inference. Firstly, it has been argued that it is unlikely for movements to be specified with single learned cost functions (Friston, 2011); active inference replaces cost functions with prior beliefs that emerge naturally from perceptual inference. Secondly, OCT states that descending signals to the spinal cord transmit motor commands, whereas in active inference these

descending signals are proprioceptive predictions about the proprioceptive consequences of the movement. This is an important difference as the type of connection described has implications for the neurobiology. In predictive coding frameworks, predictions are backward connections, whereas commands are driving, forward connections. Adams et al (2013) argue that the anatomical and physiological characteristics of the descending motor input to the spinal cord suggests that they are of the backward-type and more likely represent predictions rather than commands themselves. Finally, OCT places an inverse model within motor cortex, which generates motor commands, whereas in active

inference predictions are generated by a hierarchical generative model and these are converted into motor commands within the spinal cord. Indeed, the EPH (described

32

earlier) shares many of these characteristics with active inference. The main advantage of active inference and the EPH is their solution to the redundancy problem of action

selection: the correct motor command is automatically produced from the deviation of the descending control signal from the threshold point in the spinal cord, so there is no need for an inverse model in the cortex. Importantly, the active inference framework, unlike the EPH, can provide a unifying hypothesis of how predictive coding is implemented in the brain across a number of processes from perception and cognition to motor control.

However, there is a lack of empirical evidence that movements are controlled according to the active inference framework. In particular, there is a lack of evidence demonstrating that neurophysiological correlates of the components important for predictive coding exist in the sensorimotor cortex.

1.3.2. Bayesian Models and sensorimotor learning

Sensorimotor adaptation paradigms offer a useful method to measure predictive coding within the sensorimotor system. Perturbations to movement trajectories generate prediction errors between the predicted and actual sensory consequences of the action and the neurophysiological correlate of these parameters can be measured. Bayesian models also provide a particularly useful account to explain sensorimotor learning in the presence of uncertainty. There are a number of different models which have been used to explain participant’s behaviour on these types of tasks. Traditional

reinforcement-learning models explain how an animal receives new information, which is then used to update its beliefs about the environment in proportion to prediction errors: the prediction error must be multiplied by a learning rate to determine the degree to which a given belief is updated (Rescorla and Wagner, 1972; Sutton and Barto, 1998). Models, such as the Rescorla-Wagner (RW) learning model, are simple, computationally efficient and used widely in cognitive neuroscience; however, do not incorporate estimates of uncertainty which are integral for optimal perception and action.

Bayesian accounts of learning formalise how beliefs are updated based on new data and suggest that learning rates are dependent on current levels of uncertainty of prior beliefs relative to sensory input. Indeed, using a visuomotor adaptation task, (Wei and Körding, 2010) demonstrated that increasing uncertainty in the visual feedback of a cursor end-point position reduced learning rates, such that participants did not adapt to a visuomotor rotation as quickly. Here the Kalman filter was used for Bayesian estimation of hand position and best explained the experimental data suggesting that we do adapt to

visuomotor perturbations in a Bayesian manner and respond to visual noise as predicted.

Importantly, uncertainty can arise from a number of sources, not solely surrounding the

33

sensory input. In particular, (Yu and Dayan, 2005) suggest that the volatility of the environment (“unexpected uncertainty”) will dictate the learning rate: in a fast-changing, volatile environment where more recent experience is important, the learning rate will be large, such that prediction errors have a large influence on the update of prior beliefs;

however, if historical information is more important, such as in a stable environment, the learning rate will be smaller and require a larger prediction error to update prior beliefs.

There have been a number of studies using Bayesian learning models to understand how humans use Bayesian inference to track reward probabilities in a changing environment and actively adapt their learning rate to this (Behrens et al., 2008, 2007). However, the approach used in these studies is computationally expensive and the learning process is assumed to be identical across participants.

An alternative approach has recently been suggested which applies Bayesian updating in a computationally efficient manner using one-step update equations designed to minimise free energy in a biologically plausible way. The Hierarchical Gaussian Filter (HGF), designed based on probability theory, explains how a participant learns about their environment from the sensory information available given their own generative model.

The internal generative model represents how the participant believes sensory information is generated in the world; an inversion of this model produces a posterior probability distribution, which represents this belief and predicts what sensory

information is expected. The HGF consists of two models. The perceptual model describes how these beliefs update over time in order to explain how a participant learns about an unknown, continuous variable that modulates over time. The response model then describes how the participant should behave given those beliefs by mapping the beliefs onto actions. Here participant-specific parameters dictate individual learning rates that modulate over time based on the participant’s trial-wise behaviour unlike traditional reinforcement learning models, which have a fixed learning rate across time. Importantly, the perceptual model of the HGF is hierarchical and each level is coupled to higher levels by the variance in the modulation of the underlying hidden state; therefore, the volatility of the hidden state at the second level is dictated by the variance at the first level. In this way different forms of uncertainty can be captured at different levels of the hierarchy. For example: the first level captures irreducible uncertainty unaffected by learning in which an unexpected stimulus requiring an unexpected response generates a sensory prediction error; the second level represents estimation uncertainty in the stimulus transition probabilities describing how likely it is that the stimulus presented will be different from expected and generates a contingency prediction error; and the third level describes the volatility uncertainty that arises from the stability of the environment and generates a

34

volatility prediction error. The HGF produces individual time series of how beliefs evolve over time at each level of the hierarchy, therefore offers the opportunity to separate prediction, prediction error and precision parameters at different hierarchical levels. This is important as the active inference framework makes specific predictions about the presence of these parameters of predictive coding within the sensorimotor system and their role in motor control.

In this PhD, I used the HGF to explain participant’s behaviour on a visuomotor adaptation paradigm and recorded EEG to determine the presence of these

predictive coding parameters in the sensorimotor system specifically using a model underpinned by the active inference framework. The visuomotor adaptation paradigm used offers a useful tool to measure and manipulate parameters of predictive coding and has previously been used to assess neurophysiological correlates of predictive coding in the sensorimotor cortex (Tan et al., 2014a, 2016).

The active inference framework aims to generalise ideas from predictive coding to the sensorimotor system, therefore this study will offer an important insight into the plausibility of this hypothesis.