Characterizing multisensory integration using fMRI 28

The seminal studies that propelled the development of research on the integration of the senses were neurophysiological investigations performed on cats and non-human primates (Rizzolatti et al., 1981a, 1981b; Graziano and Botvinick, 2002; Stein and Stanford, 2008). These studies benefited from the opportunity to record the electrical activity of individual neurons exposed to stimuli from multiple sensory modalities, and led to the development of the fundamental principles of multisensory integration that are still central to modern psychological and neuroscientific research in humans (Driver and Noesselt, 2008; Stein and Stanford, 2008; Noppeney, 2012). The advent of non- invasive neuroimaging techniques like fMRI spurred an ongoing era of studies attempting to characterize multisensory integration in behaving humans (Calvert et al., 2000, 2001). Indeed, one of the aims of the present thesis was to provide evidence for the existence of multisensory integrative processes concerning the sensory signals from the hand. Because of the inherent limitations of human neuroimaging discussed previously, the extension of the basic neurophysiological principles to fMRI is far from straightforward (Calvert and Thesen, 2004; Beauchamp, 2005; Laurienti et al., 2005; Goebel and van Atteveldt, 2009; Stevenson et al., 2009; Noppeney, 2012).

Among the criteria that have been put forward in neuroimaging investigations of multisensory integration, the most direct involves the comparison of the BOLD response to a multisensory stimulus to the responses to unisensory stimuli presented in isolation. The corresponding statistical criteria applied to the multisensory BOLD responses vary in their stringency and inferential power. A graphic summary of some of these criteria is presented in Figure 2A.

V T mean max sum VT 0 1 2 3 4 0 1 2 3 4

V T mean max sum VT

BOLD  response  (a.u.) BOLD  response  (a.u.)

0 2 4 6

V T mean max sum VT

BOLD  response  (a.u.)

Criterion  of  the  mean Criterion  of  the  max Criterion  of  superadditivity

Visual  neurons Tactile  neurons Multisensory  neurons 0

2 4 6

V T max sum VT

BOLD  response  (a.u.)

0 2 4 6

V T sum VT

BOLD  response  (a.u.)

A

B C

fMRI  volume  element

Figure 2. (A) Overview of some of the statistical criteria employed to characterize multisensory BOLD

responses. The bar charts display the responses of a sample voxel to visual (V) stimuli, tactile (T) stimuli, and combined visuo-tactile (VT) stimuli, together with their mean, the maximum of the unisensory responses, and their sum. (B) A sample voxel containing separate non-interacting visual and tactile neurons. (C) A non-linear superadditive response to VT stimuli strongly suggests that the voxel contains multisensory neurons that integrate the unisensory signals.

The criterion of the mean, for example, classifies a BOLD response as multisensory if greater than the average of the responses to the unisensory stimuli, whereas the criterion of the maximum takes into account the most responsive unisensory condition. Both constitute examples of a subadditive BOLD signal, a response profile whose analogous in neurophysiology belongs to the range of the observed responses to multisensory stimuli (Laurienti et al., 2005; Avillac et al., 2007; Stein and Stanford, 2008). However, given the spatial resolution of fMRI data, where an individual voxel contains hundreds of thousands or more neurons, these response profiles cannot be disentangled from the case of a voxel that contains populations of non-interacting modality-specific neurons (Figure 2B). The most conservative of these basic criteria, the criterion of superadditivity, relies instead on a significant difference between the BOLD response to the multisensory condition and the sum of the responses to the unisensory conditions (Figure 2A). If the criterion is met, a strong argument can be made in favor of the existence of multisensory neuronal populations within the voxel (Figure 2B), whose non-linear response profile cannot be accounted for by the existence of non-interacting modality-specific neurons. Instead, a non-linear interaction between the sensory inputs is necessary to elicit a superadditive response, be it at the level of neuronal firing rates or synaptic activity (Logothetis et al., 2001; Logothetis, 2008). In light of this, the superadditivity criterion arguably ranks first in terms of

inferential power, and has successfully been adopted in several neuroimaging studies of multisensory integration (Calvert et al., 2001; Ehrsson et al., 2004; Stevenson et al., 2007; Werner and Noppeney, 2010, 2011; Petkova et al., 2011a; Tubaldi et al., 2011; Tyll et al., 2013). Nevertheless, a number of caveats must be taken into account when interpreting the results of the application of the superadditivity criterion to BOLD fMRI. Some of these factors trace back to the neurophysiological principles of multisensory integration, whereas others are intrinsic to the use of fMRI. First, superadditivity is not the only operational profile of multisensory neurons, and computations that are subadditive or linear in nature have been observed consistently and deemed relevant for neuronal processing and behavior (Laurienti et al., 2005; Stanford and Stein, 2007). Second, the occurrence of superadditive responses may vary based on the efficacy of the unisensory stimuli, a phenomenon that has been termed the

principle of inverse effectiveness (Holmes and Spence, 2005a; Stein and Stanford,

2008; Noppeney, 2012). In other words, the benefit of a non-linear combination of the sensory inputs may decrease with the increased effectiveness (neuronal and behavioral) of the unisensory signals. Third, the superadditivity criterion is likely to increase the incidence of false negatives in fMRI analyses because of the complex physiological dynamics underlying the BOLD signal, such as vascular ceiling effects (Buxton et al., 2004). All of the above factors contribute to making the superadditive criterion perhaps too conservative, and certainly not exhaustive. Nevertheless, the detection of a non- linear BOLD response represents very strong evidence in favor of the existence of multisensory processes, and is particularly powerful, for instance, when applied in the context of well controlled factorial experimental designs (Ehrsson et al., 2004; Tsakiris et al., 2007; Petkova et al., 2011a). In summary, the selection of the statistical criteria to evaluate BOLD responses to multisensory stimuli can have a large impact on the interpretation of the findings, and must always be kept in mind when presenting and discussing the findings (Beauchamp, 2005; Holmes and Spence, 2005a; Laurienti et al., 2005; Goebel and van Atteveldt, 2009; Stevenson et al., 2009; Noppeney, 2012).

Numerous other analysis techniques have been developed to address the issue of multisensory integration in fMRI. Worth mentioning here are the use of BOLD- adaptation paradigms (see next section), which tap into the sub-voxel selectivity profile of neuronal populations, and the manipulation of the congruence (spatial, temporal, semantic, etc.) of the stimuli (Calvert et al., 2001; Macaluso and Driver, 2005; Noesselt

comparisons to multisensory conditions, overcoming some of the limitations typical of the criteria reviewed above. Congruence manipulations have been key to numerous neuroimaging investigations of the multisensory mechanisms underpinning the sense of body ownership (Ehrsson et al., 2004, 2005, 2007; Tsakiris et al., 2007, 2010b; Ionta et al., 2011b; Petkova et al., 2011a; Longo et al., 2012b; Guterstam et al., 2013). However, the effects of the manipulation of multisensory congruence can be dependent upon the cognitive and attentive state of the participant, the type and predictability of the stimuli used, and on the task (if any) the participant is engaged in during the experiment (Corbetta and Shulman, 2002; Zimmer and Macaluso, 2007; Talsma et al., 2010; Noppeney, 2012). In conclusion, the characterization of multisensory integrative processes using fMRI demands careful considerations, particularly when selecting the stimuli, defining the experimental design, and selecting the statistical criteria to evaluate the BOLD responses to the multisensory conditions.

4.5 BOLD-ADAPTATION: MAKING STRONGER INFERENCES ON