Quantifying Multisensory Integration

One of the challenges in studying multisensory integration is how to isolate and quantity contributions from multisensory interactions. There have been numerous models developed to quantify multisensory integration based on neurophysiological and behavioural data (reviewed in Stevenson et al., 2014a). Most of these models assess multisensory integration based on two simple criteria: the maximum criterion or the additive criterion (Fig. 2.8A; Peelle and Sommers, 2015).

The maximum criterion model compares the response to a multisensory stimulus with that of the most effective unisensory condition (Meredith and Stein, 1983, Meredith and Stein, 1986b). The rationale is that any response measure departing from that of the most effective unisensory condition should be attributed to the multisensory nature of the stimulus, that is, to interactions between the inputs from the two modalities. When measuring behaviour, this model is only suitable when performance is either below threshold or near ceiling in at least one of the unisensory conditions (Stevenson et al., 2014a). In neurophysiology, this model can be applied when the signal being recorded is from a site that is only particularly responsive to unisensory stimulation from one modality, but displays enhanced responsiveness during multisensory stimulation. The maximum criterion model defines multisensory integration (MSI) as follows:

26

MSI AV max



A,V



, (2.1) where variables A, V, and AV represent the behavioural/neurophysiological measures (e.g., accuracy, spike rate, amplitude) for each stimulus condition. Positive MSI values indicate enhancement, negative values indicate reduction and zero values indicate no integration (Fig. 2.8B; Meredith and Stein, 1983, Peelle and Sommers, 2015). Of course, when examining something like reaction time (RT), this model can be modified to compare AV with min(A,V), i.e., the fastest unisensory condition.

The additive criterion model on the other hand, compares the response to a multisensory stimulus with that of the algebraic sum of the unisensory conditions (Stein and Meredith, 1993, Barth et al., 1995, Berman, 1961). The rationale here is that the response to a multisensory stimulus should be equal to the sum of the responses generated separately by the two unisensory stimuli, if the two unisensory signals were processed independently. Thus, any departure from the summed response should be attributed to multisensory interactions (Besle et al., 2004b). For behavioural measures, this model is most suitable when the unisensory response magnitudes from both modalities are not near threshold or ceiling (Stevenson et al., 2014a). In neurophysiology, this approach is most suited to recording sites that are responsive to both unisensory stimuli, particularly when recording from populations of neurons. Based on the additive criterion, multisensory integration is defined as follows:

MSI AV 



AV



. (2.2) Here, positive MSI values indicate ‘superadditivity’, negative values indicate ‘subadditivity’ and zero values indicate no integration (Fig. 2.8C; Stein and Meredith, 1993, Peelle and Sommers, 2015). The validity of the additive model is well established, particularly in the field of electrophysiology (Besle et al., 2004b). This is because when measuring electric signals elicited by the brain, their magnitude is governed by the law of superposition of electric fields. The principle of superposition states that the net response of a linear system (and tissue is a linear conductor at macroscopic scales) at a given position and time caused by two or more stimuli is equal to the sum of the responses which would have been produced by each stimulus individually.

However, behavioural measurements are sometimes represented as probabilities (e.g., detection accuracy, RT), meaning it is necessary to include an expression of the joint unisensory probability in the model. For instance, if detection accuracy was being

27

measured, this would account for the probability that a multisensory stimulus was detected in both modalities (Stevenson et al., 2014a). Or if RT was being measured, this would account for the probability that the stimuli were detected at the same time in both modalities. Suppose each variable in Eq. 2.2 represented the probability of detecting a stimulus in each condition, the formula could be extended to account for joint unisensory probability as follows:

MSIAV 



AV AV



. (2.3) This is equivalent to assuming that an error in the AV condition only occurs if there is an incorrect response in both of the unisensory conditions, i.e., 1−AV = (1−A) (1−V) (Blamey et al., 1989). The same model can also be applied to RT measurements by replacing each variable in Eq. 2.3 with the RT cumulative distribution function (CDF) for each condition. This is equivalent to sampling simultaneously from the unisensory RT distributions, taking the faster of the two unisensory RTs and then computing the CDF, i.e., the ‘race model’ (Raab, 1962). Violation of the race model (i.e., positive MSI values) indicates multisensory interactions or ‘co-activation’ (Miller, 1982, Molholm et al., 2002).

To quantify MSI in terms of percentage gain, Meredith and Stein (1983) defined an ‘interactive index’ that scaled MSI relative to the magnitude of their model:

Gain M SI100,

P (2.4)

where P is the multisensory response predicted by the unisensory response values, i.e., max(A,V) or [A+V] or [A+V−A×V]. In other words, this represents the percentage gain in processing attributable to multisensory interactions relative to independent unisensory processing.

28 Figure 2.8: Quantification of multisensory integration.

A, Integration criteria. B, Multisensory integration based on a maximum criterion model. C, Multisensory integration based on an additive criterion model (adapted from Peelle and Sommers, 2015).

In document Towards a greater understanding of the neurophysiology of natural audiovisual speech processing : a system identification approach to EEG (Page 45-48)