Effect of stimulus on inter-subject correlation

In document Neural Pattern Similarity and Visual Perception (Page 174-177)

We found two interesting trends relating the stimulus type to the inter-subject correlations: (1) a de- crease in visual region correlations under AV stimulation compared to V-only, (2) higher correlations in primary auditory cortex under V-only stimulation for the “People Doing Stuff” video.

5.4.1 Less attention to vision likely when audio is introduced

Fig. 5.6 summarizes the trend of generally lower visual correlation under AV stimulation compared to V-only. In early visual cortex, superior occipital cortex, fusiform gyrus, and collateral sulcus, inter-subject correlations were significantly lower. Averaged across these regions, a subject pair correlated 29% less when stimuli were audiovisual. At first, we found this result to be counter- intuitive. Subjects informally reported feeling more alert and interested in the videos when they included sound. Thus, it might be expected that subjects’ brain activity would be more locked to the stimuli globally, and that inter-subject correlations would not only appear in auditory regions (compared to the V-only case), but also be enhanced in visual regions. Instead, the opposite was found.

Because subjects always viewed the silent videos first, in a nearly hour-long stretch in the MR scanner, it could be argued that they were simply more tired during the AV videos, and that is why synchronization then was relatively attenuated. However, several observations suggest otherwise. First, as already mentioned above, audiovisual stimuli are simply more engaging to subjects, and thus increase attention, possibly enough to compensate for AV videos being presented after V-only. Second, early visual cortex, fusiform gyrus, and collateral sulcus were significantly less correlated between subjects in the AV case even just in the first two minutes (all p <8×10−4 by two-sided

t-test) and just the first five minutes (all p < 4×10−3) of stimulus presentation, when subjects

were most alert in both cases. Third, if we compare inter-subject correlations during the second 7 minutes of the Back to the Futurevideo in the V-only case to inter-subject correlations during the first half in the AV case, we might expect the trend to reverse or disappear, since correlations right after stimulus onset might be higher than those 7 minutes in. However, we find that the direction of the trend remained the same, both in terms of number of subject pairs for which correlation was lower in AV case, and mean effect size, with AV correlations lower in early visual cortex, MT, fusiform gyrus and collateral sulcus; in this most strict test, however, statistical significance was only achieved in fusiform gyrus (p <2×10−3).

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7

Corr. under Visual Stim. Only

Corr. under Audio & Visual Stim.

Region EV (Conditions different p<0.00374) 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7

Corr. under Visual Stim. Only

Corr. under Audio & Visual Stim.

Region MT

(Conditions not significantly different)

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7

Corr. under Visual Stim. Only

Corr. under Audio & Visual Stim.

Region SOcc (Conditions different p<0.00026) 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0 0.1 0.2 0.3 0.4 0.5 0.6

Corr. under Visual Stim. Only

Corr. under Audio & Visual Stim.

Region Fusi

(Conditions different p<3.38e−05)

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0 0.1 0.2 0.3 0.4 0.5 0.6

Corr. under Visual Stim. Only

Corr. under Audio & Visual Stim.

Region CoS

(Conditions different p<1.36e−05)

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0 0.1 0.2 0.3 0.4 0.5 0.6

Corr. under Visual Stim. Only

Corr. under Audio & Visual Stim.

Region PCS

(Conditions not significantly different)

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0 0.1 0.2 0.3 0.4 0.5 0.6

Corr. under Visual Stim. Only

Corr. under Audio & Visual Stim.

Region A1

(Conditions different p<1.31e−15)

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0 0.1 0.2 0.3 0.4 0.5 0.6

Corr. under Visual Stim. Only

Corr. under Audio & Visual Stim.

Region STS

(Conditions different p<3.05e−09)

Unique Subject Pair

Figure 5.6: The ROI-wise inter-subject correlations are observed to be higher in visual cortex in the visual-only stimulation condition than in the auditory & visual stimulation condition. Correlations in A1 and STS are clearly higher under auditory & visual stimulation. All p-values are computed from a two-sided t-test, with one data point per subject pair.

although they felt more engaged, visual attention itself might have been blurred as subjects could occasionally gain relevant information (e.g. dialogue) from listening alone. Put another way, shutting down auditory processing heightened subjects’ sense of visual perception.

5.4.2 Synchronization in A1 under visual-only stimulation

Back to the Future People Doing Stuff Psychedelic

−0.3 −0.2 −0.1 0 0.1 0.2 0.3 inter − subject correlations in A1

Inter−subject correlations in primary auditory cortex under visual−only stimulation

Figure 5.7: Median inter-subject correlations (ROI-wise) in A1 under visual-only stimulation are above zero for the first two videos. Box plots indicate median, 25th, 75th percentiles, minimum and maximum values over the set consisting of both brain sides in each of 21 subject pairs. Back to the Futureand “People Doing Stuff” are both higher than the psychedelic video (p <2×10−4, one-sided

t-test) and “People Doing Stuff” is slightly higher thanBack to the Future (p < .15, one-sided t-test).

Fig. 5.7 shows the distribution of inter-subject correlations in A1 under visual-only stimulation, for each of the 3 videos separately. Subjects correlated duringBack to the Future and during “People Doing Stuff”, significantly above zero (p <7×10−3, one-sided t-test), but not during the psychedelic

video. Furthermore, the mean inter-subject correlations were slightly higher during “People Doing Stuff” than duringBack to the Future (.069 and .047 in left and right hemispheres, averaged across subject pairs, compared to 0.036 and 0.027), although the difference was not enough to establish significance given the variance of the data.

We looked for times in the “Stuff” video at which group activity (ROI-averaged in each hemi- sphere of each subject, then averaged across subjects) was most above baseline, and significantly so according to a one-sided t-test (p <10−3) on the set of unique time-courses (one per hemisphere

activation closely coincided with the two longest segments in the video (of 57): the first involving toilet paper unraveling by the wind of a box fan (lasting about 20 seconds), and the second involv- ing water filling a glass container with plastic bottle caps inside (lasting about 16 seconds). That is, as these clips were being shown, activity in A1 ramped up across subjects, peaking at 151 and 219 seconds into the video (total length: 240 seconds), thus contributing positively to inter-subject correlation.

Again, because subjects all viewed videos in the same order, there is the possibility that corre- lations in A1 were higher in the first two videos only because they came first, and subjects became bored later, thus decorrelating relative to each other. However, this possibility is weakened by the observation that the correlations during “Stuff”, which came second, were stronger than those during

Back to the Future, despite the latter being shown first, and lasting longer, both of which increase opportunity for synchronization.

Although it is surprising that subjects do correlate even in A1 without auditory stimulation, it is not surprising that among the three videos shown, correlations during “Stuff” were highest. Compared to the psychedelic video, “Stuff” had clear temporal structure; it was comprised of dozens of individual segments, each with a clear beginning and end. Furthermore, many of the segments featured visual stimuli, such as objects dropping and colliding, which, in the natural world, would be accompanied by very salient and identifiable sounds. Viewing these clips without sound elicits auditory imagery, which can modulate the input to A1 and contribute to an increased fMRI signal, even in the absence of spiking activity there [62]. Similarly, “Stuff” had more clear temporal structure and featured more aurally suggestive content thanBack to the Future,the constituent scenes of which lasted considerably longer than the segments in “Stuff”. However,Back to the Future did contain some scenes which would be expected to elicit auditory imagery (e.g., a car accelerating, people talking), just not as many as “Stuff”, and so it is makes sense that synchronization was intermediate for this video.

5.5 Individual differences in cortical parcellation of anatomy

In document Neural Pattern Similarity and Visual Perception (Page 174-177)