Supplementary material

Supplementary Table 1 Mean accuracy when performing classification based on

sensor-space data, on source-space activity reconstructed with the LCMV beamformer, and on source-space activity reconstructed with the dynamic beamformer. Grand mean accuracies of the different data representations (sensor space and source space) are shown in the last row. Grand mean accuracies of the different contrasts are shown in the last column. The perceived images were corrected for luminance. Standard deviations are given between brackets. Asterisks indicate that the classification accuracy was significantly higher than chance level for all subjects (Bonferroni corrected p < 0.05). Values in bold font indicate a significant increase in accuracy for at least one subject compared to sensor-space accuracy (Bonferroni corrected p < 0.05). Values in italic font indicate a trend towards increase in accuracy for at least one subject compared to sensor-space accuracy (uncorrected p < 0.05). It should be noted that, in addition to the overall trend of improved accuracies for source-space activity, classification performance decreased for LCMV beamformer activity for one subject when contrasting faces with scenes and bodies, as well as contrasting bodies with scenes and tools for source-space activity reconstructed with both the LCMV beamformer and the dynamic beamformer. Of note is that it is always the same subject that shows a decrease in classification performance. This is also the subject for which classification performances based on source-space activity are not always significant.

Contrast Sensor level (sd) LCMV

beamformer (sd) Dynamic beamformer (sd) Contrast average face-tool 0.82 (0.07) * 0.85 (0.09) * 0.91 (0.05) * 0.86 face-scene 0.86 (0.02) * 0.85 (0.13) * 0.89 (0.12) * 0.87 face-body 0.82 (0.07) * 0.85 (0.07) * 0.91 (0.03) * 0.86 scene-body 0.77 (0.05) * 0.80 (0.12) 0.84 (0.20) 0.80 body-tool 0.72 (0.09) 0.79 (0.10) 0.77 (0.11) 0.76 scene-tool 0.68 (0.16) 0.74 (0.17) 0.81 (0.19) 0.74 average 0.78 0.81 0.85

Supplementary Table 1 Mean accuracy when performing classification based on sensor-space data, on source-space activity reconstructed with the LCMV beamformer, and on source-space activity reconstructed with the dynamic beamformer. Grand mean accuracies of the different data representations (sensor space and source space) are shown in the last row. Grand mean accuracies of the different contrasts are shown in the last column. The perceived images were corrected for luminance. Standard deviations are given between brackets. Asterisks indicate that the classification accuracy was significantly higher than chance level for all subjects (Bonferroni corrected p < 0.05). Values in bold font indicate a significant increase in accuracy for at least one subject compared to sensor-space accuracy (Bonferroni corrected p < 0.05). Values in italic font indicate a trend towards increase in accuracy for at least one subject compared to sensor-space accuracy (uncorrected p < 0.05). It should be noted that, in addition to the overall trend of improved accuracies for source-space activity, classification performance decreased for LCMV beamformer activity for one subject when contrasting faces with scenes and bodies, as well as contrasting bodies with scenes and tools for source-space activity reconstructed with both the LCMV beamformer and the dynamic beamformer. Of note is that it is always the same subject that shows a decrease in classification performance. This is also the subject for which classification performances based on source-space activity are not always significant.

Contrast Sensor level (sd) LCMV

representations of working memory in previous studies (Fuentemilla et al., 2010; Jafarpour et al., 2014; LaRocque et al., 2013; Polanía et al., 2012), and to obtain a complete view of the spatiotemporal dynamics of representations it would be vital to extend the method applied in this thesis to the spectro-spatiotemporal domain.

Finally, when accuracies are elevated such that reliable representations can be assessed, it will be important to determine what type of categorical information is represented exactly, and whether different stimulus properties are represented throughout the spatiotemporal pattern. In addition, high accuracies will allow us to assess several of the questions I set out to answer in Chapter 4, but could not reliably do. For example, the spatiotemporal patterns of the content of working memory during and after reinstatement could be assessed more reliably. Moreover, with high accuracies it would be possible to assess how multiple items are represented in working memory, as well as how information flows related to top-down and bottom-up processing.

Conclusion

In this thesis, I have assessed the spatiotemporal representations of visual perception, working memory and auditory word perception. Specifically, I have shown how visual information, both during perception and memory is represented in space and time, and how task-relevance and the specifics of stimulus properties influence these representations. As such, I have provided insight into the dynamics of various neuronal processes involved when we perceive, maintain and recall complex information from our environment, by using a method that, when adapted to the specific requirements of the cognitive process under investigation, allows us to investigate the spatiotemporal dynamics of any cognitive process.

Supplementary material

Supplementary Table 1 Mean accuracy when performing classification based on

Contrast Sensor level (sd) LCMV

Supplementary material

Contrast Sensor level (sd) LCMV

Supplementary Figure 1 Individual maps of time-averaged regression coefficients

for the discrimination between luminance corrected faces and tools based on source-space activity time-courses 115 to 315 ms after stimulus onset. Warm colours indicate positive regression coefficients; cold colours indicate negative regression coefficients. A) Regression coefficients of classification based on source- space activity reconstructed with the LCMV beamformer. The respective accuracies corresponding to these maps are 0.80, 0.80 and 0.95. B) Regression coefficients of classification based on source-space activity reconstructed using the dynamic beamformer. The respective accuracies corresponding to these maps are 0.91, 0.87 and 0.96.

Supplementary Figure 2 Average accuracy traces for luminance corrected faces

versus tools. The red areas around the traces indicate the 95% confidence interval. Stimulus onset is at 0 s. The dashed horizontal lines indicate chance level performance. The solid horizontal lines signify the FDR-corrected threshold for deviation from chance level. After the initial peak, classification performance remained sustained around the FDR-corrected threshold for this contrast. A) Average accuracy trace based on source-space activity reconstructed with the LCMV beamformer. The latency for which the trace starts to rise significantly above the FDR-corrected threshold is 81 ms. B) Average accuracy trace based on source- space activity reconstructed with the dynamic beamformer. The latency for which the trace starts to rise significantly above the FDR-corrected threshold is 85 ms.

Supplementary Figure 1 Individual maps of time-averaged regression coefficients

Supplementary Figure 2 Average accuracy traces for luminance corrected faces

Supplementary Figure 3 A) Average accuracy traces for the contrast between

luminance and spatial frequency corrected faces and scenes reconstructed with source-space activity reconstructed by the LCMV beamformer. B, C, D) Individual accuracies for this contrast. Horizontal lines indicate the FDR-corrected chance level. Although the average accuracy trace did not remain sustained above chance level, this is the case for subject S1 (B) and to a lesser extent subject S3 (D).

Supplementary Table 2 Overview of the onset latency at which the average

accuracy trace of different contrasts of luminance corrected images first rises significantly above the FDR-corrected chance level, and the peak latency at which the maximum classification accuracy is reached.

Contrast LCMV beamformer

Onset / peak (ms) Dynamic beamformer Onset/ peak (ms)

face-tool 81.7 /125.0 85.0 / 125.0 face-scene 81.7 / 118.3 81.7 / 125.0 face-body 65.0 / 118.3 68.3 / 115.0 scene-body 68.3 / 148.3 68.3 / 85.0 body-tool 148.3 / 148.3 85.0 / 148.3 scene-tool 85.0 / 95.0 85.0 / 91.7

Supplementary Table 2 Overview of the onset latency at which the average

Contrast LCMV beamformer

Onset / peak (ms) Dynamic beamformer Onset/ peak (ms)

Supplementary Figure 3 A) Average accuracy traces for the contrast between

Supplementary Table 2 Overview of the onset latency at which the average

Contrast LCMV beamformer

Onset / peak (ms) Dynamic beamformer Onset/ peak (ms)

Supplementary Table 2 Overview of the onset latency at which the average

Contrast LCMV beamformer

Onset / peak (ms) Dynamic beamformer Onset/ peak (ms)

Supplementary Figure 4 Localization of regression coefficients for single time

samples during the initial peak and the sustained period. Warm colours are indicative of positive regression coefficients; cold colours indicate negative coefficients. Data come from subject S1 (A) and S2 (B) for the contrast of luminance

and spatial frequency corrected faces versus tools. Data of subject S3 are shown in Figure 2.5 of the main text. Source-space activity time-courses were reconstructed with the dynamic beamformer. Note the involvement of similar clusters in the inferior occipital gyrus (IOG), middle occipital gyrus (MOG), inferior temporal lobe (ITG) and superior parietal gyrus (SPG). Because for the purpose of consistency data are shown at the same single time points as in Figure 2.5 of the main text, these plots are not sampled optimally to show every effect.

Supplementary Figure 5 Effect of the parameter  on the focality of the resulting

sources. Time averaged regression coefficients are shown for the discrimination between faces and tools corrected for luminance and spatial frequency, based on source-space activity time-courses reconstructed with the dynamic beamformer 115 to 315 ms after stimulus onset. Warm colours indicate positive regression coefficients; cold colours indicate negative regression coefficients. A) Regression coefficients when  is 0.01, which is used throughout the paper. The corresponding accuracies are 0.89, 0.85 and 0.96 B) Regression coefficients when  is 0.99. The corresponding accuracies are respectively 0.89, 0.86 and 0.97. Note the more focal sources, which are in line with a bias for L1 regularization. The sources, albeit

smaller, are located at the same place as with the lower . Also note that the regression coefficients are higher when α is 0.99 than when  is 0.01.

Supplementary Figure 5 Effect of the parameter  on the focality of the resulting

smaller, are located at the same place as with the lower . Also note that the regression coefficients are higher when α is 0.99 than when  is 0.01.

Supplementary Figure 4 Localization of regression coefficients for single time

Supplementary Figure 5 Effect of the parameter  on the focality of the resulting

smaller, are located at the same place as with the lower . Also note that the regression coefficients are higher when α is 0.99 than when  is 0.01.

Supplementary Figure 5 Effect of the parameter  on the focality of the resulting

smaller, are located at the same place as with the lower . Also note that the regression coefficients are higher when α is 0.99 than when  is 0.01.

In document Decoding the perception and memory trace: Neuronal representations of perceptual and memory content in time and space (Page 121-129)