Representational Similarity Analysis

We analysed the representational geometry of neural responses to our 6 experimental conditions (3 actions for 2 modalities) by calculating the representational dissimilarity matrix (RDM) (Kriegeskorte et al., 2008a). For this analysis, we used an RSA toolbox developed by Nili et al., (2014). We obtained the t-maps for each action of each modality using BrainVoyager.

For each ROI, we computed the pairwise correlation between all the activity patterns associated with conditions using correlation distance (1- Pearson linear correlation), yielding a 6×6 RDM. RDMs were calculated separately for each run and averaged for each subject, which

156

produced 10 RDMs (one per subject) for each of the 12 ROIs. These RDMs were used to estimate the noise ceiling and the similarity between each subject’s RDM and model RDMs. Then we employed multidimensional scaling (MDS) to arrange the high-dimensional RDM space onto 2 dimensional space, such that the distance between them reflected the similarities between the response patterns.

We then compared the 12 ROIs RDMs against 5 model predictions to arbitrate between theoretical stances regarding AO+MI and MI. These models were (figure 5.4):

 Pure modality model: This model assumes a categorical distinction between the 2 modalities (AO+MI and MI). In this model RDM, the dissimilarities for all within- modalities were 0 and for a cross-modality were 1.

 Pure actions type model: This model assumes a categorical distinction between the 3 actions (lifting, knock and throwing), regardless of the modality. The dissimilarities between identical action types were 0 and between different actions were 1.

 Mixed models: 3 models were used to predict the dependency of a given neural pattern on modality as well as action type. In all models, the dissimilarities between different actions within the same modality were 0.5 and between modalities were 1. The dissimilarities of an identical action between modalities varied in accumulated steps of 0.25, and they were 0.25, 0.5 and 0.75, named as M25, M50, and M75 respectively. These model predictions examined different degrees of similarity between neural patterns evoked by the AO+MI and MI of a given action. For instance, the M75 model assumes a low degree of similarity between neural patterns associated with a given action type during AO+MI and MI.

We compared ROIs and model RDMs using Kendall’s rank correlation coefficient τA, which is the proportion of pairs of values that are consistently ordered in both variables. Kendall’s τA is recommended when comparing models that predict tied ranks (model RDMs)

157

with models that make more detailed predictions (ROIs RDMs) (Nili et al., 2014). To assess the significance by which these models explain variance in a given ROI RDM, a one-sided Wilcoxon signed-rank analysis was used across subjects. To account for multiple testing, we controlled the false-discovery rate at 0.05.

The amount of variance explained by a model RDM is limited by the variability across subjects. Thus, an estimation of the noise ceiling is needed to indicate how much variance of ROIs RDMs was expected to be explainable by a model RDM (given the noise level). The noise ceiling consists of upper and lower edges corresponding to upper and lower bound estimates on the group-average correlation with the RDM predicted by the unknown true model. The average of all subject RDMs can be used as an estimate of the true model RDM. The average correlation of this average RDM provides the upper bound. We estimated the lower bound by employing a leave-one-subject-out approach. We computed and averaged each single-subject RDMs correlation with the average of the other subjects’ RDMs. A model RDM is assumed to capture the true dissimilarity structure of a given ROI RDM when its correlation reaches the lower bound of the ceiling.

158

Figure 5.4. RDM of different model predictions that assume different similarities of action pattern based on action type and modality. The first RDM shows a modality based model that

assumes equal neural patterns for each action within a similar modality. The second RDM is an action type based model that assumes similar neural responses for a given action across modalities. The remaining 3 mixed models assume variable dependency between modality and action type, and the dissimilarity of action types were varied as: 0.25(M25), 0.5(M50) and 0.75(M75) respectively. All the mixed models assume a dissimilarity of 0.5 between different action types within one modality, and a dissimilarity of 1 between different action

types between two modalities. AO+MI: Action observation and motor imagery, MI: motor imagery, L: Lift, K: Knock and T: Throw.

5.4 Results

The results of the two fMRI sessions are reported. In Session 1, 10 subjects performed an AO+MI task to presentation of 25 videos which displayed a parametric set of action blends of lifting, knocking and throwing. These data were used to compute hyperalignment transformation parameters that allowed us to transform the data of individual subjects into a common model space. In Session 2, the same subjects performed AO+MI and MI to presentation of 3 videos which displayed the actions lifting, knocking and throwing. From these data, SVM classifiers were used to decode the action types during the AO+MI and MI conditions, by using individual subject data and hyperaligned data. In order to do this, we first explored the subject order that produced the highest performance of the common model space, and mapped all the subjects’ AO+MI and MI data onto it. Second, we used searchlight to explore the voxels that carried information about the content of the actions using the individual

Modality Action type M25 M50 M75

A O +M I MI L K T L K T MI AO+MI L K T L K T

159

data and the hyperaligned data. Then, we compared WSC and BSC based on hyperalignment for each modality separately. Finally, we examined the similarity of the neural codes between AO+MI and MI of different arm actions, using RSA.