Mobile brain imaging to facilitate embodied experiments with humanoid

In Chapter 5, my co-authors and I provided an intermediate reflection and opinion on the current state-of-the-art neuroscience tools in embodied human- robot interaction studies. Overall, we foresee that the field will be driven forward by more transparent, embodied and mobile neuroscience. Here we especially highlighted the ‘promises and pitfalls’ of fNIRS, which affords the advantage of allowing brain imaging during embodied and interactive encounters with social robots. Recent advances in the development of portable and

lightweight fNIRS systems can support more ecologically valid experimental interaction paradigms.

Finally, my co-authors and I argue that it will become important in the coming years to move beyond a predominantly anthropocentric approach and consider other comparison categories (for example pets or objects), as well as brain regions beyond popular ‘hub regions’ of social cognition. I will reflect on this point in the final sections of this thesis discussion.

In this chapter, we also highlighted the repercussions of the replication crisis, and the move towards open science practices in psychology and neuroscience. Especially when implementing fNIRS in embodied human-robot encounters it is important to follow a stepwise approach, starting with a replication of effects. We argued that it is in the interest of various stakeholders that research builds on strong foundations, using rigorous and robust methods.

Chapter 6 implemented the proposed stepwise approach of the previous

chapter: here we sought to validate a novel mobile fNIRS system, by comparing this brain imaging modality to the current gold-standard in social neuroscience: fMRI. A second contribution of this chapter was the adaptation of a new,

photogrammetry-based method to co-registering the positions of the fNIRS probes. The main focus of this chapter was to transparently describe and highlight challenges encountered when using this brain imaging method, especially in comparison with the results yielded by fMRI.

We investigated two main questions: the first being the overlap sensitivity

between the placement of the optodes and the functional regions of interest – as determined by fMRI. Secondly, we were interested in the detection rate of each modality at the single-subject level. After extensive piloting we constructed a reliable photogrammetry pipeline that resulted in 11 out of 12 excellent quality head models. We provide a detailed description of every step in the openly available manual (Henschel, Kent, et al., 2020c). This new method to digitise optode positions was only successful after adding colourful markers to the fNIRS cap. We found that the automatic identification algorithm of the janus3D co- registration toolbox (Clausner et al., 2017) was not reliable for the fNIRS optode holders (as opposed to the EEG electrode holders it was designed to detect).

Consequently, the experimenters had to choose a more time-consuming route of manual tagging the optode locations in janus3D.

While we found that the position of the optodes overlapped well with

participants’ functional MRI activity, this method, as highlighted by Powell and Saxe (2018), is not without issues. It is challenging to support the success or failure of probe location overlap based on single-subject functional activity, as anatomical and functional locations were highly variable. The detection rate on the single subject level was better for fMRI, as compared to fNIRS. Indeed, by showing composite images of the optodes mapped to subject anatomical and functional MRI images, we identified that strong inter-subject variability in terms of scalp-brain distance was one of the contributing factors. However, we also investigated the possibility that an anticipatory shift had taken place, as a recent paper by Richardson and colleagues had shown that repeat viewing of this particular localiser movie led to a predictive response of the brain (2020).

Shifting the events forward in time did not lead to more observed fNIRS activity; on the contrary, the activity was even more subdued.

Overall, we can conclude that on the single-subject level, the signal recorded with fNIRS showed a lower signal to noise ratio than the fMRI signal, which replicated reliable activation of the left and right TPJ almost 100% of the time across all subjects. This point is important, as future studies will seek to implement the fNIRS system in mobile interaction paradigms with robots, and thus through more subject movement, noise levels might further increase. In the current task, subjects sat still and passively observed a movie, which still only resulted in a success rate of elicited fNRIS activity in just over half of the participants. This will be a crucial factor when planning subject recruitment numbers, and when calculating power for future studies.

7.4.

What does the replication crisis mean for the future

of HRI research?

Most empirical findings reported in this thesis are null results, which are

inherently challenging to interpret due to the complex factors that could be at play contributing to the failure of conceptually extending an effect (Shrout &

Rodgers, 2018; Simmons et al., 2017). Publishing null results has become more acceptable in the research community in recent years, as more and more scientists grappled with replicating well-known effects. This led to the

replication crisis in psychology and neuroscience (Aarts et al., 2015; Schimmack, 2020; Shrout & Rodgers, 2018). As is perhaps evident through examples

presented in this thesis, the ripples of these crises have not been limited to psychology and neuroscience. They are reaching HRI as well, exacerbating the interdisciplinary tensions discussed in Chapter 2 (Irfan et al., 2018).

Some researchers note that trust in the ‘social sciences’ has been shaken as a result of the crisis (Irfan et al., 2018), others have a more optimistic outlook: indeed, as Shrout and Rodgers (2018) argue, the sense of urgency invoked through the imagery of a crisis has sparked fundamental and sustained positive change. The authors highlight how, as a result of this crisis, some of the open science movement’s most valuable tools have been created: the Open Science Foundation (Nosek, 2013) and a global shift in research conventions, such as the increased adoption of pre-registering hypotheses, design and analysis plans (2018). As Schimmack (2020) writes in his perspective on replication ‘failures’ (where a recent Nature Human Behaviour editorial argues strongly for the fact that “replications do not fail”; Kousta, 2020, p. 559), the crises in psychology and neuroscience might serve as an important warning message to other

disciplines, who have not yet felt the severe repercussions riding on their back.

7.4.1. Adoption of open science methods among the HRI community

Researchers in HRI have acknowledged this crisis and have started to adopt some of the practices that scientists in related fields have been lobbying for over the past nine years (Baxter et al., 2016; Belpaeme, 2020; Irfan et al., 2018;

Schimmack, 2020; Strait et al., 2020). Baxter and colleagues (2016), who analysed three years of proceedings stemming from the field’s most important meeting, the ACM-HRI conference, note that they observe a clear trend towards the adoption of open science practices: for example, journals such as PLOSOne now require datasets to be shared openly. Taking even more positive steps in this direction, in 2020, ACM-HRI for the first time explicitly invited replication studies in a dedicated conference track, with five proceedings replicating their studies across robotic platforms and data collection sites (Kubota et al., 2020;

James Li et al., 2020; A. Pereira et al., 2020; Sandygulova et al., 2020; Strait et al., 2020).

Strait and colleagues (2020) conducted a conceptual extension and three-site replication of the robot-adapted Joint Simon Effect (JSE), which examines how people represent the actions of a robotic co-actor (Stenzel et al., 2012). In their collaborative replication effort, the authors found that the expected JSE

replicated. The authors noted that a key component to the success of this effort was that all three, international replication sites had access to the same

platform: the Nao robot (Strait et al., 2020). This speaks to an observation described in Chapter 2: commercially available humanoid robots offer many advantages to HRI researchers, despite their limited social abilities ‘in the wild’. Strait and colleagues (2020) conclude with the reflection that replicating effects will become ever more important in the HRI community, especially given the fact that attempts of running source code from the 2017 IEEE International Conference on Robotics and Automation (ICRA) proceedings were crowned with success in only 2% of the cases (Cervera, 2019).

7.4.2. Adoption of open science methods in the fNIRS community

However, HRI is not the only research community to be weighing up the next steps in a move toward more open and reproducible research practices: the relatively young field surrounding fNIRS research has been relatively slow to adopt open science conventions already embraced in other neuroimaging communities, for instance among fMRI researchers (Bratt, 2017). Currently, there are not many open fNIRS data sets available, and perhaps as a

consequence, meta-analytical investigation of the robustness of social and cognitive tasks measured with fNIRS are scarce (Bendahan et al., 2019). This is mainly due to the heterogeneity of fNIRS devices and data processing methods used – for example Bendahan and colleagues (2019) sought to conduct a meta- analysis on the connection between cerebral oxygenation and the presence of delirium in patients. The meta-analysis could not be completed due to a large variability in fNIRS systems and pre-processing procedures. Increased use of standardized localiser tasks in fNIRS could lead to a better comparability across studies, and perhaps an increased motivation to share data. As of October 2020, there are only 7 datasets shared on OpenFNIRS (Figure 26,

https://openfnirs.org/data/).

Figure 26 - The Openfnirs database (https://openfnirs.org/data/).

Another recent development in the fNIRS community may prompt an increased uptake of open science practices, as the sNIRF format has been introduced (https://github.com/fNIRS/snirf). Encouragingly, a brain imaging data structure (BIDS) extension proposal for NIRS has been raised (BIDS Extension Proposal,

BEP030), which implements the file-naming and file-structure convention initially developed for fMRI (and which was used in Chapter 6 to run the standardized pre-processing pipeline fMRIprep). The BIDS format, which is essentially a standardized brain data management plan, has the potential to greatly enhance reproducibility and open science efforts (Gorgolewski et al., 2016). The extension of BIDS for NIRS will incorporate the already developed sNIRF raw data format, which can be converted from most proprietary

manufacturer file formats (e.g. in Chapter 6 we initially translated the Shimadzu files to the nirs format).

Bratt (2017) laments a lack of open data repositories for fNIRS (in studies on emotion – yet, as Figure 26 shows, this is a global problem), in particular relating to the ‘curse of dimensionality’ known in the machine learning field.

Researchers who may want to use machine learning procedures to analyse their data, are faced with small samples in fNIRS studies. Using machine learning classifiers on highly dimensional data, such as the fNIRS signal, is challenging as these data have many features with relatively few observations (Jie Li et al., 2016). As a result, models are ‘overfit’ and are not generalisable beyond individual datasets (Bratt, 2017). One way to address this issue (and in the future implement machine-learning methods for fNIRS) may be the aggregation

of large, open datasets. In addition, Bratt (2017) argues that while fNIRS and FMRI are facing validity issues, increased data sharing, including text-based metainformation (see Neurosynth, https://neurosynth.org/), also has the

potential to facilitate communication (and collaboration) between international fNIRS labs.

To summarise, the HRI research community can greatly benefit from using fNIRS as an innovative tool in embodied interaction studies with humanoid robots. HRI researchers, who use this new methodology, may further profit from improving data sharing infrastructure like the sNIRF file format and data bases such as Openfnirs.