This section describes the methodology employed to collect self-reported, phys- iological and motion data, as well as the pre-processing and feature extraction of this data.
3.2.1
Participants
Eight players (four pairs) took part in the study with a mean age of 30.88 (SD: 4.28). A similar sample size of 10 pairs was used by Mandryk and Inkpen in a previous study [67]. Three of them were female and five male. Although none of participants played Wii frequently, four of them played video games between 3 and 5 hours in the last week and only one more than 5 hours. The most common platform to play video games was smartphones. Half of the players reported to prefer playing video games alone, three collaboratively and only one competitively. Participants were recruited via email or word of mouth, trying to involve people with different backgrounds, ages and sex. No prior experience was required to participate in this study, except both players within a pair had to know each other before the experiment to increase the chances of collaboration between them [32].
3.2.2
Sensors and Questionnaires
Self-reported and physiological data was recorded from all participants in every play mode. Before starting the experiment, participants were instructed about the data measures, sensors used and how they should wear them. Table 3.1 summarises the data gathered during the study as well as the sensors employed and features extracted.
Two physiological sensors were used with each participant. A Shimmer3
ECG sampled at a rate of 512Hz, measured the heart’s electrical activity. Par- ticipants were instructed about how to place the four ECG electrodes in their chest (see Fig. 3.2). The second physiological sensor was a Shimmer GSR, which measures the electrodermal activity of the user’s sweat glands. This ac- tivity is measured by passing a low voltage across two electrodes attached to the user’s index and middle finger (see Fig. 3.2). The electrodermal activity varies with the state of the sweat glands of the skin, which are normally associated with stress and anxiety as they are related to the sympathetic nervous system, being a good indicator of emotional arousal [41]. This sensor was placed in the hand holding the game controller and sampled at 51Hz. The GSR also had an accelerometer incorporated to record the movements of the hand holding the
controller. Due to individual differences in physiological signals, baseline activ- ity levels were recorded at the beginning of the study for all sensors to normalise these differences. Both sensors broadcasted the data wirelessly to a Windows laptop, where the Multi Shimmer Sync software recorded them.
Table 3.1: Objective (continuous) and subjective (self-reported) data recorded.
Measure Sensor / method Features Type of data
ECG ShimmerECG sensor
Heart Rate (HR): mean & SD. Inter-Beat Interval (IBI): mean. Heart Rate Variability (HRV): Root Mean Square of Successive Differences (rMSSD)
Quantitative
GSR ShimmerGSR sensor Skin Conductance Level (SCL): mean & SD.Skin Conductance Response(SCR): mean & SD. Quantitative Motion Accelerometer Number of throws (peaks), Quantity of motion,highest peak (velocity throw). Quantitative
Video Front-facingcamera Gestures, posture (body position), spatial behaviour,number of gazes, positive and negative facial expressions...
Quantitative and qualitative Self-report PRE, PPQ & POST Engagement, immersion, frustration, stress,enjoyment, effort, boredom. Quantitative
Three questionnaires were designed using 5-point Likert scales (see Appendix A). A pre-experiment questionnaire (PRE) was given at the beginning of the study, asking demographic (i.e.: gender, age, occupation...) and gaming habit questions such as number of hours spent playing video games in the last month or preferences of video games types. A second questionnaire was given to partic- ipants after they finished each play mode. This post-play questionnaire (PPQ) contained questions extracted from two validated questionnaires: the Game Engagement Questionnaire by Brockmyer et al. [29], and the Immersion Ques- tionnaire by Jennet et al. [69]. In this questionnaire participants reported their levels of enjoyment, effort, engagement and immersion overall and with his or her partner. Questions were randomised to avoid any bias due to repetition of the questionnaire after each play mode. Finally, a post-experiment ques- tionnaire (POST) was completed at the end of the study, where participants reported their preferences of each play mode in terms of the most fun, bor- ing, frustrating, etc. Participants also reported their overall engagement and enjoyment levels for each play mode.
Finally, a video camera placed next to the monitor displaying the game recorded the study for later observational analysis and qualitative data extrac- tion. Since participants had to play standing up, these recordings were impor- tant to look at their non-verbal behavioural cues such as postures, gestures or facial expressions.
350 400 450 500 0 1000 2000 3000 4000 5000 Time (ms) µ Siemens GSR 0 10 20 30 40 0 1000 2000 3000 4000 5000 Time (ms) m/s 2 Accelerometer 50 60 0 25 50 75 100 125 Time (s)
Beats per min
ute
Heart Rate
Figure 3.3: Sample data of GSR (top), accelerometer’s magnitude (middle) and HR (bottom).
3.2.3
Data pre-processing
All the data gathered from the ECG, GSR and accelerometer during the study was imported into MATLAB. Although a sampling rate of 512Hz and 51Hz were set up for the ECG and GSR sensors respectively, all the sensors’ data was recorded at 51Hz due to problems with the software recording all sensors’ data (Multi Shimmer Sync).
Data from all sensors was plotted to check if it was correct. The GSR data was extremely noisy since it was placed in the hand holding the controller, which was constantly moving and shaking. As shown in Figure 3.4, the peaks in the GSR signal match the accelerometer’s peaks, which correspond with the throwing gestures made while playing. Different filters such as lowpass or moving average filter were applied to remove the noise introduced by the hand movements, making the data considerably smoother although it was still too noisy to use in this analysis so it was discarded. Previous works in affective gaming research have unadvised using GSR for fast paced games that require rapid movements or fingered dexterity [154]. Thus, GSR is suitable only for games that induce relaxation as the hand with the GSR sensor attached needs to be still at all times.
ECGTools4 was used to analyse the ECG data and extract the R-peaks,
which corresponds to individual heart beats. The distance between consecutive R-peaks (also called R-R intervals) was calculated to find the Heart Rate (HR) values per second. The R-R intervals were also used to compute the Inter-Beat Intervals (IBI), which represents the distance in milliseconds between individual heart beats. The HR and IBI are closely related since a higher HR implies a smaller the time between R-R intervals, which mean smaller IBI values.
3.2.4
Feature Computation
Using the IBI calculated from the R-R intervals, different Heart Rate Variability (HRV) features can be extracted. HRV measures the variation of the frequency of heart beats over time. Since there are different ways to measure the HRV, both in the time domain (AVNN, SDNN, rMSSD, etc) and frequency domain (LH, HF), the Root Mean Square of Successive Differences (rMSSD) was calcu- lated as it is one of the most common measures in the time domain [150, 153]. While HR has been demonstrated to be associated with emotional regulation and arousal [158], HRV is linked with stress and mental efforts like engagement [10][67]. The mean values of HR, IBI and rMSSD were calculated for each par-
400 500 600 700 0 5000 10000 15000 20000 Time (ms) µ Siemens GSR 0 10 20 30 40 0 5000 10000 15000 20000 Time (ms) m/s 2 Accelerometer
Figure 3.4: GSR (top) and accelerometer magnitude (bottom) during the com- petitive condition.
Figure 3.5: Accelerometer peak detection. Each triangle corresponds with one throw.
have an evenly spaced time series in order to be able to do a continuous analysis over time.
The acceleration magnitude (change over time in velocity, m/s2) of the hand
holding game controller was computed using the absolute value of the accelerom- eter’s X, Y and Z axes. This acceleration magnitude was not gravity-free. The mean and standard deviation of the acceleration magnitude, hereafter referred to as Quantity of Motion (QoM), were calculated. Using a peak detection (see Fig. 3.5) algorithm in MATLAB, participants’ throwing gestures were identi- fied. Thereby, the number of throw gestures of each participant per play mode were calculated. In order to assure all throw gestures were detected, the pa- rameters of the peak detection algorithm were adjusted until the peaks detected corresponded with all the observed throws in the video recordings.
A qualitative analysis of the videos was carried out. Since some record- ings were a bit blurry, it was difficult to use Computer Vision techniques to accurately detect participant’s facial expressions to perform an automatic emo- tion recognition analysis. Therefore, video recordings were manually annotated to determine the predominant facial expressions in each play mode as well as the non-verbal behaviours such as gestures, postures or spatial behaviour (i.e.: moving around the room). This analysis investigated how participants express the experienced affective states through their body movements and facial ex- pressions, distinguishing between positive, negative and neutral states. Finally, the data gathered from the PRE, PPQ and POST questionnaires was imported into the statistical analysis software SPSS (Version 30) since it did not need any pre-processing.