Feature extraction

Various features were computed from the motion (HMD and LM) and physi- ological (HR and EMG) sensors used in the previous study. Table 5.2 details

Table 5.2: Features extracted of each sensor.

Sensor Activity Measured Features Extracted

HR and EMG

Cardiac Activity and

Hand’s Muscle Activation Mean

Standard deviation Maximum

Mean of 1st derivative of raw signal Mean of 1st derivative of normalised signal Mean of 2nd derivative of raw signal Mean of 2nd derivative of normalised signal

HR Cardiac Activity Mean of HRV AVNN Mean of HRV rMSSD Mean of HRV LF/HF Ratio HMD and LM

Head and Hand’s Velocity and Acceleration

Mean

Standard deviation Maximum

Mean of peaks’ width as well as of the initial and final slopes Standard deviation of peaks’ width as well as of the initial and final slopes Maximum of peaks’ width

as well as of the initial and final slopes Number of peaks

LM Hand’s motion Number of zeros1

1_{Time the playing hand is relaxing, out of Leap Motion’s field of view.} Total number of features of each sensor: HMD: 26; LM: 27; EMG: 7; HR: 10.

all features extracted. This section describes the features extracted from each sensor.

Motion features

Motion features were calculated using the velocity and acceleration magnitudes of the HMD and the hand used to interact with the game, tracked with LM. Only the hand’s palm motion was recorded for feature extraction, no fingers were analysed. Prior to the extraction of motion features, a z-score normalisation was applied to the motion’s velocity and acceleration of both HMD and LM, having zero mean and one standard deviation (Formula 5.1). Since velocity has a magnitude and direction, absolute values were calculated for each axis of both the HMD and the hand’s motion, disregarding the direction. Acceleration is the

rate of change of velocity of an object with respect to time 2_{. Magnitude and}

direction were preserved as they provided information about the acceleration and deceleration. Once velocity and acceleration were calculated, the axial components of each feature were summed and square-rooted to calculate the magnitude of both features. The acceleration’s magnitude, also called quantity of motion (QoM), has been one of the most successful motion features to classify emotions [35]. Mean, maximum and standard deviation of both velocity and acceleration were computed for the HMD and hand’s motion.

When playing in VR, participants had to keep their hand suspended in front of the HMD so Leap Motion could track their hand gestures. In order to avoid arm fatigue, participants were encouraged to relax and put the arm down when possible, disrupting the hand’s tracking. This absence of hand detection turned into another feature describing how much time participants had their hand down. This feature was computed counting the number of zeros (i.e. no hand detected) in Leap Motion’s data.

x = xi− µ

σ (5.1)

Following Castellano’s mathematical model to analyse gestural expressivity [35], various motion features were derived such as the peaks’ slopes and duration of the velocity and acceleration of both the HMD and Leap Motion (LM). Since there was no specific gestures labeled with emotions, as in Castellano’s work, a peak detection algorithm was used to find all the peaks with a threshold of 2 and a minimum distance between peaks of 0.5. Prior to the extraction of these features, velocity and acceleration were normalised (Formula 5.1), having zero mean and one standard deviation. To calculate the initial and final slopes as well as the duration of the peak, it was necessary to detect the valleys before

and after each peak found. This was accomplished by inverting the signal, so that what used to be valleys would present as peaks. Another peak detection algorithm was applied to the inverted signal, with a threshold of -0.1 and no minimum peak distance. This returned all the valleys in the original signal smaller than 0.1. For each peak’s location, the valleys before and after were identified according to the valley’s location. Then the initial slope was calculated subtracting the value of the valley before the peak from the peak’s value, and then dividing by the time difference between this valley and the peak. The same method was used for the end slope calculation. Finally, the peak’s duration - in milliseconds - was calculated by subtracting the timestamp of the valley before the peak from the timestamp of the valley after it. Mean, standard deviation and maximum values were calculated for the peaks’ duration, start and final slopes. The number of peaks detected were also used as a feature. This resulted in a total of 36 features for the HMD, and 37 for the hand’s motion. A z-score normalization was applied to all computed features using their own mean and standard deviation.

Physiological features

Before computing the physiological features, participants’ HR and HRV were normalised using their own baseline recorded while resting at the beginning of each session. No normalisation was needed for the EMG as it was calibrated in every session.

Six features were computed for the physiological sensors (HR and EMG) fol- lowing Picard, Vyzas and Healey’s proposed features [122] to measure emotions from physiological signals (see Formulas 5.2-5.7). The main advantage of these features is that they can be easily computed in real-time, which makes them suitable for implementation in Memory Break. The maximum of each signal was also added to the feature sets.

The HR feature set included the mean of three heart rate variability (HRV) features: the root mean squared of successive differences (rMSSD), the average N-N intervals (AVNN) and the low to high frequency ratio (LF/HF). Since the HRV recordings were made every 30s (sampling rate of 0.03Hz), the same time as one game’s section, only one HRV observation could be considered for each section. Therefore, no further features could be obtained from only one HRV observation. The raw values of AVNN, rMSSD and LF/HF ratio were included in the set of HR features. This resulted in a total of 10 HR features and 7 EMG features. Again, a z-score normalization (Formula 5.1) was applied to all

µ = 1 N N X n=1 Xn (5.2) σx= ( 1 N − 1 N X n=1 (Xn− µx)2)1/2 (5.3) δx= 1 N − 1 N −1 X n=1 |Xn+1− Xn| (5.4) ˜ δx= 1 N − 1 N −1 X n=1    ˜ Xn+1− ˜Xn   = δx σx (5.5) γx= 1 N − 2 N −2 X n=1 |Xn+2− Xn| (5.6) ˜ γx= 1 N − 2 N −2 X n=1    ˜ Xn+2− ˜Xn   = γx σx (5.7)