3.5.1
Affect in Education
The importance of affect during learning episodes has been established in literature. Picard (1997) asserts that the reason humans are so good at many tasks, is because we process data through the lens of emotion, particularly when it comes to perception. Learning is significantly enhanced when students feel empathy from their teacher (Graham and Weiner 1996;Zimmer- man 2000). The interpersonal relationships between teachers and students are linked to long term increases in motivation (Royer and Walles 2007;Wentzel and Asher 1995). D’Mello and Graesser (2009); D’Mello et al. (2006); Graesser et al. (2006); Kort et al. (2001) argue that emotions play a significant role in the deep learning of conceptual information.
Figure 3.2: Proposed Emotional Framework for Learning (Kort et al. 2001)
Figure3.2provides a framework for the assessment of student affect during learning episodes. While there are a number of ‘negative emotions,’ the proposed model of learning has learners cyclically moving between a number of different stages, such as curiosity to confusion, to frus- tration and ultimately hopefulness. This cycle of positive and negative affect is supported by Shute et al.(2015).
General affect detection focuses on anger, fear, sadness, happiness, disgust, and surprise, but these basic emotions are infrequent in the learning context (Bosch et al. 2015b). Rather, students’ experiences during learning episodes centre around engaged concentration, boredom, confusion, frustration, happiness, and anxiety (DMello 2013). Although they are not exactly
the same, the affects listed by Kort et al. (2001) and DMello (2013) are fundamentally in agreement.
The universality of facial expressions and mappings with the basic emotions were largely established byEkman et al.(1980) andReisenzein et al.(2013). The same process is still under way for the learning centred affects (McDaniel et al. 2007). As such, there is very little work done detecting the students’ affects during learning episodes using vision and most systems primarily rely on contextual and other sensor information. Current systems are discussed in the next section.
3.5.2
Detection of Affect for Educational Purposes
Bosch et al.(2015b) present one of the first works detecting learning-centred affective states in the wild using vision. They perform supervised learning on data collected from 137 students in 8th and9th grade computer laboratories. The students each sat at a computer and played a physics based computer game while being recorded by webcams. The students were coded using the five affects shown in Table3.1 as well as being on-task or off-task. Being off-task involved watching other students, using a cellphone, or having unrelated conversation. They use the area under receiver operating characteristic curve (AUC) to measure accuracy and achieve impressive results considering the challenges. Their results are reported in Table3.1.
Table 3.1: Classification Accuracies inBosch et al.(2015b)
State AUC
Off task behaviour 0.816
Boredom 0.610
Confusion 0.649
Delight 0.867
Engagement 0.679
Frustration 0.631
5-way classification accuracy 0.655
Commercial software was used to extract the likelihood estimates for the presence of 19 action units (AUs), as well as head pose, orientation and position information. Poor lighting, extreme head pose or position, occlusions from hand-to-face gestures, and rapid movements resulted in 25% of data being discarded. Using clustering and Bayesian approaches, the ex- tracted features were used to classify the relevant affects. The work byBosch et al.(2015b) is the most similar to the work presented in this thesis and will be used a benchmark for results.
Problems identified primarily related to issues with noisy data, which was a problem for the work presented in this thesis as well. As data was collected using webcams on computers in front of the students it provided an unoccluded, consistent, frontal view of the subjects. The data presented in this thesis is more challenging as it considers students sitting in a lecture theatre viewed from a single camera at the front of the class. This means that there is significantly more variation due to occlusions, background changes, resolution, angle from the camera to the student, and others. Kai et al. (2015) expands upon Bosch et al.(2015b) and again notes
problems with video based approaches and noisy classroom environments. Again the students were playing the physics-based game and were recorded from cameras on the machines in front of them. The issues related to poor video data are discussed further in Chapter5.
Raca and Dillenbourg(2013);Raca et al.(2015) focus on the use of head motion to identify students that are paying attention. They show that head pose can be used to approximate eye gaze and, in turn, use that to analyse whether students are behaviourally engaged in the class. They also make use of optical flow to track how much the student is moving. Their work gives accuracies between 61.86% and 65.72% classifying students on a 3-point scale of attention. The data was collected in a manner similar to this work, but on a smaller scale. Attention was labelled with students self-reporting their attention levels periodically using a questionnaire during class.
A further literature survey was unable to find any work seeking to apply vision based affect detection methods to real-life ‘wild’ learning environments where subjects were not seated in front of a computer. It is believed that this is the first work to focus on issues of affect detection in the wild, without subjects seated in front of computers. However, there has been a lot of work in the creation of Intelligent Tutoring Systems (ITSs) in order to make an affect-aware online tutor. This was proposed byKort et al.(2001) and the research groups of D’Mello and Picard have contributed much literature towards the development of ITSs to maximise learning. A system that responds appropriately to negative affective states is more likely to re-engage the student (Kapoor et al. 2007). The work already done in this field is very promising (Alyuz et al.2016;Arroyo et al. 2007;Conati and Maclare 2004;D’mello and Graesser 2007;D’Mello et al.2007 2010;Graesser et al. 2007;Johns and Woolf 2006;Kapoor et al. 2007;McQuiggan and Lester 2006). AutoTutor has received the most attention recently. Systems like this use a combination of hardware monitors and contextual information from the current activity to predict the student’s current affect.
In these settings, the researchers have used various methods and sensors to detect emotions such as boredom, fatigue, joy, surprise, anger, disgust, and interest while the student works on problems with the ITS. It then adjusts both the difficulty of the problems as well as the phrases it uses to respond to the user. The ITS applications developed so far have been based primarily on vision for facial expressions. Other sensors, including a Body Pressure Measurement System (BPMS) (D’Mello and Graesser 2009) and skin conductance sensors (Arroyo et al. 2009) have been used. Almost all of the current systems use a pressure sensitive mouse which provides vast information about frustration (Dennerlein et al. 2003).
The ITS developed inArroyo et al.(2004 2007 2009) was equipped with mouse pressure, posture, video, and skin conductance sensors. It was then applied in both High School and Un- dergraduate settings. They used self-reported student data as ground truth. A major component of the system was the use of MindReader which performs facial expression recognition to infer aspects of a subject’s state of mind (El Kaliouby and Robinson 2005a). They found that the extra sensors improved the predictive power of the system. They also identified correlations be- tween Sitting Forward and both frustration and excitement. With contextual information from the online tutor, they are able to differentiate between the two. Confidence and concentration were strongly correlated and were well predicted by the facial expression software.
system to measure posture. The BPMS system is placed on the horizontal and vertical parts of a chair, it then feeds pressure information to the system that can be used to infer posture. Based on posture alone, machine learning experiments found detection accuracies of 73%, 72%, 70%, 83% and 74% when differentiating boredom, confusion, delight, flow and frustration from neutral affects. When differentiating between two, three, four and five of the states, accuracies of 71%, 55%, 46% and 40% were achieved respectively. Although facial expressions were once considered the gold standard for emotional expression, they argue that “there is converging evidence that disputes the adequacy of the face in expressing affect. . . at the very least, it is reasonable to operate on the assumption that some affective states are best conveyed through the face, while others are manifested through other non-verbal channels.” Of interest to this research, D’Mello and Graesser(2009) note the face is particularly sensitive to social display rules when dealing with negative emotions, such as frustration and boredom.
Still applicable to education, but with many other uses, MacHardy et al. (2012) and Ben- zaid and Dewan (2010) look to detect the affect of subjects in the audience of a distributed presentation. When live streaming a presentation over the Internet, presenters have no way to assess the interest levels of their audience. Some new online presentation tools allow audience members to indicate their level of interest, but this feature is both rare and can be distracting for the user. Their work looks to use built-in webcams in the computers of each audience member to perform facial expression recognition and report levels of interest back to the main server. Aggregate information could then be supplied to the presenter automatically to assist presenters in understanding their audience.
Their system was trained on self-reported data, where a user would watch a video while being recorded. They would then review the video afterwards and tag sections as bored or engaged. They were unable to detect frustration better than chance. The studies were too small to be of statistical significance, with only five subjects inBenzaid and Dewan(2010) and twenty inMacHardy et al.(2012). In spite of this, their results show promise, in that a simple SVM trained only on the pixel values of the mouth, eyes and nose was able to distinguish between the two affects.
More recently work has focused on the development of affect detection for distance learning systems (Gogia et al. 2016). Their goal was to detect and report affect of remote learners. A multi-modal approach was developed based on a EEG hardware worn by the participants as well as a facial tracker based on the Kinect camera. While the results are promising and show the power of a single user system, the hardware constraints are unsuitable for large scale roll- out. Poulsen et al.(2017) show that EEG sensors can be used to reliably recognise engagement and that subjects in classroom settings exhibit reliable neural responses that were correlated across both subjects and experiments. This shows that EEG sensors are a potentially feasible way to collect ground truth data in the future. The idea of providing EEG sensors to all students in a large class could be problematic and will likely introduce issues practical when scaling up even if students were prepared to wear the equipment for extended periods of time.
Paquette et al. (2016) use a mix of the Kinect, software logs and quantitative field obser- vations for subjects in front of a computer based learning game. Their field observations were made using the Baker-Rodrigo Observation Method Protocol (BROMP) (Ocumpaugh 2012) where a certified coder is used to provide labels. To be BROMP certified, the new coders must have a Cohen’sκ score of 0.6 when compared with other previously certified coders. Unfortu-
nately such coders were not available for this research. Paquette et al.(2016) found issues with subject posture and interaction-based (software interaction logs) detectors, but ultimately found that the interaction-based detectors perform better. Conversely,Bosch et al.(2015a) found that face detector based systems functioned better than the interaction-based systems, although the latter suffered less from missing data. They note the difficulty of feature engineering and how it is difficult to know a priori which features will prove effective.