9.2.1
Model of non-verbal behaviour
The model of behaviour is based on a survey of features used in similar applications (section 6.2). Table 9.1 details the behavioural channels under observation.
Table 9.1 Behavioural data model
Type Channels Examples
Learner 2 Gender, Ethnicity
Eyes 17 Eye openness, Gaze direction, Blink Geometries 18 Position, Rotation, Movement Physiological 2 Blush, Blanche
9.3
Classifier training and testing tool
A comprehension classifier has been developed based on an artificial neural network (Chapter 5). An artificial neural network has been chosen as the classifier due to the successes reported in literature for similar applications (discussed in section 6.5).
The artificial neural network (ANN) will be trained on data produced by the indexer. The topology of the artificial neural network has been designed to match the inputs generated by the indexer, with 39 input nodes to accept the 39 variables in each behaviour vector. The ANN has a single output node with a threshold output function to produce a classification of either +1.0 for comprehension or -1.0 for non-comprehension.
The network is trained using back-propagation of errors (section 5.4.1), and tested using 10-fold cross validation Haykin (1994).
While the input layer and output layer of the network are defined by the shape of the data and the desired classification output, the precise topology of the network’s hidden layers are a parameter for the experimentation discussed in Chapter 10.
166 Image pre-processing and comprehension classifier training
The classifier training tool (algorithm 7), a command-line application, has been developed to facilitate rapid evaluation of a range of topological parameters. The command-line trainer accepts arguments to define the structure of the network and instantiates, trains and tests a network of the specified topology.
Algorithm 7:Pseudo-code algorithm for training artificial neural network 1 Load CSV as results;
2 Split resultsinto 10 each sized partitions as f olds;
3 for each f old in f olds do
4 Set T e =f old;
5 Set V a = next f old;
6 Set T r = all f olds except T eand V a;
7 Create new MLP neural network asnetwork; 8 Train network on T r;
9 Evaluatenetwork onV a;
10 Testnetwork onT e;
11 Savenetwork and performance statistics to log file;
12 end
The trainer allows the network configuration to be determined empirically by testing each combination of parameters in Table 10.1.
9.4
Conclusion
This chapter has presented two tools designed and developed to facilitate a pilot study of computational analysis of non-verbal behaviour and classification of comprehension from image data using artificial neural network. The tools pre-process the image data into descriptive vectors of non-verbal behaviour for a given time window and then use the pre-processed data to train and test a comprehension classifier based on artificial neural networks. In Chapter 10 the command-line tools presented in this chapter will be used to evaluate the parameters for data pre-processing, window duration and interval, and for network topology.
Chapter 10
Study: Exploring the
relationship between reading
comprehension and learner
non-verbal behaviour
10.1
Introduction
This chapter presents an initial exploratory study intended to identify, from data, the feasibility, performance, requirements and constraints of modelling learner comprehension from non-verbal behaviour, using an artificial neural network classifier. An artificial neural network has been chosen for initial exper- imentation as the classifier has performed well on related problems discussed in section 6.5.
Using a dataset of comprehension labelled web camera images gathered during the pilot of Hendrix 1.0 CITS (Chapter 8), the effectiveness of a neural networks based approach identified in literature (Chapter 3.4) is empirically evaluated in terms of classification accuracy.
168
Study: Exploring the relationship between reading comprehension and learner non-verbal behaviour This exploratory study aims to answer several high-level questions which will provide guidance on experimental design, data treatment and technical implementation of a real-time comprehension classification system (section 10.2).
Section 10.2 details the research questions the study aims to answer and section 10.3 explains the motivation for conducting this study. Section 10.4 outlines the study procedure. The method is detailed in section 10.5 and results in section 10.6. This aim to identify the optimum parameters for data pre-processing and classifier topology. Conclusions are shared in section 10.6.
10.2
Research questions
The research questions explored in this experiment are necessary to define the treatment of image data in extracting and measuring non-verbal behaviour over time and in selecting the optimum topology for an artificial neural network comprehension classifier.
1. When and for how long does comprehension indicative behaviour occur? 2. Can an MLP achieve binary comprehension classification at above chance
levels? (>50%)
3. What set of classifier hyper parameters produce the highest accuracy?
10.3
Motivation
A core pedagogic device in cognitivist educational practice is scaffolding (Chap- ter 2.2). Scaffolding involves the introduction and fading of support for learners during problem-solving. Literature suggests that successful scaffolding should allow the learner to first attempt problem-solving without explicit intervention or instruction, promoting the demonstration of competency. If a learner expe- riences difficulty, the tutor should then introduce increasing levels of support
10.4 Study procedure 169
to help develop task and subject understanding. When competency is again demonstrated, the explicit support and instruction should be withdrawn, or
faded, allowing the learner to take control of problem-solving processes again.
Intelligent tutoring systems aim to mimic the flexibility of human tuition by adapting to a variety of real-time learner feedback channels. However, a generalised, affordable, practical and non-intrusive method of identifying e- learner comprehension automatically in near real-time is absent, as established in Chapter 3.
The Hendrix 1.0 CITS discussed in Chapters 7 and 8 is able to introduce and fade support in response to answers a learner provides to direct questions. However, observation of tutor practice in the classroom (Chapter 2 section 3.3) suggests that human tutors enact timely interventions, to scaffold the learning experience, by estimating accurately the learner’s degree of comprehension from non-verbal behaviour.
To develop the ability of the Hendrix CITS to perform human-like cogni- tive apprenticeship, the platform must be equipped with the functionality to model and classify e-learner comprehension in real-time from coarse grained analysis of non-verbal behaviour. Accurate real-time classification of e-learner comprehension would provide a feedback channel for intelligent adaptation of materials, user interface elements and discourse, without depending solely on incorrect answers as a trigger for the introduction of scaffold.
Literature (section 3.4) has suggested that a neural networks based approach (Rothwell et al., 2006, 2007) of behavioural summary and classification can be used to classify learner comprehension in dyadic verbal interactions under interview conditions (Buckingham et al., 2012, 2014).
10.4
Study procedure
15 undergraduate, masters and PhD students at Manchester Metropolitan University undertook the experiment. The participant size was based on
170
Study: Exploring the relationship between reading comprehension and learner non-verbal behaviour participant numbers for similar pilot studies of CITS identified in literature (Cai et al., 2011; Serón and Bobed, 2016).
To assess the effectiveness of the method discussed in literature (Buckingham et al., 2012, 2014; Rothwell et al., 2006, 2007), the author has attempted to recreate the data treatment and classification process. A new dataset of labelled images was collected during the pilot study of Hendrix 1.0 CITS (Chapter 8). Each answer period during the on-screen tutorial was recorded using a front-facing web camera attached to the PC. Each recording was stored as a sequential set of images, along with the score assigned for the answer provided. Answer scores are used as a ground-truth measure of comprehension.
Answers were marked automatically by the Hendrix 1.0 CITS during the pilot study discussed in Chapter 8. Correct answers are labelled as ‘comprehension’, while incorrect answers are labelled as ‘non-comprehension’ based on the tutorial log files.
This pilot study consisted of the following steps:
1. Participants took a seat at a computer in a computer laboratory at Manchester Metropolitan University
2. Participants read and signed a consent form for the experiment and reviewed a set of instructions detailing their tasks for the experiment 3. Participants started the Hendrix 1.0 CITS application by double clicking
the executable on the desktop
4. Participants were instructed to take a tutorial on ‘For loops’ using the Hendrix 1.0 CITS
5. Participants were recorded using a front-facing web camera while answer- ing each question
10.4 Study procedure 171
7. Participants were recorded, subject to ethics, during the tutorial using a front-facing web camera
8. Participant non-verbal behaviour was analysed for each question
9. A bank of artificial neural networks were trained to classify comprehension from observed e-learner non-verbal behaviour using an array of hyper- parameters
10.4.1
Ethics
Under ethical approval from Manchester Metropolitan University study par- ticipants consented to undertake an on-screen tutorial with a conversational intelligent tutoring system called Hendrix 1.0. During the tutorial participants’ dialogue with Hendrix 1.0 was recorded. In addition, image data captured from a front-facing web camera was recorded and saved to encrypted portable media.
As data collected during the experiment was not anonymous and contained personally identifying information including names, email addresses and de- mographic information, the data collected was transferred to and stored on a physically secured computer with an encrypted hard drive. In accordance with ethical approval and the terms of consent for the experiment, all information used for redistribution or publication purposes was anonymised or aggregated so as not to be personally identifying.
10.4.2
Participant information
The participant group consisted of 15 student volunteers from the School of Computing, Mathematics and Digital technology at Manchester Metropolitan University. The participant group consisted of 11 male and 4 female participants, with 7 of participants studying at under-graduate level and 8 studying at post- graduate level.
172
Study: Exploring the relationship between reading comprehension and learner non-verbal behaviour
10.5
Method
In this experiment a grid search is performed to evaluate the optimum set of parameters for both data pre-processing and artificial neural network topology.
Data pre-processing parameters are temporal, determining the period of analysis, the way in which the total answer period is segmented and summarised and the intervals between summaries (Table 10.1). ANN topology is tested with 10, 15, 20 and 25 hidden layer nodes.
In a grid search, each combination of parameters is trained and tested. The full search grid of 120 parameter combinations is detailed in Table B.1. The exhaustive search aims to identify the set of parameters which produce the highest possible classification accuracy.
Image data and ground truth comprehension scores were collected during the pilot study of Hendrix CITS (Chapter 8). Image sets have been pre-processed using the behavioural indexing process detailed in section 9.2. Varying the temporal parameters of pre-processing has produced 30 data sets for evaluation. The parameter combinations are shown in Table 10.1.
Each row in Table 10.1 shows the parameters for indexing of the non-verbal behaviour. Each row represents a separate data set of labelled non-verbal behaviour, generated using the temporal variables detailed in Table 10.1.
For each data set in Table 10.1, four configurations of network topology have been tested - 10, 15, 20 and 25 hidden neurons within a single hidden layer. Testing classification accuracy with each configuration of the artificial neural network will identify the best network topology to use for classification of behavioural data.
The full results table is shown in Appendix B. Please refer to the parameter Table 10.2 for descriptions of the column headings in the results table.