Student

The Hendrix 1.0 student model keeps track of a learner’s conversation, tutorial position and progress. The student model is used to store attributes of the stu- dent including name, gender and ethnicity of the learner, tutorial conversation dialogue, tutorial progress and answer scores.

Hendrix 1.0 has been designed to gather visual behavioural data during tutorials. Hendrix 1.0 uses a web camera (figure 7.2c) to record learners during each question-answer interaction. As shown in algorithm 5, Hendrix 1.0 saves the image data for each answer period on disk, in a folder structure mapped to the question-answer. As a step towards the overall research objective of developing a comprehension-responsive CITS, during the pilot study of Hendrix 1.0 image data and answer scores will be collected to form a pilot data set for analysis of learner non-verbal behaviour during on-screen learning.

142 Hendrix 1.0: A conversational intelligent tutoring system for programming Algorithm 5: Pseudo-code algorithm for saving sequential image sets

against learner answer response periods

1 createDictionary asD for holding bitmaps for questions; 2 while web camera is on do

3 if current event is Question then

4 if D contains key current_question[id] then 5 get bitmap from webcamera stream as b;

6 add b toL where L[key] equals current_question[id]; 7 else

8 fetch list of folder names from image storage location as dir; 9 if not all composite keys session_id + L[key] in dir list then 10 create folder in image storage location as session_id +

L[key];

11 for each bitmap b in L[key] values do

12 save b in session_id + L[key] as index of b inL[key]

values;

13 end

14 end

15 add new tuple (current_question[id], bitmap container)

16 end

17 end 18 end

7.5

Conclusion

This chapter has presented an overview of the Hendrix 1.0 CITS (section 7.3), detailing the novelties, challenges and requirements of the system. Architecture has been presented (section 7.4) and domain knowledge structures detailed (section 7.4.2). Key functions of the software have been highlighted, discussed

and presented in pseudo-code algorithms (sections 19, 7.4.2 and 7.4.3).

The contribution of research engineering work discussed in this chapter, is Hendrix 1.0’s ability to structure tutorial plans dynamically using shortest path queries over a directional graph of concepts, reducing the overhead of pre-planning and encoding specific tutorial progressions in static scripting files, and its ability to parse discursive, mathematical and programmatic content in conversations.

The contribution of research discussed in this chapter is to implement the micro-adaptive behaviours discussed in section 4.8.2, producing a system with

7.5 Conclusion 143

more micro-adaptive behaviours than previous CITS discussed in literature VanLehn et al. (2017).

Hendrix 1.0 micro-adaptations include hints and feedback on solutions, hints and feedback on approach to problem-solving, decomposition of tasks and answering students’ questions.

The system will be used to evaluate empirically whether a CITS can be used to tutor computer programming effectively using natural language interactions and whether learners can develop knowledge through directed tuition. The study procedure, method and results are detailed in Chapter 8.

Chapter 8

Study: Evaluation of Hendrix

1.0 CITS

8.1

Introduction

This chapter presents a pilot study carried out to evaluate the conversational and educational ability of Hendrix 1.0 in a real-world learning environment, a university computer lab, with enrolled undergraduate and post-graduate students.

In the pilot study 15 students from Manchester Metropolitan University undertook a tutorial on Java programming with Hendrix 1.0. Hendrix 1.0 is evaluated using a combination of statistical performance measures, accuracy and error rates and user feedback on coherence of discourse, perception of benefit and usability. In the study, Hendrix 1.0 classified correctly the utterance type of 91% of input sentences, marked 94.5% of question answers correctly and was rated 4 out of 5 for user satisfaction.

Section 8.2 presents the research questions for the study. An overview of the study is presented in section 8.5, with method and results for each research question detailed in sections 8.7.1 and 8.8. A discussion of the pilot study results is presented in section 8.9 and conclusions are detailed in section 8.10.

146 Study: Evaluation of Hendrix 1.0 CITS

8.2

Research questions

The study presented in this section evaluates the effectiveness of Hendrix 1.0 CITS in terms of conversational functionality and ability to teach.

1. Does Hendrix 1.0 converse effectively? 2. Does Hendrix 1.0 facilitate learning?

Figures 8.1 and 8.2 (see sections 8.7.1 and 8.8.1) define the GQM (see section 4.9) for evaluation of Hendrix 1.0, integrating both objective and subjective measures.

8.3

Contribution

The contribution of this research is to evaluate whether an intelligent CITS designed to deliver syllabus material through discourse using adaptive and micro- adaptive behaviours (see literature in section 4.8) is successful in conversing to facilitate learning via computer mediated interaction.

The research in this chapter explores not only whether the technology is reliable and accurate in conversation, but also how learners feel about interacting with and learning from the virtual agent. The outcome of this experiment, performance metrics and user feedback, will inform the development of a second prototype system.

8.4

Tutorial content

The tutorial focused on concepts and applied skills relating to a basic iterative ‘for’ loop. The ‘For’ loop was chosen as a tutorial topic based on the first

year computing syllabus, while tutorial dialogue (hints, feedback and tone) were derived by observation of first semester classroom tutorials at Manchester Metropolitan University in which tutors and students iterated a topic through

8.5 Study overview 147

question and answer group discussion. The tutorial content for the pilot study is detailed in full in Appendix A.

8.5

Study overview

In this study 15 students from Manchester Metropolitan University volunteered to complete a tutorial on the construction and application of ‘For’ loops. The experiment consisted of four steps:–

1. Participants completed a 10 question MCQ on Java programming; 2. Participants were instructed to take a tutorial on ‘For’ loops using the

Hendrix 1.0 CITS, during which metrics for evaluation (figure 8.2) of Hendrix 1.0 were recorded in log files;

3. Participants repeated the 10 question MCQ on Java programming from

step 1;

4. Participants completed a user experience survey to rate Hendrix 1.0 performance and provide feedback on their experience of using the system.

8.5.1

Ethics

Participation was voluntary and not compensated for participation. Participants were required to sign a consent form prior to participating in the experience. The consent form detailed the data collected during the experiment, how that data would be analysed and the intention to distribute or publish data from the experiment. Data collected during the experiment was not anonymous and as such personally identifiable information such as names, email addresses and demographic information has been stored on a physically secured computer with data fully encrypted. All information used for redistribution or publication purposes is anonymised or aggregated so as not to be disclosive.

148 Study: Evaluation of Hendrix 1.0 CITS

8.6

Participant information

The participant group consisted of 15 adult student volunteers from the School of Computing, Mathematics and Digital technology at Manchester Metropolitan University. The participant group consisted of 11 male and 4 female participants, with 7 of participants studying at undergraduate level and 8 studying at postgraduate level.

8.7

Does the intelligent conversational agent