Critical reflection

There were various challenges faced in undertaking this research, the discussion below outlines some of the factors that influenced the options and the decisions taken.

 The MRCHIPS agent was devised to provide mentoring support for novice programmers within the framework of the cognitive apprenticeship pedagogy. Cognitive apprenticeship has a number of features that made it an attractive choice for use in this research. First the pedagogy correlates to the practice used in mentoring, most notably the coaching and scaffolding methods. The exploration method would also be provided by the availability of a mentoring resource to support the learner when experimenting with the programming language. Second the pedagogy provides a structured framework with separate methods, where the aims and outcomes of each method may be considered in isolation and easily measured if required. Third, the methods of the pedagogy may be implemented in different ways, such as by exercise, reading material, discussion, etc. This flexibility allows the possible use of a technological solution where the details of activity may be different, but aims and outcomes are used to determine how the activity contributes.

 The development of the architecture went through many iterations of design, mainly due to attempts to integrate a CBR engine based on the MOPS data structure (Riesbeck & Schank 1989) with the BDI engine based on the Prolog Horn clause. During the development of the agent no method could be found for integrating the Horn clause with the MOPS data structure that would not cause a loss of data or become time consuming when converting of data was to be passed back and forth between

184

subsystems. Once it was decided to base the cases on the same Horn clause data structure and use a discrimination network to control storage and retrieval the development progressed quickly. Using a single knowledge representation scheme the different reasoning subsystems simplified communication. Concepts that mean the same thing have the same representation in the knowledge base even though they are processed in different ways by different subsystems. The single representation also allows for some agent resources to be shared such as the Prolog language parser, which is used by all subsystems to read the agent knowledge base and the unifier used for matching data.

 Although the MRCHIPS was designed to provide mentoring in a desktop environment the architecture was designed to follow the principles of a cognitive architecture. The reason for this was to allow for the likely range of reasoning requirements within the desktop environment. The MRCHIPS architecture satisfies nearly all of the commitments for a cognitive architecture as described by Langley (2006) and discussed in chapter five; the commitment to long-term memories is currently underdeveloped; it would be addressed by the ability to retain new cases in the CBR or the inclusion of an autobiographical memory similar to that used in agents like FatiMA (Aylett et al. 2007). It is likely that other cognitive agent architectures such as Soar (Laird et al. 1987), Icarus (Langley et al. 1991), or ACT-R (Anderson 1993) would also be suitable frameworks on which to build MRCHIPS. The decision was taken to build MRCHIPS in Python for two reasons. First to gain an insight into how to implement a cognitive architecture. Secondly in addition to its suitability for teaching the properties of Python make it an attractive choice for prototype application development as would be required for this research. In addition implementing the agent in the same language as would be used by the learner would simplify its installation process. As MRCHIPS is simply a Python application all the resources required for its execution would be available once Python was installed. It was

185

imagined that student volunteers would install MRCHIPS on their own computers without supervision so the installation process was made as simple as possible.

 Although the Microsoft agent character interface is integral to the way MRCHIPS operates no experimentation was attempted on changes to the interface. Work had been carried out to provide a dialog text box to handle inputs to the agent. The Microsoft agent engine only allows speech input and as the presence or quality of a speech input engine was unknown for the computer on which students might use MRCHIPS a dialog box was added. Consideration was given to assessing the effect of the degree of embodiment and animation on learning but this was not pursued as research elsewhere had been carried out to investigate this (Lusk & Atkinson 2007). It is also worth noting that Microsoft has withdrawn support for MS-agents on operating system versions after Windows XP; an open-source alternative application called Double Agent from Cinnamon Software Inc. is free to download from the Internet, it is designed to be fully compatible with MS- agents and available for more recent versions of Windows but at this time no evaluation has carried out to its use with MRCHIPS. A significant effort had been made to supply MRCHIPS with a natural language parser but no solution could be developed that supported a large enough vocabulary, that could process statements rapidly enough, and would remain stable enough to be used for the experimentation. What had not been anticipated was how important the text-to-speech feature was to engagement with the agent, with learners commenting that they preferred to have the agent speak to them as they read the text of the help message from MRCHIPS.

 The amount of experimentation with the agent was only enough to establish that MRCHIPS had a positive effect on the outcome for learners in a task requiring coaching support. There were also good indications of scaffolding, as some learners did not use

186

MRCHIPS once they had recognised the reoccurrence of errors and applied a remembered correction. Due to a change of employment there was no opportunity to test whether MRCHIPS had an effect on learner exploration. Ideally a larger evaluation would be carried out taking place over several months, involving numbers of students comparable to the cohort size and including a similar sized randomly selected control group with access to similar resources working to a similar lesson plan, but in the absence of the agent. At the end of the trial students of both groups would be tested on what they had learned. Given that both sets of students had access to similar resources any difference in the outcome of their results could be then attributed to the presence of the agent. However, even under ideal experimental conditions other factors would still be present that would influence or cause to question the outcome. For instance as people partake in any process their experience tends to grow. It would not be unreasonable to expect learners to become more proficient programmers with or without an agent assistant leading to the conclusion that there is no significant measurable difference after a sufficient period of time. Therefore, in order to demonstrate the usefulness of MRCHIPS, it was necessary to show that novice programmers’ were able to make more progress in practical exercises as a result of the agent than they would without it. However a larger evaluation of the agent is still required.