With this goal to obtain pedagogically meaningful and computationally tractable tasks in mind, we propose a cyclic span of life for ICALL materials to be used in settings combining face-to-face instruction with computer-based instruction. This cycle of life of the ICALL task is presented in Figure 5.2.
Figure 5.2 reflects the processes and interrelationships between Activity, Re- sponse, Linguistic Analysis module, and Feedback Generation module during the
design and the execution phase. The evaluation phase, at the bottom part, is the last phase of this iterative process: from design to execution, from execution to evaluation, and then back again to design.
Figure 5.2: The ICALL task life cycle.
In the design phase, we distinguish an activity (Act), a set of expected responses (ExR), and the automatic assessment (AA) consisting of the linguistic analysis mod- ule (LA), and feedback generation (FG). In the execution phase, instead of expected responses we have elicited responses (ElR), that is, actual responses.
Figure 5.2 shows the teacher (T) is on top of the design phase. The teacher is responsible for conceiving and producing the learning materials, and in doing so has the target learner in mind (Lt). In contrast, the learner (L) is on top of the execution phase; the learner performs the activity, and the teacher is in this phase in her/his monitoring role (Tm), which can be performed with the assistance of a virtual tutor, and which is in fact the case in an ICALL setting.
Our approach to encompassing pedagogical needs and NLP limitations is to de- scribe and analyse the different interrelationships that emerge between activity, re- sponse, NLP-based analysis and feedback generation. Of course, the interrelation- ships that we refer to are influenced by the teaching and learning processes, as well as by the participants in the development and execution of the learning activity. However, this latter type of influence falls out of the scope of this thesis, since these are research areas corresponding to more behavioural or cognitive studies.
In the following sections we explore the interaction flow and the relationships between the elements identified in Figure 5.2. We start with the execution phase, because of its central place in the teaching/learning experience. Then we go on with the design and the evaluation phase.
5.2.1
Interaction flow in the execution phase
In the interaction flow during the execution phase several relationships emerge, and they follow a concrete chronological order: there is an activity that is responded to by a learner; the response is analysed with tools for the automatic analysis of language; and, eventually, the automatically analysed learner responses are used by the feedback generation module to provide the learner with assessment. This is a one-way step-by-step process that the learner can start up again after interpreting the system’s feedback, as shown in Figure 5.2 by the arrow labelled “learning experience”. Note that the “only” varying element is the learner response. The activity, the linguistic analysis module and the feedback generation strategy do not change over time in the execution phase, in spite of the fact that assessment might be different according to changes in the learner response – unless there is a learner modelling mod- ule (see next paragraph). Different learner responses generate different automatic analyses: The feedback generation strategy is dynamic within a range of limited possible forms of behaviour. Automatic assessment is, should be, systematic, that is, feedback messages are repetitive and repeated over a range of learner responses.
The right-hand side representation of the ICALL task’s life cycle in Figure 5.2 reflects the existence of a learner modelling (LM) module during the execution phase. Its outgoing and incoming arrows reflect the possibility for the learner model to influ- ence the selection of the FL learning activities chosen for the learner; or the selection of feedback types and strategies to be followed. In this context, the linguistic analysis module provides information to the learner model about linguistic phenomena that can inform about the learner’s progress. Nevertheless, learner modelling falls out of the scope of this thesis, indicated by the dotted-lines in its box in Figure 5.2.
5.2.2
Interaction flow in the design phase
As shown on left-hand side of Figure 5.2, the starting element in the design phase is the activity, produced by the teacher – or a content creator. During this phase, the four elements of the ICALL activity maintain their one-way relationships described in the execution phase, but other non-linear relationships emerge.
The first relation in the “straightforward” information flow emerges between the activity conceived and a range of expected responses. The teacher, with a target learner in mind, identifies a set of pedagogical needs related to specific communica- tive and linguistic skills. The activity results in a focused task (Ellis, 2003: p. 16–17), in which the wording of the instructions and the means given or pointed to to learners are oriented to the practising of the targeted skills. These skills are related to par- ticular linguistic structures to be elicited on the learner side, which might allow for the specification of a range of expected responses – in other words an NLP domain. We identify a second relationship between the expected responses and the linguis- tic analysis module, one that connects the contents and the language of the expected responses with the linguistic analysis module. The language analysis module requires a fine-grained specification of the lexical elements that need to be in the response, as well as the corresponding linguistic relations: fonetic, morphosyntactic, semantic or pragmatic.
This relationship between the expected responses and the linguistic analysis mod- ule is influenced by pedagogical considerations such as the pedagogical goals, the in- structions, or the input data. Thus, if an activity targets at training learners on the use of a particular syntactic structure, an automatic analysis module that provides the appropriate syntactic analysis is required. If the activity targets at practising the use of writing abilities such as expressing interest for a job, then pragmatics, the functional contents in pedagogical terms, have to be correspondingly modelled.
The third straightforward relationship is the one between the linguistic analysis module and the feedback generation module. Whatever it has to be said by the virtual tutor it has to be based on evidence found in the linguistic material that is present or absent in the learner’s response. Thus, the capabilities of the software for the automatic analysis of language play a key role in the appropriate detection of the expected language and contents in the elicited learner responses.
As for the non-straightforward interrelationships, we identify two of them. On the one side, the pedagogical goals of the activity might affect the feedback generation strategy; on the other side, the capabilities of the language processing tools, known by the NLP developer, might affect the pedagogical design.
The limitations of the NLP tools determine the kinds of ICALL activities that can be successfully implemented. To put a simple example, a semantic analysis module that works only at the sentence level will not be enough to assess texts containing more than one sentence. In such a case, the NLP tools might be enhanced to work beyond the sentence level, or one might decide to rethink the pedagogical concept.
The second non-straightforward relationship emerges between the pedagogical goals and the feedback generation strategy. If the activity focuses on form or on meaning, or on specific aspects of form, different kinds of linguistic or communicative issues will be prioritised as part of the feedback. Similarly, depending on the purpose of the assessment (low, medium or high-stakes), the feedback generation process will require different levels of linguistic information and different types of post-processing of the linguistic information. Last but not least, the desired nature of feedback has an impact too. To provide both positive and negative feedback, a module for the automatic analysis of learner responses requires analysing both correct and incorrect elements in the response.
5.2.3
Interrelationships in the evaluation phase
In the evaluation phase all the interrelationships considered in the two previous phases have to be re-visited. At this stage, the activity has been performed by learn- ers, assessed by the virtual tutor, monitored by the teacher, and can be evaluated in terms of success. The goal of the evaluation is to validate the activity as one that helps teachers and learners accomplish their respective goals. By comparing the re- sults obtained to the objectives initially defined, a set of recommendations regarding changes and improvements can be made.
From our perspective, there are mainly three aspects that need to be looked at in the evaluation of an ICALL activity: (i) whether the activity is pushing the learner to practice the targeted communicative and linguistic skills; (ii) whether the learner is capable of improving its outcome with the help of the automatically generated
feedback; and (iii) whether the performance of the feedback generation module is undermined by the performance of the NLP tools. The first one is not specific of an ICALL setting, but the second and the third are.
If learner outcomes correspond with those that favour the acquisition of the tar- geted skills, the activity is accomplishing its goal. If not, there are at least two important questions to be considered. First, whether there is a flaw in the design or in the execution of the activity that prevents the learner from achieving the ex- pected pedagogical goals. Second, whether unexpected or incoherent behaviours in the NLP-based correction functionalities can be attributed to the incapability of the NLP system to adjust to linguistic structures different from the expected.
If the learner is incapable of improving her/his learning outcomes, then the feed- back is not achieving its goals.1 The causes might be on the learner side, because
s/he might not be paying attention to it, or not noticing it. Or on the virtual tu- tor side (the ICALL design team), because the feedback strategy chosen does not correspond to or is not compatible with the learner’s style, background or level.
Finally, if the performance of the feedback generation strategy is flawed by the performance of the NLP tools, then critical inconsistencies or misleading feedback messages might show up. For instance, the feedback generation strategy will not be reliable in the identification of the use of the definite/indefinite determiner if either the analysis or error detection modules for that linguistic phenomenon do not perform with the required precision and recall.
The comparison between expected performance and actual performance will in- form of the changes and improvements to be made in the ICALL materials from its pedagogical conception to its use “in class”, through the computational implemen- tation of the NLP resources, its graphical presentation, and so on.