ICALL systems in use

In this section, we describe the most relevant aspects of the three ICALL systems that are fully integrated into real-life foreign language learning programmes at the university level. First, we provide some basic information on each of the systems: language taught, approximate starting date according to research publications, the levels of proficiency for which they are conceived, and some general comments on their use. Then we compare how theses three systems were conceived, developed or tested with regard to a series of issues particularly relevant for ICALL systems: instruction context, pedagogical orientation, design principles, system architecture, language processing strategy, feedback generation and learner modelling.

Basic system information

ROBO-SENSEI (Nagata, 2004) was initially called Nihongo-CALI (Nagata, 1995), and over a period of time was also known as BANZAI (Nagata, 1997b). The work around this system began with Nagata’s thesis (1992), and several papers on its de- sign, its development and its use were published over the years – the most recent being Nagata (2010). The system was developed for teaching Japanese to English speakers at the university level. Although it was mainly used with beginner courses, the au- thor claims that its NLP analysis engine “can process all grammatical constructions introduced in a standard Japanese curriculum from beginning through advanced levels” (Nagata, 2002: p. 584). This system is currently commercialised through a publishing house – see http://www.cheng-tsui.com/store/products/robosensei.

ETutor (Heift, 2004) is the name of a system previously known as the German Tutor (Heift and Nicholson, 2001). ETutor is an ICALL system whose development started with Heift’s thesis (1998), and it is still further maintained and improved Heift (2010b). It was developed for the teaching of German to beginners at the university level in Canada (Simon Fraser University). It has been used there for a long time, and it is available for use at other universities for a semester-valid user fee (Nina Vyatkina, p. c.) – see also http://www.etutor.org.

TAGARELA, which stands for Teaching Aid for Grammatical Awareness, Recog- nition and Enhancement of Linguistic Abilities (Amaral, 2007: p. 50, Amaral and Meurers, 2006), is a system developed and piloted in 2006 and first used in regular Portuguese teaching courses in 2007. It has been used at The Ohio State University and is currently being used at the University of Massachusetts Amherst, and the au- thors are extending its contents and improving its technical functionalities (Amaral et al., 2011). The system is aimed at instructing learners of Portuguese in their first courses at the university level and specifically at students in Individualised Instruc- tion Language Programs in a North American context (Amaral, 2007: pp. 49–53) – see also http://purl.org/icall/tagarela.

Detailed feature comparison

Instruction context The three systems (and their predecessors) have been and are being used in combination with face-to-face instruction (Nagata, 1993, 1995, 1997b; Heift and Nicholson, 2001, Reeder, Heift et al. 2001) and as a complement to standard textbooks (Nagata, 2010; Heift, 2004, 2010b). ETutor is nowadays adapted to accompany the first three German courses at the university level when using the book Deutsch: na klar! (Heift, 2010b; Di Donato et al., 2004; Sanders, 2012), while ROBO-SENSEI is evolving into an independent textbook for Japanese (Nagata, 2010). As for TAGARELA, it was initially designed be used as an intelligent electronic workbook integrated in the Portuguese Indi- vidualized Instruction Program at The Ohio State University (Amaral, 2007: pp. 51–64). TAGARELA is also used in combination with face-to-face instruc- tion (Amaral, 2007: pp. 132–133), and is used in distance education courses at the University of Massachusetts Amherst.

As for curriculum embedding, the three systems are being used in commu- nicative instruction settings at the university level in combination with stan- dard instruction textbooks or other electronic materials – Nagata (2002: p. 583), Heift (2001a,b, 2010b), and Amaral (2007).

Pedagogical orientation The three systems explicitly follow SLA and FLT theo- ries where corrective feedback and focus on form are considered, and are always embedded in settings following a communicative approach (Nagata, 1993, 1995, 1998; Heift, 2001a, 2003; Amaral, 2007: pp. 46–48). Moreover, Nagata has in- vestigated the benefits of explicit inductive feedback – that is, feedback driven to help learners figure out the linguistic rules underlying certain structures or sentences – and language production practice (1997b), and her system is de- signed to produce this kind of feedback. As for TAGARELA, according to

Amaral it “was designed to help fill one common gap of language instruction at the university level in the United States: the lack of personalized feedback students receive on their language production because of the small amounts of time instructors can spend with each individual student” (2007: p. 50, but also the whole of Ch. 3). All three systems are consistent with the corresponding North American syllabi.

As for their pedagogical goal, both ROBO-SENSEI and ETutor aim at fos- tering the acquisition of grammar and vocabulary competences of beginner to intermediate levels of the second language. ROBO-SENSEI includes linguistic aspects such as sentence particle usage, verb inflection, auxiliary verbs, and passive voice – see Nagata (1995: p. 51), Nagata (2002: p. 598) and Nagata (2010: p. 461) –, and ETutor includes aspects such as noun phrase agreement, subject agreement, auxiliary verbs, and use of punctuation – see Heift (2003: pp. 542–545). TAGARELA, in addition to being aimed at the acquisition of certain grammar and vocabulary competences, also aimed at fostering listening and reading comprehension skills (Amaral, 2007: pp. 70–71). It covers spelling and grammar errors and provides information about the semantic appropriate- ness of student production on the basis of shallow content analysis (Amaral, 2007: pp. 90–91, Amaral and Meurers, 2008: pp. 321–322).

As for the activity types offered, all three systems offer exercises where relatively- free (pedagogically constrained) short answers are required. In ROBO-SENSEI the learner is required to write a sentence following specific instructions on what to write (Nagata, 1997b: p. 518) – since the instructions are provided in En- glish they can be considered translation exercises, as pointed out in Amaral and Meurers (2011: pp. 8–9). Its forthcoming version will include activities where listening and reading comprehension activities are found, as well as character writing activities (Nagata, 2010). ETutor included from the beginning dictation, build a phrase, which word is different, word order practice, fill-in-the-blank, and build-a-sentence activities (Heift, 2001b), and later on has incorporated reading comprehension, listening comprehension and short essays (Heift, 2010b).3 _As

for TAGARELA, it includes listening and reading comprehension, descriptions, vocabulary, rephrasing, and fill-in-the-blank (Amaral, 2007: pp. 64–79).

Design principles Underlying ROBO-SENSEI’s design choices there is the need for an interactive system that can provide an immediate response, that enhances the student-textbook interaction and favours self-paced learning, that provides specific and linguistically principled feedback to correct or incorrect input, and that encourages the development of production skills (Nagata, 2010: p. 461 and 463). Similar criteria are provided by Heift, where she mentions the emulation of “learner-teacher interaction” (Heift, 2010b: p. 445). The authors of the three systems argue for sophisticated answer-processing tools because it is not feasible to anticipate every possible mistake made by learners. Quite appealing in this sense is the procedure followed by Amaral (2007). To better define and learn about the language teaching context in which the system is to be integrated he

interviews teachers of Spanish and Portuguese as to their real-life needs (Ama- ral, 2007: pp. 53–64). In all cases, NLP limitations were taken into account and specifically tackled by appropriately restricting the language required through careful activity design (see Amaral and Meurers, 2011).

System architecture All systems follow similar processing architectures, but the most thoroughly discussed one is TAGARELA’s Amaral et al. (2011). It consists of six modules: the interface; the analysis manager, which allows to configure and select the appropriate NLP tools for each learning activity; the language processing module; the feedback manager; the learner model; and the instruc- tion model (Amaral, 2007: pp. 84–85). These modules or corresponding func- tionalities are present in the other two systems too (Heift and Nicholson, 2001; Heift, 2003; Nagata, 1997b, 2002). All systems use client/server architectures and include ad hoc learning management functionalities. The programming languages they use are JAVA, Prolog, LISP, cT, and Python.

Language processing strategy The language processing architecture in the three systems evolved – one should say evolves – over the years (Nagata, 1995, 1997b, 2002; Heift and Nicholson, 2001; Heift, 2003; Amaral, 2007; Ziai, 2009). They all present an architecture consisting of a processing pipeline including word segmentation, spell checking, lexicon look-up, part-of-speech tagging, par- tial syntactic parsing, specific grammatical or language use and error detection (e.g., agreement checker). TAGARELA includes simple semantic (or content) checking on the basis of shallow linguistic information (Amaral, 2007: p. 95– 110). ETutor and TAGARELA use a combination of mal-rule detection tech- niques and relaxation techniques, ROBO-SENSEI relies exclusively on mal-rule techniques often intermingled with the language analysis process. All three systems opt for keeping things simple whenever possible, so string matching techniques are also strategically used to reduce the response time of the system. TAGARELA was recently re-implemented using UIMA4 _{as an underlying soft-}

ware platform, which results in a more flexible, modular architecture that allows for the specification of the processing modules on the basis of the required NLP tasks (Ziai, 2009: pp. 16–18). UIMA is based on a data structure that allows for the flexible combination of analysis features which can become as complex as required – not only linguistic features determined by the NLP, but also any other type of information that can be determined by the learner or instruction models.

As for linguistic knowledge representation, all systems provide theoreti- cally informed linguistic structures, but only ETutor uses a sophisticated formal- ism for the representation of linguistic information. Heift and Nicholson (Heift and Nicholson, 2001; Heift, 2003) use Head-driven Phrase Structure Grammar (HPSG), where linguistic information is formally represented as feature struc- tures which encode partial descriptions of a linguistic sign following a lexicalist approach (Pollard and Sag, 1994). Heift’s approach to the analysis of ill-formed

input is to relax the constraints imposed by features such as gender, number or case. Ill-formed constructions are allowed, but marked as incorrect. This way, a combination of incompatible linguistic features is allowed and recorded in so-called descriptors, whose information is percolated to their respective heads. This information can later on be used to provide information about both cor- rect and incorrect learner production. TAGARELA’s newer version uses UIMA types, which “are equivalent to typed feature structures used in formalisms such as HPSG” (Ziai, 2009: p. 44).

As for computing algorithms, all systems use standard finite-state automata techniques for certain modules (word segmentator, disambiguator, agreement checker, content checker), and some version of the Damerau-Levenshtein algo- rithm for spell checking, except for Japanese, where a mal-rule approach to certain spell checking error types is used. As for the syntactic tree-building algorithms, ROBO-SENSEI uses a Generalised Left-to-right Rightmost (GLR) parser (Nagata, 2009: p. 566), and TAGARELA’s recent version uses an imple- mentation of the Cocke-Younger-Kasami (CYK) algorithm (Ziai, 2009: pp. 49– 58). ETutor uses genetic algorithms for the detection of incorrect word orders (Heift and Nicholson, 2001).

Feedback generation In the three systems feedback generation is a two-step pro- cess: (i) error diagnosis and (ii) the actual presentation of feedback messages to the learner (henceforth feedback presentation). In the first versions of ROBO- SENSEI, error diagnosis was encoded in the parser’s rules following a mal-rule approach (Nagata, 1995). As described in Nagata (1997b), the system checks on a surface level the learner’s answer and, after an initial filtering based on heuristics, it selects and analyses the answer stored in the system with a higher proximity to a target answer encoded in the system as part of the activity. This target answer is used to compute all the error diagnosis operations. In the lat- est version of the processing architecture, error diagnosis modules are separated from analysis modules but interleaved in the processing sequence (Nagata, 2002: pp. 590–592). ETutor performs error diagnosis during the parsing itself, by us- ing relaxation techniques. Since the parser incorporates the ability to handle ill-formed input, no added procedure is needed for the detection of sentence- level errors. Errors related to extra or missing words, or wrong word choices are handled using other mechanisms ranging from string-based pattern matching to more complex rule-based or statistics-based algorithms (Heift and Nichol- son, 2001). The error diagnosis amounts to interpreting the analysis provided by the NLP tools in the context defined by the activity model and the learner model. TAGARELA follows a strategy in which each module contributes to the analysis and evaluation of the learner response with its specific functionali- ties: the resulting analysis is a combination of the different analyses, sometimes overlapping, provided by the different modules (Ziai, 2009: p. 39).

As for feedback presentation, ETutor and TAGARELA follow a similar strat- egy. A specific and externalised feedback module collects all errors detected by the diagnosis module and orders them according to specific criteria, which can

be configured – e.g., errors related to the specific goal of a learning activity are prioritized (Heift, 2003: pp. 543–544, Ziai, 2009: pp. 40–41). These two sys- tems present only one feedback message at a time. ETutor is in this respect a pioneering system in that its authors worked out a strategy to adapt the feed- back to the learner’s level; depending on learner performance, different feedback messages reflect different degrees of explicitness. In contrast ROBO-SENSEI’s feedback results from a mapping of all the collected error codes to feedback messages. Feedback messages are grouped into categories of an internally de- fined typology, which includes classes such as missing word, particle error or predicate error (Nagata, 2002: p. 592).

Learner modelling ETutor (Heift and Nicholson, 2001; Heift, 2003) and TAGA- RELA (Ziai, 2009: pp. 34–35) include learner modelling capabilities. Both sys- tems collect and maintain information about learners profiles and behavior. In Heift’s words this allows for “modulation of instructional feedback” and “as- sessment and remediation” (2003: p. 541). In both cases the modelling is based on a network structure whose nodes correspond to grammar skills (grammar phenomena handled by the parser) for which the student is internally penal- ized or rewarded (node scores). This allows for a classification of students into beginner, intermediate and advanced learners. Every time a learner receives feedback on a specific grammar skill, this feedback is made more or less explicit according to his or her recorded performance.