7.2 RIF: the Response Interpretation Framework
7.2.2 Applying the RIF to four FL learning tasks
7.2.2.4 Task type IV
The task that exemplifies Task type IV corresponds to Activity 5 of Subtask 2 in Atenci´on al cliente, the Spanish version of Customer Service and International Communication in ALLES. The task’s title is Expresa tu satisfacci´on o insatisfacci´on con el producto Smint.3
The TAF-characterisation of the task is:
Description Writing task that requires the learner to put herself in the shoes of a Smint consumer, a candy; the consumer sends a letter to the manufacturer to express her opinion on the candy and to ask for further information.
Focus Meaning and form. Outcome A consumer letter.
Processes Expressing positive and negative aspects of consumer products, particularly candies; asking for information related to consumer prod- ucts.
Input The learner is provided with a prompt describing the fictive setting and is given some hints on how to ask for information.4
Response type Extended written production: a letter. Teaching goal Structured communication.
Assessment Formative.
This writing task focuses on meaning with a communicative outcome. Again, a role-play activity in which the learner is put in the place of a consumer that writes a letter expressing an opinion about a product. The processes that underlie this learning activity include processes such as giving an opinion about a product or asking for information.
The task requires an extended production response, a letter. The type of learning task, the teaching goal, could be classified as structured communication and the assessment it requires, not being a final task, is formative assessment.
7.2.2.4.1 Characterisation of the response in pedagogical and linguistic terms
Table 7.16 characterises the response to this task in terms of instructions and in- put. The task’s prompt requires the learner to put himself or herself on the role of someone who is invited to send her/his opinion on Smint, produced by the Spanish
3The title in English would be: Express your satisfaction or dissatisfaction with the product
AC, Subtarea 2, Actividad 5
Prompt Imagina ahora que t´u tambi´en has participado en la encuesta de satisfacci´on para conocer la aceptaci´on del nuevo producto de Chupa Chups SA, SMINT, y quieres escribir una carta a la empresa para expresarles direc- tamente tu opini´on tras probar los caramelos. Adem´as, quieres aprovechar la circunstancia para pedir m´as infor- maci´on sobre el producto.
Instruction En el cuadro de abajo puedes consultar algunas estruc- turas de c´omo pedir informaci´on.
Input data
C´omo “PEDIR INFORMACI ´ON”
Aqu´ı tienes algunas estructuras que puedes utilizar para solicitar informaci´on:
– ¿Podr´ıa/puede decirme si ...? – ¿Quer´ıa saber si ...?
– Me gustar´ıa saber si ... Ejemplos:
– Por favor, ¿podr´ıa decirme si el tren que va a Zaragoza tiene parada en Lleida?
– ¿Quer´ıa saber si los estudiantes tenemos descuento en los museos?
– Me gustar´ıa saber si la pr´oxima semana podemos visitar la nueva f´abrica para ver todos los adelantos t´ecnicos que se han incorporado.
company Chupa Chups, SA. The learner is encouraged to send a letter and to seize the opportunity to ask for further information.
The prompt assumes that the learner has gone through the previous tasks in Subtask 2 in Atenci´on al cliente, where s/he is introduced to the topic by reading a corporate report summarising customer satisfaction for Smint, as well as some consumer opinions. In the previous tasks the learner was introduced to some linguis- tic resources useful to express satisfaction and dissatisfaction. The prompt and the instructions do not pose any further restriction on the response.
As for the input data, the learner is provided with a space to respond and a list of formulas frequently used in Spanish to ask for information. For each of these formulas an example is provided. Examples are not related to the task’s topic.
As for the relationship between input and response, this task presents a consid- erably broad scope. The length of the instructions and the input is relatively short, and the length of the response is open. They need to produce a letter expressing their opinion, as much as they need to say, as little as they need to say.
As for the directness of the relationship between input and response, this is relatively indirect. To complete it, learners can rely on non-linguistic and linguistic resources of their own choice. It is not a free composition. The topic is a specific candy and it has to be about expressing likes and dislikes, and asking something about it. But the distance between input and response is big. The actual contents of the response are not much restricted by the input. Learners are expected to express opinions and ask for information, and are given some hints on how to do it, but not required to use specific expressions.
The thematic and the linguistic contents for this task are characterised in Table 7.17. The first block in the table presents the entities expected in the response: there has to be a reference to the product Smint. There might be references to entities such as sweets, candies, lollipops, or product, and even to entities such as company, Chupa Chups, SA. But the reason that made consumers happy or unhappy with Smint is uncertain, or open. It could be the price, the size, the flavours, the colour, the packaging, etc.
As for the relations, instructions impel the learner to state that s/he bought or tried Smint. Anything else related to satisfaction with Smint depends on the learner’s background or imagination. The instructions require learners to ask about the product. However, might it be about whose idea was it to pack it in such a box? Or about what flavour is to appear next in the market? Or what else? Answers are open in terms of thematic content, and difficult to predict in lexical terms.
As for the analysis in terms of linguistic contents, instructions are explicit about part of the expected functional contents: ask for information. The linguistic input data below the space reserved for the answer give specific formulas for this.
As for the expression of opinion, only the previous tasks in the Subtask might help. Learners have to express their opinion about a candy, a type of food. This will include describing it, being able to say positive and negative things about it, and so on. The exponents of function provided to learners in previous activities to express likes and dislikes are in the functional contents area in Table 7.17.
Thematic content of the expected response
Entities
– Smint
– Chupa Chups – Sweets, products
– Consumer (the learner role) Relations
– You buy or bought Smint and tried it.
– You like and disliked certain things about Smint – You want to know more about Smint
Linguistic content of the expected response
Functional
– Express what you think about a product: Me gusta mucho/much´ısimo SN/que... Me encanta/encantan...
¡SN est´a Adj. Superlativo!
¡Qu´e malo/bueno (que est´a) (SN)! No me gusta (nada) SN/que... Odio SN/que...
– Ask for information:
¿Podr´ıa/puede decirme si ...? ¿Quer´ıa saber si ...?
Me gustar´ıa saber si ...
Syntactic
– Past tense to express what you did – Present tense to explain what you think
– Conditional tense (to express preferences or to ask for information)
– Courtesy forms in pronouns and verbs (3rd person sin- gular)
– Use of relevant prepositions and conjunctions (3rd per- son singular)
Lexical – Smint, Chupa Chups, caramelo, caramelo de palo, gus- tar, encantar, odiar, sabor, rico, malo, bueno, caro, pr´actico, sano, etc.
Pragmatics – Use the appropriate register. – Use an appropriate letter structure Graphology – Use the appropriate spelling.
Table 7.17: Thematic and linguistic content according to the RIF for Activity 5 in Subtask 2 in Atenci´on al cliente.
experiences, present tense to express opinion, conditional tense to express prefer- ences or to be polite, and the use of courtesy forms because of the formality of the setting. Also the use of prepositions and conjunctions with topic relevant words will be assessed.
As for lexical content, not all of the expected elements are easy to predict. In- structions and the task topic ensure the appearance of Smint, candy, Chupa Chups, (no) me gusta, me encanta, odio, .... Moreover, talking about sweets might facili- tate the use of words such as rico [yummy], sabroso [tasteful], pegajoso [sticky], sano [healthy], az´ucar [sugar], diente or muela [tooth, back tooth], and similar. However, instructions and input data are open enough in this respect to make it diffcult to predict.
Finally, in terms of pragmatics and graphology, the letter that learners are ex- pected to write has to comply with the norms of the communicative setting and the text genre. If it is a letter, it should include the addressee and sender contact details, a place and a date, usually at the end. It will also require a greeting, the contents of the body (opinion and asking for information), a complimentary close and a signature. Given the relative formality of the communicative setting, the text should observe the courtesy, spelling, grammar, and norms socially required.
7.2.2.4.2 Assessment
Since this task requires formative assessment, the criteria for correctness need to be established in terms of correct/incorrect, so that feedback is provided appropriately. However, correctness for this task is harder to establish than the previous ones in terms of the thematic content: There is no one particular aspect of the product to be praised, nor one to be criticised. A correct response should consist of:
• A letter expressing an opinion about Smint with addressee and sender contact details, greeting, body, complimentary close, and signature.
• The body of the letter with:
– Statement of have tasted Smint – Opinions about the product Smint – Questions about the product Smint
• In terms of language knowledge the response has to include:5
– Expressions to communicate personal experience with a product – Expressions to show satisfaction or dissatisfaction with a product – Expressions to ask information about products
– Structure the text in a coherent and cohesive manner – Use expected text type structure
– Use expected register, as well as appropriate spelling and grammar norm
5All these aspects have been described in detail above and are reflected in the corresponding
7.2.2.4.3 A RIF-based list of predicted responses
Despite producing the criteria for correctness, we do not provide a sample response for this task. None of the content designers provided one during the ALLES project, and its thematic contents are too open. Later on, in Chapter 8, we propose a particular approach to assess this kind of task, and in Chapter 9 we analyse some real learner responses to this task, which we can compare with our initial analysis.
7.3
Chapter summary
In this chapter, we presented two frameworks that we consider an essential part of what should become a methodology for the design and development of ICALL tasks. In our view, these two frameworks establish a connection between the needs and requirements of FLTL, and the linguistic and assessment specifications for NLP. This connection is made in a detailed characterisation of learner responses and assessment criteria in the design phase, as a primary input to NLP.
The TAF and the RIF help describe the pedagogical properties of FL learning ac- tivities using concepts and terminology generally accepted in FLTL and SLA. They also help us specify in detail the linguistic contents and the assessment criteria of these tasks using concepts and terminology from the field of linguistics, a field that serves as a common language for FLTL and NLP, both of which are concerned with language but with different perspectives. FLTL looks at language as a communi- cation system that has to be learned and used in a competent manner, while NLP looks at it as a complex system to be computationally formalised.
To exemplify the use of the TAF and the RIF, we applied them to four FL learning activities that exemplify four learning task types, three of which are good representatives of the viable processing ground. Figure 7.1 shows how these four tasks could be approximately situated on the viable processing ground (extending the figure in Bailey and Meurers (2008: p. 108)).
Task type I is most to the left, because its responses are limited production re- sponses, and the interrelationship between input and response is narrow and direct. Task types II and III are slightly further to the right because their responses are extended production responses, the interrelationship between input and response is relatively broad, and for both of them the input-response interrelationship is no- tably direct both in terms of thematic content, and a bit less in terms of linguistic content. Task type III is different from Task type II in that it requires summative assessment. Finally, Activity type IV is the one that is most to the right, because it presents extended production responses, and a broad and indirect input-response interrelationship, particularly in terms of content.
Once FL learning activities are characterised with the TAF and the RIF, the analyses inform the feedback generation strategy about the pedagogical needs, the nature of the expected responses and the assessment criteria. With this information software specifications can be developed and the implementation of the NLP-based analyses modules started. This is what we focus on in the following chapter.
Chapter 8
NLP functionalities to respond to
FLTL demands
This chapter presents a proposal for the specification and implementation of an auto- matic assessment (AA) module on the basis of automatic linguistic analysis. Because the automatic assessment module we are aiming for is a pedagogically informed one, we propose to base the specifications for the AA module on the RIF, presented in the previous chapter. The RIF provides the pedagogical and linguistic information, sets of expected responses and criteria for correctness, so that we have access to an explicit representation of the information upon which the language processing and feedback generation modules can be based.
We first introduce the Automatic Assessment Specification Framework (AASF), designed for the specification of the technical requirements for both the language analysis module and the feedback generation module. The AASF is step-wise con- version of pedagogical and linguistic features into specific analysis needs at the level of linguistic knowledge and the level of topical knowledge, using Bachman and Palmer (1996)’s terms, or at the level of meaning and form, using Bailey and Meurers (2008)’s terms. We first introduce the AASF formally and then exemplify its application on a selection of ICALL tasks, including both tasks requiring formative feedback and tasks requiring summative feedback.
Second, we present the actual implementation of an automatic assessment module on the basis of the specifications derived from the AASF. Our description includes a general approach to feedback generation on the basis of the shallow NLP pro- cessing tools presented in Chapter 6. After this, we describe the implementation of NLP processing strategies to handle task-specific linguistic analysis beyond the morphosyntatic level under the assumption of the task characteristics as a domain. Moreover, we describe how this general approach to feedback generation can be instantiated for specific ICALL tasks, where we distinguish between the approach followed for the generation of formative feedback and the approach followed for the generation of summative feedback.
8.1
From pedagogical requirements to specifica-
tions for the Automatic Assessment
This section presents the Automatic Assessment Specification Framework (AASF) as a means to establish the requirements of the linguistic analysis strategy and the feedback generation strategy on the basis of the RIF.
8.1.1
AASF: Automatic Assessment Specification Framework
The AASF consists of two main parts: the Specifications for Automatic Lin- guistic Analysis (SALA), which will provide NLP-oriented specifications for the analysis of learner responses to a particular activity, and the Specifications for the Feedback Generation Logic (SFGL), which will provide a feedback genera- tion logic to make hypotheses on the correctness of learner responses that link the linguistic analysis to “canned” feedback messages.
8.1.1.1 SALA: Specifications for Automatic Linguistic Analysis
The linguistic analysis pursues both the analysis of meaning and the (relevant) anal- ysis of form. The analysis of meaning will be related to the thematic content speci- fications of the RIF, where the topical knowledge of the activity is characterised. As for the analysis of form, it will be related to the linguistic content specifications of the RIF, where the linguistic knowledge of the activity is characterised.
For each item in the thematic and linguistic content parts of the RIF, the SALA provides the following information:
1. Reference: a description of the individual or the relation that a particular piece of language will be referring to
2. Linguistic cues: specific linguistic units and structures that can be associated to the reference
3. Code: a codification for the analysed phenomenon to be linked with particular feedback generation logics
4. NLP module: the NLP module or functionality required to trigger the ex- pected piece of language
Figure 8.1 reflects the procedure through which the elements of the thematic and linguistic content parts in a RIF analysis of an ICALL activity can be converted into NLP specifications. From step one to step two, for each of the items in the RIF specifications a set of cues has to be defined in terms of linguistic or textual information. The third step consists in establishing a code for the phenomenon on which the logic for feedback generation will be based. The last step is to identify the NLP module that can provide the function, the automated procedure, to analyse (detect, annotate) the expected linguistic cues.
Choose item Define linguistic cues Establish analysis code Determine NLP module
Figure 8.1: NLP specification procedure.
8.1.1.2 SFGL: Specifications for the Feedback Generation Logic
The feedback generation logic is the brain – the “reasoning” – of the Feedback Generation module. It pursues the assessment of the learner response in terms of criteria concerned with a reduced number of words in the response, local level, or with the response as a whole, global level. Independent of the locality or globality of the criterion, assessment aims at the meaning and the form aspects of the response, which are related to the thematic and linguistic contents in the RIF.
For each of the items in the RIF-based criteria for correctness, the SFGL provides the following information:
1. Criterion: the particular criterion to be checked for in the learner response 2. Priority: the criterion’s priority can be high, medium or low
3. Match: the type of match is defined as full, partial or zero match 4. Message: the feedback message associated with each type of match
5. Type: messages can be mutually exclusive, marked as main, or can be added to messages of type main, marked as addable.
Figure 8.2 reflects the procedure for the specification of the feedback generation logic. The first step is to select the criterion, or the conditions (codes or code sequences), for which specific feedback messages will be generated. Afterwards the priority of the criteria can be established. A higher priority will result in a greater