Accuracy Measures

As mentioned in 2.4.3, our definition of accuracy relates to “the extent to which an L2 learner’s performance (and the L2 system that underlies this performance) deviates from a norm (i.e. usually the native speaker). Thus, deviations from targetlike performance

would be considered errors” (Housen et al. 2012, p. 4). In addition, accuracy also relates to “appropriateness and acceptability” (p. 4). For the purpose of this study, deviations from targetlike performance relate to grammatical errors and communicatively inadequate use of the targeted forms: OS and OO RC types as well as the accompanying use of cognitive state verbs. In other words, if a learner produced an OS RC type that was grammatically correct but it did not reflect the context of the storyline it would be considered inaccurate. It was therefore necessary to use a measure that could gauge grammatical errors relating to OS and OO RC types and the accompanying use of cognitive state verbs. As the participants in the present study were of similar intermediate proficiency to previous studies (for example, Yuan & Ellis, 2003, Foster & Skehan, 2005), it was decided that similar accuracy measures should be used to enable comparisons with other studies. As we saw in 2.4.4, studies which have investigated syntactic accuracy such as Yuan & Ellis (2003) used ‘error-free clauses’, specifically, the percentage of clauses produced that do not contain any lexical, syntax or morphology error. A typical lexical error could include “I was waiting you” (p. 14). As a result, the present study measured grammatical accuracy according to the percentage of ‘error-free relative clauses per relative clause’. This involved counting all the relative clauses that contained no grammatical errors and dividing them against the total number of relative clauses produced per narrative as follows:

Number of error-free relative clauses

Number of relative clauses *100

An example of an error-free RC could be ‘the girl likes the dog which has long

ears’, in this case, only the RC would be measured. A typical grammatical error could

therefore be ‘the dog which have long ears’.

In terms of the remaining part of our accuracy definition: communicatively adequate use of the targeted forms, learners’ accuracy was based on the context in which they used the forms. In other words, in line with Mochizuki & Ortega (2008), the learners were required to produce seven obligatory contexts of RCs that reflected the storyline, thus if an RC was produced that was grammatically correct but communicatively inadequate i.e. it did not relate the storyline it would be excluded from the analysis. For example, describing a picture which contains a dog with long hair but commenting ‘the cat which has short hair’. In terms of over-use of the forms, repeated

RCs were excluded from the analysis. For example, as in the underlined structures ‘he likes the dog which has black hair, which has black hair’ therefore if a learner produced twelve RCs during a narration, they would only be graded on the seven obligatory contexts as shown in figure 8.

As we saw in 2.4.4 previous studies have also used multiple measures for CAF, therefore an additional measure for accuracy was used ‘percentage error-free relative

clauses per AS-unit’. An AS-unit is defined as “an independent clause or sub-clausal

unit, together with any subordinate clause(s) associated with either” (Foster, Tonkyn & Wigglesworth, 2000, p. 365). Although the definition of an AS-unit is very similar to the definition of a t-unit or a c-unit which was described in 2.4.6, as they all consist of an independent clause with any subordinate clauses attached, an AS-unit was chosen because it “allows for the inclusion of sub-clausal units which are common in speech” (p. 366). Sub-clausal units comprise of elliptical utterances which are shortened phrases that can be inferred into full clauses based on the pragmatic meaning of the situation. Foster, Tonkyn & Wigglesworth (2000, p. 366) provide an example:

A: How long you stay here B: Three months

Although these phrases contain grammatical errors, they have pragmatic meaning within the context and therefore each utterance would be considered an AS- unit. However, repeated clauses and incomplete clauses were excluded from the analysis. For example, as in the underlined structures ‘he likes the dog which has black

hairs, which has black hair.’ Thus the following formula was used:

Number of error-free relative clauses

Number of AS-units *100

Accurate use of cognitive state terms was measured in a similar fashion to relative clauses: ‘percentage of error-free cognitive state verbs per AS-unit’ and ‘percentage of error-free cognitive verbs per cognitive state verb’. For the purpose of the pilot study, cognitive state verbs concerned the underlined verbs on the guided planning note-sheets, for example: ‘She thinks that she likes the dog which looks

cognitive state verbs included incorrect subject-verb agreement, for example, ‘He think

that he…’, ‘He thinks like dog…’ The analysis involved counting all the cognitive state

verbs that were used accurately and dividing them against the total number of AS-units for the former measure, and total number of cognitive state verbs produced for the latter measure as follows:

Number of error-free cognitive state verbs

Number of AS units *100

Number of error-free cognitive state verbs

Number of cognitive state verbs *100

Further examples and formulas for measuring accuracy will be explained in more detail in the next chapter (5.7.2).