Content coding of idea units

In order to look into the retold content candidates produced, a second layer of coding was carried out which assigned categories to the idea units in the retellings in terms of how they correspond to the source content.

The generation of categories for the idea units was based on models of text comprehension and observations from the retellings. Kintsch and van Dijk (1978) proposed three sets of mental processes involved in the comprehension and production of summary protocols: first, the meaning of the text is comprehended and organised into a coherent whole; second, the full meaning of the text is condensed into its gist; third, new texts are generated from the previous two processes and produced. In the last process, memory of the source text and the speaker’s background knowledge interact in the construction of new texts (Frost et al., 2012). For summary tasks which this comprehension model is based on, the three mental processes are seen as sequential with the production of a summary happening at the end of the third process. In the TEM4-Oral retelling task, however, since candidates are specifically instructed to ‘retell’ and not paraphrase or summarise, they are less likely to condense, synthesise or generalise the source content. This means the three processes are parallel in that the production of an idea unit can happen at the end of each process. This was supported during the pilot coding process, when it was found that the idea units in the retellings ranged from near replications of source story idea units (e.g. as Jane was expecting her first child, compared to the source story, just as Jane was joyfully expecting her first child) to bits of improvised content (e.g. the story is between mother and son). Idea units in the retellings were thus categorised

into three main kinds: (1) comprehension and replication of source content, (2) generalisation of source content and (3) construction and improvisation.

The first two of the three categories were then each broken into two subcategories denoting correct and distorted productions accordingly. This decision is based on Johns and Mayes (1990), which argued that second language users may encounter breakdowns at any stage of the production process, leading to inaccurate representations of the source content. For replications and generalisations of source content, a breakdown could potentially result in distorted meaning. This contrasts with the construction and improvisation stage, in which the content produced involves background knowledge of the candidates and therefore deviates from the content of the source story. In this case, idea units produced cannot be determined of their correctness in relation to the source story.

Regarding the second category, ‘generalisation of source content’, observations from retellings suggested that candidates tended to merge multiple idea units in the source story into one idea unit without much generalisation. Therefore, ‘replications of multiple idea units in the source story’ seems more appropriate as the definition of this category.

Building on the above considerations, each idea unit in the retellings could be assigned one of the following categories (idea units of focus are underlined; for the complete codebook, see Appendix E):

1. Correct replication of a single/partial idea unit

Retelling: Just as Jane was going to give birth to her first child

2. Distortion of single/partial idea unit Source story: At only fifty-five

Retelling: And at, just, er, only at forty-five years old

3. Correct replication of multiple idea units (condensation of multiple idea units in the source story into a single idea unit in the retelling)

Source story: the fiercely independent and courageous mother had fought | she became totally disabled

Retelling: her independent and active mother was totally disabled

4. Distortion of multiple idea units (condensation of multiple idea units in the source story into a single idea unit in the retelling, with the content distorted) Source story: For ten years, | the fiercely independent and courageous mother had fought, | but none of the surgeries or treatments | had been successful. | At only fifty-five, | she became totally disabled.

Retelling: her mother, an energy an energer energetic mom, an energec woman was lose her consciousness, was lose her consciousness.

5. Construction/improvisation (generation of new information, related to source story content, but not corresponding to any of the idea units in the source story) Source story: Jane called home. | “Mom, listen. | My baby is coming! | You’re going to have a new grandchild. | Do you understand!”

6. Other (irrelevant to source story content)

Source story: On February 3, 1989, | at about the same time Jane began to feel birth pains, | Mother opened her eyes.

Retelling: On February the third, 1989,| you will know what happened.

The limitations of this coding procedure lie in two aspects. First, since the focus of this analysis is on content only, grammatical errors in the idea units do not count towards their categorisation. To illustrate, a retold idea unit such as

her mother is being dead was considered the same as her mother was dying under the criteria of this coding. Although content is considered important in the marking criteria of this task, grammar is undoubtedly something raters would attend to. Another aspect this coding procedure excludes is the sequence of idea units in the retellings as opposed to in the source story. Since idea units were coded individually, the sequence in which they appeared was not accounted for in this analysis, though it might influence the overall structure of the retelling. Nevertheless, this additional layer of coding is revealing about the quality of the content produced by candidates of the two ranks in relation to the source story.