Evaluation of the final texts

In both languages, the plans and the essays were read by two independent readers. The English texts were evaluated by (bilingual) L1 English speakers (Canadian and South African), the German texts by L1 German speakers. At the time of the tests, all of the readers taught at Cologne University: the English readers were lecturers in English-language essay writing classes, the German readers taught German literary studies. One of the German readers was also a teacher of academic writing. The readers did not know who the participants were and they were not given any information about their university background (e.g. their level of study or whether they had spent time abroad). When evaluating the papers, the readers were asked to mark all errors they found, both in terms

of language and content. They were also asked to evaluate whether the statements were clear and whether they were appropriate to the genre (Rijlaarsdam et al. 2012: 203).

The L1 English evaluators marked the errors in both the essays and in the plans.

They were asked to be ‘hyper-critical’ in their error marking and to refrain from ‘helping’

the participants by trying to interpret what they might have been intending to express in particularly ‘un-English’ passages – that is, passages where the semantics of the wording or the sentence structure were such that an L1 English speaker with no knowledge of German would have been incapable of understanding the message. Still, in the discussion of the errors, it became clear that in some cases the readers did do some interpreting – if only subconsciously – so that some semantic node-switches might have gone unnoticed.

4.2 Analysis

The data that were generated in the tests were used to analyse the fluency of the production processes, the types of errors that were found in the final texts and their rootedness in the L1, and the kinds of revisions that were done in the writing processes. Although think-aloud protocols are currently popular for analysing writing processes (e.g. Braaksma et al.

2004, Göpferich 2007, van Weijen 2008: 23), in this study, no think-aloud protocols were made, since it has been shown that the demands of speaking while writing impede the writing processes and lead to less fluent writing (Emig 1971: 92, Janssen, van Waes, and van den Bergh 1996: 248/249, Spelman Miller 2005: 299, see also: Stevenson, Schoonen and de Glopper 2006: 209). Instead, the keylogging method was used.

4.2.1 Keylogging

To make it possible to analyse fluency and revision, the participants’ text production was recorded using the keylog programme Translog 2006. This programme was initially developed for translation process research, but it is also very efficient for research into task-related writing (e.g. in Jacobsen 2006, Lindgren and Sullivan 2003, Sullivan and Lindgren 2006). Translog records all of the movements a writer executes on the keyboard:

it logs the use of the mouse and records where the keyboard and mouse events take place on the screen. The duration of the individual processes is recorded as well.

Translog has two interfaces. The user interface simulates a typical word processing programme (Fig. 4.1). In the upper half, the assignment is presented, and in the lower half the participants can write their text using simple editing devices such as bold or italic letters, deleting, moving the mouse, etc. Neither a spell checker nor a grammar or style

checker is provided. The writers cannot change the font, nor can they use any graphic devices, symbols, footnotes or other more refined word processing tools.

Fig. 4.1 Translog user interface

The supervisor interface consists of three parts (Fig. 4.2). In the upper left block, the analyst can read the assignment that the writer was given. Immediately below, the programme can simulate the writing process: it can recreate in real time the way in which the text was generated, including where and when the writer made pauses and where and when they revised, reformulated or went from one place on the screen to another. The analyst can also press a button to see the edited text. The right side of the screen provides a visualisation of the linear version of the writing process. In the example above, the participant (Thor – a participant in the pilot study) started with a ‘pause’ of 2 minutes, 4 seconds, and 498 milliseconds during which he read the assignment. After this, he moved the mouse and a new pause of a second’s duration began. He pressed the return key to indicate that he had finished reading the assignment and then spent 11 minutes, 14 seconds and 481 milliseconds reading the source text. After this he pressed return again and started the planning process – in this case, freewriting. The stars indicate pauses of a second each.

When he wrote ‘dont’, he deleted the d, as indicated by the framed back-arrow following the letter. The simple arrows in the line below show that Thor then pressed other keys, and so on.

Fig. 4.2 Translog supervisor interface

The script can be saved as an Excel document. This document presents each keystroke/movement in an extra row, and it can be used as the basis for further analyses.

For example, it can be used to determine how many characters the writers were able to produce in one burst without pausing or revising. The table also includes the time the participants took for the writing task, the process (planning, formulating, revising) they were engaged in at each point in time, and so on. It is possible to evaluate whether corrections were made in an automatised way (deleting unwanted characters/words directly) or whether they took place after a pause, etc.

The results of these analyses were integrated into an SPSS document, which was used for the further statistical analysis of productivity and fluency in the writing processes.

Since only ten students participated in the study, it made little sense to conduct significance tests (e.g. ANOVA, Fisher-test), since no reliable probability could be calculated (Cohen, Manion, and Morrison 2007: 93). For this reason, the analysis was made in form of descriptive statistics.