The preliminary segmentation of all the TA data prior to full coding

The next stage we felt to be needed prior to a full coding was careful segmentation of all the TA protocols collected. Segmentation is performed prior to coding as way of identifying those stretches of transcript which signal something that is felt to require a separate code, even if at this point what the code will be is not yet clear. It is a controversial matter, which some researchers define as arbitrary and intuitive (Shirazi, 2012; Lumley, 2005). In qualitative research, the segmentation process often involves breaking verbal data into separate chunks or idea units of different lengths (e.g., phrases, clauses, sentences, etc.) to be later categorized and coded. Even at this stage certain tentative labels may be assigned to the segments (Dörnyei, 2007).

aloud protocols. Ericsson and Simon (1993) point out that in speech the boundaries of phrases are usually signalled by the pauses. Linguistic boundaries cannot always solve the segmentation problem, however. Paltridge (1994) also suggests that segment boundaries in transcripts need to be decided on the basis of the content of the texts instead of the way in which the content is expressed linguistically. Moreover, he claims that the divisions within texts are ‘intuitive’ and frequently made in the absence of “textual indicators” (1994, p.295).

Someren, Bernard, and Sandberg (1994) put forward another method for segmentation. They suggest that “the combination of these pauses and linguistic structure provide a natural and general method to segment a think-aloud protocol” (1994, p. 120). They underscore that a high level of agreement between people exists while they listen to think aloud protocols. They believe however that what makes segmentation more difficult and less reliable is to segment the protocol on the basis of the written transcript only. Green (1997) by contrast pays more attention to the content, and suggests that each segment should represent a different process and will usually comprise a phrase, clause or sentence. Lumley (2005) on the other hand, believes that this process is simply arbitrary as it leaves “a lot of scope for decision making” (p. 135).

Taking account of the above views, eventually, it was decided that in our study segmentation into segments or text units (TUs) would be guided by attention to a combination of linguistic structures and pauses plus a small number of other principles. Above all, however, the segmentation of the raters’ think aloud talk was based on units of thought or content. In addition, the following guidelines were developed for segmenting the TA transcripts.

• All the talk associated with beginning the rating task and identifying the script was considered as a single unit because it was dealing with instructions that had been given to the raters by the researcher about what to say at this point. A separate text unit was assigned for identifying the name of the student if mentioned.

• A separate segment / unit was used for:

§ Each piece of text read aloud from script

§ Each separate comment made by the rater. This raised several dilemmas since some comments could be perceived as concealing single or multiple ideas and sometimes it was impossible to determine when a rater begins to express a new idea. However, the appropriate approach was to attempt to create a new text unit for each new idea or topic or apparent change of direction. The idea was to create too many texts units rather than too few.

§ The nomination of an evaluation category was considered a separate segment.

• The rater's first reading of the essay was considered as a single unit even if disrupted.

• When there was repetition of the same idea or phrase, it was treated as a single segment.

• When the rater identified many evaluation features in one sentence (for instance, spelling, grammatical mistake, punctuation) each one of these was considered as a single segment.

• Any comment about giving the score in the final evaluation was treated as separate.

• Reading part of the script, the whole script, or rereading the script were each treated as one segment.

Excerpt 1 presents an example of segmentation arrived at this stage for the start of a TA protocol. The rater first identifies the script number, then mentions the essay topic and thirdly reads part of the title. He next reads the writer's name (4), followed by activating background knowledge of the writer, then (6) indicates that he will start reading the text. Excerpt 1 Example of segmentation of TA at the start of a script

In Excerpt 2 we show an example of segmentation in part of the data where criteria and scoring were covered. The rater offers hedged positive evaluation of grammar (80) with the words 'has potential' and states his negative evaluation of one specific grammatical feature (81) and how it affects the score for grammar, coincidentally implying that he is working with separate rating scales for different aspects of the language (82). Another feature ‘Vocab’ is targeted (83) with the positive word 'OK' hedged with a negative reference to being 'not very advanced'. 84 shows the rater’s negative evaluation of ‘spelling mistakes’ in the text. In 85, there seem to be reference to a previous negative

judgment about ‘paragraphing’, followed by deliberation on the score given in 86, before making a final score judgment (87).

Excerpt 2 Example of TA segmentation containing both positive and negative evaluation on various criteria

Transcript segmentation raised several dilemmas. One was incompleteness of some comments. Transcripts are recording what was said rather than thought processes directly, and raters focused on the rating task not on completeness or explicitness of utterances. That is why segmentation followed the principle of content above linguistic features. However, some comments were left unfinished or contain opaque references. In Excerpt 3 we see three examples of this. In 11, T4 fails to complete saying what he is going to 'make'. Later (12) he says that that the student 'is using this’ but the reference of 'this' is not clear to the researcher. Again in 13 the rater referred with demonstratives to a previous topic in class teaching without saying what it was. Since segments are meant to be potentially codable chunks, these all were problematic since the lack of information made it hard to see how they could later be coded.

Another issue was repetition. In Excerpt 12-15 are multiple utterances about the same thing, the voice of a specific bit of student text. Some are similar in wording, others less so. This then raises the segmentation issue as to whether these are all the same segment or not.

Excerpt 3 Examples of incomplete or inexplicit comments

Excerpt 4 Example of repetition of the same idea

These working analyses, as produced at this stage, showed that the process of segmentation of the text and a preliminary coding analysis are closely intertwined. The segmentation process is itself an iterative process subject to continual modification in the course of the data analysis (Miles & Huberman 1994; Green 1997).

TU teacher’s talk

11. I am going to make a …. for extra marks

‘’He is fairly clever’’ 12. um... he is using this….

13. I was telling him that in class. He did not seem to know this! But he does!

To sum up, transcripts were tentatively segmented into TUs based on content; TUs were numbered. At this point we additionally provisionally identified the following types of content in the units that emerged, ranging much wider than just different types of criteria:

• Identification of each script by mentioning the number or name. e.g.Let’s start with number 1, she is a Saudi lady from C1B.

• Reference to reading as part of the rating process e.g. I am going to read this again to make sure.

• Criteria that raters refer to on the way to making their decisions e.g. Usually, these essays are full of grammar mistakes,

e.g. That is a nice introduction.

e.g. While the first student was so grammatically accurate actually. I focused mainly on ideas

• Reference to the text and/or the writer

e.g. She is so intelligent, she knows how to write complex sentences and she is supporting her ideas.

• Non-verbal act e.g Will put a tick.

• Justifications made for specific rating or score

e.g. And if she provided another paragraph before the conclusion, she would get 8, perhaps

• Activation of knowledge of the student background or level

e.g. This is (Ba**) from Turkey.

e.g. Umm, as far as I know she got problem in grammar I think. So, let me start. e.g. Ok who is this? Oh, this is (**za). She is nice. She got 7.5 in listening. So, let’s see how she writes?

• Problems raters encounter in the rating process. e.g It is difficult to read his handwriting.

3.9.6 Transferring the data into qualitative data analysis