Reviewing the list of codes after the first cycle of coding

The list of codes that emerged during the first cycle was reviewed and examined with the help of my two advisors and with an expert in qualitative research software to increase the validity of the analysis. The process of code examination encompassed many aspects. Firstly, it looked at the content of the codes, i.e., what they cover or reflect. It also looked at the length of the code list. Furthermore, it covered the names of the categories and the relationship of the code to each category. The following

paragraphs discuss each area of the code examination and the decision made in relation to them.

Regarding the first area of examination, which is the content of the codes, the review showed that the codes cover many different content areas, most of which are evidenced from more than one teacher. The codes were designed to fit all the data produced by teachers’ think-aloud and relevant parts of what they said in the immediate retrospective interviews. The system includes codes for rater (teacher) actions that are not in themselves evaluative: for instance, the code Reading behaviour is a general code that covered five sub-codes that represent individual behaviours, e.g., Read the whole text, Reread, Read part of the text, Scan whole composition and finally Read essay prompt. Furthermore, there were many other codes that represent forms of evaluation, e.g., Consider error gravity and Consider error frequency. Moreover, there were subdivisions of certain main codes to capture further aspects of that code, as seen for Reading behaviour above and again for Classify error into types which included many sub-codes like: Verb form, Sing./pl. noun, Prepositions, Plurals and Articles. In the analysis, I tried to concentrate mainly on the evaluative writing criteria without looking too much at how teachers give written feedback.

The second area of concern was the length of the code list. A close analysis of the code list however showed that its length needed to be considerable for several reasons. Firstly, the nature of the present study obliged me to design a considerable number of codes to fit the different behaviours of teachers as each one had his/her own procedure and criteria in assessing their own students’ writing. Indeed, in the final coding, 59% of the codes occur in data from only two or one rater and none occur in data from all six teachers. Furthermore, in our analysis it is inevitable to describe reading behaviour as an essential component of writing assessment. Hence, there had to be detailed codes

for Reading behaviour as we mentioned earlier, which inflated the number of codes. Moreover, no one can deny that the Language aspect is an integral feature of any writing rating scale. Hence, our scheme of codes needed to cover this aspect with its specific criteria, e.g., Grammar, Mechanics (spelling, punctuation, and capitalisation), Layout and Language fluency. One more reason is because, due to the special interest of our study that investigates teachers’ rating of their own students’ writing, I created a code for Teachers’ reaction to accord with the uniqueness of this research. Hence it was of importance to my research to keep many codes regardless of their effect on the length of the code list.

Furthermore, the review considered the names of the categories and the grouping of the codes under each higher category. For example, the category Cohesion and coherence seemed initially to refer to a general aspect of writing assessment but we later realised should be part of the Organization category. The Vocabulary code was also placed under the Language category, and so on. These issues were considerable, however, and the final code categorisation was only arrived at after the second cycle of coding. Finally, the review helped me to refine some codes and exclude others. For example, there were a Positive evaluation and a Negative evaluation codes that ware excluded from the coding list.

Having reviewed the codes, the next step was to apply the outcome of this review to the list of codes we had. First, I created a new folder in Nvivo 10 called the Thematic Coding Framework in which I copied the first list of codes from the previous nodes folder. In this new folder, I copied the refined codes only and left out the unrefined ones. Next, I added some new codes that captured in more detail teacher’s reactions

which were not evaluative, for example: Classroom teaching actions and Emotional reaction.

At this stage, it seemed reasonable to compare the list of codes I developed with other codes for the assessment decision making process in the literature to see if I missed something important. Hence, I looked at two list of codes from Cumming et al., (2002) and Lumley (2005). When I checked my list against these two lists, there were some common areas between them. The complete new list of codes at this point consisted of 72 codes.

The review phase required us to think deeply about the entire code list and therefore increased our general understanding of the codes. The outcome of this examination was used to start our second cycle of coding from a more informed position.