Pilot Study and Inter-rater Reliability

While conducting qualitative research, intersubjective judgements on the data being observed needed to be prepared for whilst ensuring trustworthiness, credibility, confirmability, transferability and dependability (Cohen, Manion, & Morrison, 2018, p.290). In order to do this, an inter-rater can be used in order to assess whether an annotator independent of the research using the same framework would make the same interpretations (Cohen, Manion, & Morrison,

4 _{In the AWC, the writer identifier refers to the same student throughout, in order to facilitate the}

possibility of analysing individual development over time. In the EWC, each text is written by a different writer.

2018, p.271). Any differences in classifications should be discussed and a common approach to annotation adopted, which could then be used and referred to for the remaining textual analysis and to modify any existing analysis.

One intercoder was asked to look at a sample of texts (n=6), annotate them for instances of modality using the MAI and then classify them according to Halliday’s Modality System’s Network. The intercoder was an advanced English speaker of Turkish origin, who was a university lecturer in linguistics and, at the time, completing a PhD in Education in the area of Metadiscourse.

Both the researcher and the inter-rater independently annotated and coded four amateur writer texts (A-ES(L)02, A-ES(S)07, A-CS06, and A-RR01) and two expert texts (E- CS05 and E-RR04). The texts were chosen at random from each genre. It was the intention of the researcher to include an additional text from the expert essays corpus to ensure a spread of texts had been used across the genres; however, the inter-rater was unable to commit to any additional annotations due to other obligations and it was therefore excluded.

The results of the inter-rater process identified a number of issues for discussion. Despite closely matching in the number of instances of modality in both types of text (see Table 3.12), the raters identified the same item as modality in only 79 instances in the amateur texts and only 51 of the instances in the expert texts. Of the 79 amateur writer items, and the 51 expert writer items, 43 of the instances of the amateur annotations and 27 of the instances of the expert annotations were awarded the same classifications in all elements by both raters. Exact matches in classifications would be needed in type of modality (including sense groupings), orientations, values and polarity. The remainder of the modal instances received varying classifications.

Table 3.12 Results of inter-rater process

Instances of modality recorded in:

Amateur Texts Expert Texts

Total number of modal instances

identified:

Researcher: 108 85

Inter-rater: 104 78

Identified the same item as modal 79 51

Exact matches in all elements of classification

43 27

Difference in one or more elements of the classification

36 24

Closer inspection of the results show that divergence between the raters was focussed principally on the notoriously difficult classifications of the modal verbs can and could. As discussed in section 2.3.1, the potential meanings of the verbs (possibility, ability, permission) cross over both modalization and modulation within the SFL modal system and there is much discussion in the field about where their associated meanings lie, both in terms of type of modality and on the scale of values of low to high. The main differences between the raters in this regard were in how to classify the affirmative form of can with meanings of possibility. The inter-rater classified all instances of can as ‘modalization-probability-median value’; however, the researcher classified the modal verb as ‘modulation-ability-low value’. The difference was discussed in the subsequent follow-up meeting and it became clear that the inter-rater was basing his assumptions on his experience of metadiscourse, and the ability to classify possibility without reference to any of the main typological distinctions such as epistemic or non-epistemic modalities. Further discussion on the difference between possible can in modality highlighted the difficulty of placing it within Halliday’s system. The reason being that possibility within modalization is equivalent to epistemic possibility [glossed as ‘it is possible that…’] that aims to speculate the value of the truth of a statement. As noted in section 2.3.1, it is only possible for can to be epistemic in interrogative (Can it be John?) or negative form (It can’t be John) and never in affirmative. Affirmative can is either possibility or permission in the deontic sense [glossed as ‘it is possible for…to …’] or dynamic (with meanings of ability). Deontic modality is loosely connected with modulation (obligation) and therefore linked to

permission. The difficulty then was how to distinguish between the three sense groupings of possibility, permission and ability, as many examples are ambiguous and difficult to classify. This meant finding a way to mark ambiguous examples of can and could from the current framework, however, in a way that ensured that they were included in the overall results. The final decision was to adopt the modified framework of Halliday’s system of modality as described by Iedema, Feez, & White (1994) and used by Martin and White (2005) in their research on appraisal theory (within which modality exists in a narrow sense at the level of ‘engagement’), as it created a distinct category of ability separate from inclination (as opposed to being part of it as per Halliday and Matthiessen’s (2004) framework) and both within a new grouping entitled readiness (see Figure 3.2 below and Figure 2.2 in section 2.4 to compare with the original). The MAI was updated accordingly, as shown in Appendix I. It also meant that the working definition on modality used in this research had to be updated to take this change into account. The newly formed system network is not an ideal solution, but it did indicate where potential ambiguity in the results occurred and ensured that they did not influence the results in other groupings.

Source: Iedema, Feez, & White (1994), cited in Martin and White (2005, p.54)

Figure 3.2 The system of modality according to Iedema, Feez, & White (1994) [highlighting where a distinction has been made with Halliday’s system to include meanings of ‘readiness’]

In terms of the inter-rater differences in the values attributable to can and could, it was decided that both modal verbs would follow Halliday and Matthiessen’s (2014, p.696) classifications and be considered as ‘low value’ modality unless broader contextual information suggested otherwise. This would also be the case with the other modal verbs and their lexical equivalents, with the exception of ought to, which were included as equivalent to the value as should:

Median: will, would, shall, should (probable, probability, recommended, going to, ought to, etc.)

High: must (need, has to, is to etc.)