A Framework for Analysing Written Feedback Comments

The case study approach for this research envisaged data collection from a small number of participants, so analysis of feedback from a small sample of feedback was considered useful for triangulation purposes. I considered this type of data as central to an attempt to explore usability of written feedback, as such data can provide useful insights into the style and content of tutor comments. A number of systems for analysing comments were reviewed in Section 2.5.1. (Brown & Glover 2006; Read, Francis & Robson, 2005; Hyatt, 2005; Ivanič et al., 2000; Leki, 2006; Mutch, 2003; Walker, 2009). The analytical frameworks coded written feedback comments with a view to presenting data, often for different purposes, so it was important to arrive at a framework that was valid for the needs of the present study.3.6.1 Adapting a

framework

Some analytical frameworks discussed above were limited in the detail they could offer (Ivanič et al., 2000), or they focused on science disciplines and assignment types less relevant to this study (Brown & Glover, 2006; 2009; Mutch, 2003). In Section 2.5.1, Hyatt’s study (2005) was seen to provide useful categories for the study envisaged here. The feedback Hyatt analysed was generated from ‘essay style’ assignments given to Master’s students in Education, and they offered a reasonable depth and breadth of coverage. Given that comments could lend themselves to a number of interpretations, however, I decided to carry out further coding to deal with the level of depth of feedback, the language that it employed and the positive / negative messages it conveyed.

I added a depth coding categorisation adapted from Brown and Glover, (2006) to enable a stronger focus on the usability of the feedback. This involved a

101

second coding to identify any points that provided elaboration of earlier points, or

explanation, where further guidance was given about why a point was important or

how improvement could be made in future work.

Given the importance of the impact of positive or negative messages in written feedback (see Section 2.4.3) I decided to carry out a further re-coding to assess the relative balance of such comments in the sample using a similar approach to Read, Francis and Robson (2005). This system counted wholly positive and wholly negative comments, but also counted the ‘softened negative’ comments that mitigated the criticism in some way (see Section 2.6.1). Unlike some studies (e.g., Mutch, 2003; Walker, 2009) I decided to take the clause as the unit of analysis, not the sentence, on the basis that individual comments could be conveyed within one clause. While the study by Read discounted any sentences that had a positive and negative construction, such positive-negative pairings can be quite common, e.g., “Although the candidate has identified most of the relevant sources, I think the essay could have been strengthened by making stronger connections to the Nara criteria..” (Summative 2, Peter). In this example, I interpreted the first clause as a positive comment, and the second clause as an example of a softened negative comment. I agreed with Read et al. who argued that the cumulative effect of positive / negative feedback would have more impact on the reader than the ratio of such comments, but I felt that positive and negative constructions in one sentence did not necessarily nullify each other, but rather added to this cumulative effect. Examples of softened negative comments from the preliminary study are given below:

It is advisable to include page numbers.. (suggested action):

I’m not sure you came to grips with Fraser’s article (personal comment) How do the concepts of age value and art value sit within such a culture? (interrogative)

Part of the problem may lie in the structure of the essay (qualifier) You could have given real examples to illustrate how the problem can be resolved (suggested alternatives)

The categories above were based on Read et. al. (2005) but with the addition of the ‘personal’ comment category, which arguably introduces an element of contingency and reader response to lower the force of the criticism. I also decided to add another re-coding stage for comments that communicated messages around argument and criticality, and this is dealt with below.

102

3.6.2 Argument and criticality in written comments

It is often the case that a poor mark is justified in reference to a lack of argument (Chanock, 2000; Ivanič, Clark, & Rimmershaw, 2000; Mutch, 2003) or a poor attempt to construct an argument and engage critically with an assignment. Clearly, comments on content may also link to issues of criticality, and developmental comments are often aimed at making the student aware of the need for better argument and critical engagement. Hyatt’s content related categories included argumentation but criticality could be the focus of comments within his structural category, as this comment suggests:

It has a rather brief superficial feel and I would have expected at this level a critical engagement. (p.345)

I decided to add a re-coding stage for criticality and argument (CA) references, but that comments which related to specific content points in an argument would not be considered as CA comments, rather I was only interested in comments that used the language of critical analysis. An example of such a comment would be:

…you were descriptive rather than analytical (Katy, formative 1)

In this way, I employed a narrower interpretation based on terms explicitly related to the discourse of critical analysis, e.g., argument, critique, critical, evidence, analysis,

description, depth of analysis. As discussed in Section 2.4.5. (Haggis, 2006), such

language relating to critical analysis is not transparent to students, particularly international students, so it can act as a barrier to understanding feedback. This coding was aimed at gauging the way explicit reference to issues of criticality and argument weakened the impact of written feedback. The framework that was adapted with the modifications suggested above is seen in Figure 3.1 below.

103

COMMENT TYPE EXAMPLE 1.Phatic

(maintaining social relationship)

There are a lot of encouraging signs here Katy.[P]

You have tackled a difficult question and made some thoughtful observations.[P]

2. a Developmental alternatives

(task focused- on gaps

related to current work)

b. Developmental future (process focused on closing

gaps in future work)

c. Reflective questions

d. Informational content

It would have been useful to bring to the fore the specific and fundamental issues that arise in the conservation of redundant buildings[SN]

The essay would have been improved by deeper and wider reading around the topic. [SN]

The candidate needs to think about writing in their own words…[SN] Quotes should be used sparingly and to illustrate points you making, not to make points for you.

It’s important to consider limitations, were these the only ones? [SN] How do the concepts of age value and art value sit within such a culture? [SN]

The one I recommend is Northedge’s Good Study Guide… 3. Structure

(at discourse level )

This introduction covers the main structural elements of aims, scope and sequence. [P]

4. Language and expression

(punctuation / lexis / sentence construction/ - grammar/ proofreading and spelling

Be careful with commas [SN]

Instructor is not a neutral term (lexis) (E)

This is not a complete sentence (grammar/ sentence construction) [N] Always proofread to check spellings [SN]

5. Referencing /source use (conventions/ quotation/

bibliography

This source does not appear in the bibliography [N]

The references in the text are broadly in line with departmental guidelines [P]

6. Presentation It is enough to use the same font size and type (i.e. no italicisation), but to indent from the left [SN]

The illustrations are a useful addition[P]

7. Register The style of writing needs to develop an academic flavour[SN] 8. Positive content The case studies are relevant to the discussion [P].

9. Negative content The second case study was less good….[SN]

The candidate does not explore the issues or the conflicts that can arise from a value-led system [N].

10. Non evaluative summary

The essay sets out the broad area of study and provides an overview of a major conservation project

10. Explicit justification of the mark

My mark signifies that this piece of work is a clear pass, but with room for significant improvements

Further Coding

Critical analysis [CA] There is a clear attempt to present a well-informed argument. The essay illustrates a rather descriptive view on the subject.

Explanation [E] The first case study was better because you talked about good and bad aspects of the restoration.

Positive [P] Negative [N] Softened Negative [SN]

See annotations added to comments in 1-10 above

104