Analyses of feedback comments

A relatively small number of studies has focused on the analysis of feedback comments themselves (Brown & Glover, 2006; Read, Francis & Robson, 2005; Hyatt, 2005; Leki, 2006; Mirador, 2000; Mutch, 2003; Walker, 2009; Weaver, 2006; Yelland, 2011). Two studies have taken up the idea of written feedback

74

Mirador’s earlier twelve ‘moves’ and explored them in some depth. As it was published after data collection for both of my studies, this work did not inform the design of the research.

Yelland applied Mirador’s approach to 140 feedback comments on undergraduate English Studies assignments and concluded that they had a

“predictive power” (p.223). He argued for the addition of a new move, Mitigating

Negative Comment (MNC) based on his argument that making negative comments

without harming the student was “...the most difficult task in writing feedback comments” (p.225). The latter point highlights the emotional impact of negative feedback comments, a theme that has been taken up in more recent work (Molloy et al., 2013). Arguing that feedback comments are indeed a specific ‘genre’, Yelland sought to discover the extent to which first and third year undergraduates were able to mirror the genre in their own feedback on other students’ work. He found that both groups rarely used the ‘standard’ approach he had identified, and although students frequently attempted to mitigate negative comments at sentence level, they did follow the approach he had identified as a feedback genre in tutor feedback. Yelland’s conclusion that markers more clearly share the same discourse community with their external examiners than their students is worth taking up in the present study. The current research did not focus on written feedback in terms of its status as a genre, however, though a study connecting feedback and the development of CAW might be expected to provide insights relevant to the topic.

In the earliest study of relevance in the UK context, Ivanič et al. (2000) analysed a sample of nine sets of written feedback comments from EAP tutors and subject tutors. The sample was rather small, and did not claim to be representative, but it is unclear how it was selected. The study used six categories to analyse the feedback in the typology listed below:

1. Explain the grade in terms of strengths and weaknesses; 2. Correct or edit the student’s work;

3. Evaluate the match between the student’s essay and an ‘ideal’ answer; 4. Engage in dialogue with the student;

5. Give advice which will be useful in writing the next essay; 6. Give advice on rewriting the essay.

75

Ivanič gave no rationale to justify this six-point classification system and it is clear that these categories are not mutually exclusive: explaining grades in terms of strengths and weaknesses is arguably the main intention behind many tutor

comments, and a good match between a student answer and an ‘ideal’ answer, for example, would constitute a ‘strength’. It could also be argued that ‘justifying the mark’ is a basic function of written feedback, and it implicitly encompasses many of the comment categories here, with the possible exception of the two ‘advice’

categories. Thus, there may be serious issues of overlap here. On the other hand, it may be useful to know when tutors explicitly refer to ‘the mark’ in their wordings, as this is closely bound up with summative evaluation and judgement.

In their findings, Ivanič et al. highlighted the salience of the function of justifying the mark, and indicated the problem of usability resulting from vague, general comments. The low frequency of developmental comments usable for future assignments was also noted. These points were supported by later studies (Brown & Glover, 2006; Walker, 2009)

A study by Mutch (2003) provided an example of a large-scale study of written feedback comments in an undergraduate modular programme within a Business school in a post-1992 University. The Mutch study took a sample of over one hundred feedback sheets from eleven degrees at different undergraduate levels. Broad positive / negative categories were further divided and included comments which had ‘developmental content’. Mutch found that many comments were

categorical, noting the often ‘terse’ and pared down style of comments as well as the issue of mitigated comments and how these may not always be interpreted as

intended (p.31). He observed, however, that tersely stated comments should be understood in terms of the conditions of production, as feedback is required to do a lot in a reduced space, which in turn suggested alternative methods of delivery rather than avoiding direct comments. Mutch also identified a final category as

‘conversation’ where the tutor seemed to be musing in response to the student’s work, offering a personal reaction which did not imply developmental action.

Mutch found an overall balance of positive and negative comments, but commented on variation in the amount of feedback provided by individual markers,

76

observing that this was mainly due to individual practice and not related to the material or grade. He also found that there was a significant element of what he termed “implied development” comments requiring interpretation on the part of the reader, e.g. “evidence of some good basic sources (implies a need for more

sophisticated sources), or “too much of your word count was on descriptive content” (p. 32), which implied a need to replace some description with more analysis.

While Mutch’s ‘developmental’ comments focused clearly on feeding forward to the student, the notion of implied skills development seems problematic; it has value in recognising the force of what is left unsaid, but may be inappropriate in a study of international students. Any message within feedback can only be effective if it is picked up, understood and acted upon by the student. International students are likely to be more challenged than home students in respect of

understanding such messages, and ‘implied’ messages could be interpreted in different ways by the researcher.

In a later study, Hyatt (2005) carried out an analysis of a corpus of 60 feedback scripts from Master’s courses in Education in a university in the north of England. The study did not specify the nature of the participants and did not

differentiate between home and international students. As with the Ivanič and Mutch studies, it was also limited to an analysis of feedback taken out of context of specific programmes or modules of study. Hyatt’s classification system, however, appeared to be useful for the purposes of this doctoral study. His categories were:

1. Phatic (maintaining social relationship)

2. Developmental (helping the student with subsequent work related to the current assignment)

3. Structural, (organisation of whole assignment or sections within it) 4. Stylistic (punctuation, lexis, syntax/grammar; proof -reading/ spelling

referencing, presentation, register

5. Content related (positive, negative, non-evaluative) 6. Methodological (for assignments based on research) 7. Admin (relating to administrative issues)

(Hyatt, 2005, p.344)

Hyatt’s main categories were further subdivided to allow for considerable detail in the analysis. Like Mutch, Hyatt also included a ‘developmental’ category, arguably central for ‘feed forward’, but he extended this with subdivisions of

77

focused feedback similar to the example from Mutch above. He also included a separate ‘future development’ category and two further developmental sub- categories: ‘reflective questions’ and ‘informational content’ (p. 344), covering Mutch’s tutor musing or ‘conversation’ category. This was understandable perhaps in relation to his principal concern, to explore the extent to which written feedback was dialogic in its approach. Hyatt’s main finding was that there was a lack of comments engaging students in disciplinary dialogue, with only occasional evidence of tutors engaging with definitions of academic discourse such as argument,

criticality etc. With the exception of Hyatt’s ‘methodology’ category, which would only be useful in relation to Master’s level assignments featuring research projects, his overall listing of content, development and stylistic features provided useful, relevant and comprehensive coverage in the context of the present study on assignments on taught Master’s programmes.

Read, Francis and Robson (2005) carried out a study with fifteen male and sixteen female lecturers from History departments across a range of twenty-four UK universities in order to investigate gender differences in marking and feedback. The lecturers marked and gave feedback on one male and one female authored

undergraduate history essay and were also interviewed on their responses and marking. While the study found few gender differences in terms of the nature and pattern of the comments, it found wide variations in the judgement of quality and marks awarded. The authors attempted to analyse the tone and feeling of the feedback, focusing on negative and positive comments, finding four times more negative than positive comments. Following an earlier idea by Ivanič, they differentiated wholly positive or negative comments from what they termed

“softened negative comments”, or comments that were less authoritative and offered a more open ended tone, e.g., “In parts you tend to slip into description”-softened by a qualifier, or “Try to plan your essay more logically”-softened by suggesting action in the future” (p. 255). This approach recognised the limitations of a simplistic positive / negative approach to comments, providing a way of accounting for mitigating comments, and it will be referred to later in Section 3.6.1.

Leki (2006) carried out a study in a US university, based on interviews with twenty one L2 graduates from different disciplines, (thirteen PhD and eight Master’s students) and feedback on fourteen of their texts. Her analysis of feedback appeared

78

to focus mainly on in-text marginal comments made on a variety of text genres including lab reports, research reports and exams. Leki used a number of general response categories which included “language and writing; evaluative comments; grades; task management; substantive response”. She further categorised the latter under categories such as “correcting interpretation, “requesting elaboration” or “requesting clarification”. This approach to analysis may have reflected the fact that her sample consisted of mainly marginal comments on texts.

One of Leki’s main findings was the problem of illegibility of tutor writing, which seriously hampered understanding for three of her twenty-one participants. Handwriting issues were also raised in some UK studies (e.g., Higgins et al., 2001; Hulme & Forshaw, 2006; Robinson et al., 2013). Higgins, for example, found that forty per cent of interviewees in his study complained of illegible handwriting in their feedback. Students in Leki’s study also claimed to be attentive to feedback and all wanted more, even in some cases where Leki observed that they were “drowning in feedback” (p.279). More than half of her respondents wanted more feedback that gave direction and guidance. Praise alone was not always valued, with a call for more guidance to accompany it. Surprisingly few students, given the number at PhD level, referred to iterative feedback which allowed revision of drafts, though those that did clearly valued it. In her final recommendations, Leki called for collaboration between writing experts and disciplinary tutors in providing EAP courses to support such students.

Although it was set in a large US university, covering a wide range of disciplines, with participants from a number of different countries, the study is still relevant to my own research. It is worth noting that the study did not focus

exclusively on taught Master’s students, and her data collection focused on a varied sample of texts from a range of courses and modules, with interviews exploring participants’ experience and attitudes to feedback generally, but not in the context of any specific assessment events. Its focus on L2 graduate writers, however, and the implications she draws for EAP support for such students makes it particularly pertinent.

79

Brown and Glover (2006) studied feedback from science undergraduate courses at Sheffield Hallam University and the Open University. In a large scale study they examined one randomly chosen assignment and associated feedback for 147 students, focusing on both marginal and end comments on a range of assignment types including lab reports, research projects and essays. In their analysis, Brown and Glover assigned comments to general categories such as content, skills

development, motivational feedback, or future study, but also used three categories to explore depth of feedback. Category one was assigned to comments that indicated errors or omissions, category two was for advice on how to correct, or actual

corrections, while category three was for explanations of why answers were appropriate or inappropriate.

Over half of all comments in the Brown and Glover study were found to be content focused, and where skills issues were indicated they were rarely given explanations that would feed forward. One finding was that the amount of feedback did not correlate with the grade, i.e. that lower grades received more feedback, and in fact similar amounts of feedback were found at different grades, but with large variations for each grade. They found an overall lack of depth in comments. A major conclusion was that there was a tendency in both institutions to provide summative feedback regardless of the assessment, and that timing of feedback on final products meant that it was not used to improve future work. They also found a lack of shared understanding around assessment criteria leading to student misunderstanding, a finding echoed in later studies (Walker, 2009; Weaver, 2006) and commented on the large amount of mark loss focused rather than learning focused comments,

comments justifying a grade and looking back to what has been achieved, not forward to future learning (Ivanič, Clark, & Rimmershaw, 2000; Weaver, 2006).

In a later study, Walker (2009) used the Brown and Glover’s analytical framework described above along with questionnaire and telephone interviews within a faculty of technology in the Open University. She attempted to link analysis of written comments with student responses to them, taking a sample of 106

summative assignments with accompanying feedback sheets and interviewing 43 students by telephone. Walker drew on Hattie and Timperley’s (2007) notion of retrospective and future gap altering comments on skills that could be used in future

80

tasks. Students in her study reported not understanding over a quarter of comments, fifty per cent of which were comments on content. Unsurprisingly, students

mentioned that comments explaining points were more usable, and sixty four percent of skills comments were deemed usable for future work. This is not surprising as skills can be applied to subsequent work, while content usually refers to a completed task which may not be repeated. The Walker study, like that of Brown and Glover, was based on a variety of assignment types in a scientific discipline, so was less relevant for the purposes of this study and did not provide analysis of feedback and students’ perceptions of its usability over a prolonged period of time.

The studies reported above, particularly those in the UK context, focused mainly on samples of undergraduate feedback, often in science disciplines and rarely provided data from the tutors who wrote the feedback. Where student reactions to feedback were further investigated, it was usually in the form of survey or one-off interviews. Their findings, however, converge on a similar set of points summarised by Nicol (2010) in his survey of the field. They are that feedback should be:

 Understandable  Selective  Specific  Timely  Contextualised  Non-judgemental  Balanced  Forward looking  Transferable  Personal (p. 513).

Many of these points also concur with a report setting out ten principles20 by the National Union of Students (Porter, 2009 as cited in Burke & Pieterick, 2010).

Ferguson’s recent study (2011) in an Australian university provides relevant findings on student views of feedback. Ferguson surveyed 101 undergraduate students, and more importantly, 465 graduate students studying teacher education at a major Australian university. He used a pen and paper questionnaire to explore research questions focused on student preferences for feedback and their perceptions

20 _{Although worded in different ways, the principles agree on all points above, but also add that} feedback should be “continuous”, and “for learning, not just of learning” (p.82).

81

of its usefulness. The questionnaire was administered in the latter half of three undergraduate and one post-graduate pre-service education programme. He reported a high degree of consistency across the sample which prompted his claim that “...regardless of discipline, background and the nature of university experience, students had considerable agreement about what constituted quality assessment feedback and process” (p.54). Ferguson’s findings concur on most of these ‘qualities’ of feedback listed above, emphasising the personal, guiding and motivational aspects, and he concluded that:

The most important factor in ‘good’ feedback was a clear link between assessment tasks and guidelines, assessment frameworks and criteria and the feedback offered (p.60).

Where claims are made that students acted upon comments (Walker, 2009; Weaver, 2006) they should be treated with caution, however, as they are based on student self-report only. At the same time, these studies rarely make clear the nature of the assessment, so one is often left to assume that they were based on summative feedback reports. What is evident is that studies on feedback quality and on student perceptions of feedback quality were often framed by an expectation of formative feedback in situations where summative feedback practices predominated. One line of research this suggested was to explore the extent to which the claims made for Ferguson’s study held true in my own context. That is, did the findings reported above (based mainly on native speaker undergraduates), compare with international taught Master’s students perceptions of feedback in my university context?