Questionnaire Used with the Evaluators

This section presents the questionnaire that was used to survey the team of evaluators with regard to evaluating the formative feedback comments generated by the tool. The three categories of quality, relevance and coverage that the team were invited to rate the comments against are introduced and defined. The Likert scale adopted in the evaluation is specified.

The team of evaluators were presented with a set of comments and asked to rate them. The comments came from a sample that included both those that were generated by the team of expert markers and those that were generated by the tool. This meant identifying suitable criteria against which the comments would be rated. This is a substantial problem because, for example, giving students detailed and comprehensive feedback may not be a good policy as a long list of issues that need to be addressed may not always be read and acted upon. There is a need to strike a balance between issues that represent a misunderstanding of the main learning outcomes being assessed and those that are relatively minor or tangential to the aims of the assignment brief.

Fourteen evaluative statements were designed against which evaluators would be asked to rate the comments. The statements were derived from considering three broad criteria of :

• Quality • _Relevance

• _Coverage

Quality is concerned with the extent to which the comments adequately described the item being fed back upon and the extent to which it provided support to the students for their learning.

Relevance is concerned with, for the student submission as a whole, the priority given to feeding back on one particular issue at the expense or even omission of another. For example, it is possible to imagine that good quality feedback that is focused in one area of the submission is at the expense of generating feedback of a similar quality in another. The issue of relevancy applies when this second area relates to a fundamental error in the student submission or is crucial to supporting the student’s learning.

Coverage is concerned with ensuring that feedback comments are generated across the spectrum of all issues of relevance contained in the student submission.

Each team member was presented with 14 evaluative statements. The team member was asked to judge the feedback comments against these statements utilising a Likert scale. The Likert scale adopted was a 5-point named scale for all three sets of statements. Five-point was chosen because of its reliability over scales with fewer points (McKelvie 1978). It also contained a mid-point to minimise results that may be misleading (Matel and Jacoby 1972). The same five point scale was used for all statements to ensure consistency of responses by the team (Norvell 1977). The names of the scale were similar to those observed by Goeb et

• Strongly Agree

• _Agree

• _{Neither Agree nor Disagree}

• _Disagree

• _{Strongly Disagree}

The 14 statements that each team member was asked to consider for each set of feedback comments they were asked to review were:-

Quality

1) The majority of comments contained in this set are clear. 2) The majority of comments contained in this set are concise.

3) The set of comments provide sufficient detail in order for a student to know what concept or issue is being fed back upon.

4) The set of comments contained in this set provide sufficient detail in order for a student to know what further work they need to undertake.

5) The set of comments will help the student with his/her learning.

Relevance

6) The comments contained in this set are relevant for this type of assignment brief and the associated indicative learning outcomes.

7) The comments contained in this set address important areas of strength found in the student submission that is considered to be of significance. 8) The comments contained in this set address important areas of weakness

found in the student submission that is considered to be of significance. 9) It is clear which concepts the comments in this set are addressing. 10) The comments in this set will help the student with his/her learning.

Coverage

11) This set of comments, when viewed in its entirety, fully encapsulates all pertinent feedback needed for the student to recognise where there are areas of strength in the submission.

12) This set of comments, when viewed in its entirety, fully encapsulates all pertinent feedback needed for the student to recognise where there are areas of weakness in the submission and where further learning is required.

13) This set of comments would provide a useful enhancement to the type of comments that I gave during stage 1 of this evaluation.

14) This set of comments would have been sufficient to replace the type of comments that I gave during stage 1 of this evaluation.

Each question was designed to be as unambiguous as possible, covering a single idea, in an attempt to ensure consistent interpretation across all members of the evaluative team. The questions were comprehensively reviewed by a group of academic colleagues and a set of guidance notes were produced and sent to each member of the team (see Appendix C).

This section presented the questionnaire that was used to survey the team of evaluators with regard to evaluating the tool- and human-generated formative feedback comments. The 14 statements that the team were asked to rate the comments against were introduced and their grouping into the three categories of quality, relevance and coverage was highlighted. The questionnaire that was sent to the team of evaluators was discussed and the Likert scale adopted in the evaluation was specified. Appendix C presents the questionnaire together with the explanatory notes given to the team of evaluators.