Instrument trialling

The multidimensional nature of teaching is discussed in the sizeable body of literature on SETs (Marsh, 1987). One thing that becomes clear in the literature is that inappropriate or poorly-constructed items will generate poor-quality data: asking appropriate questions in suitable ways is crucial to the collection of useful data. The widespread use of Marsh’s SEEQ in higher education contexts brought a degree of confidence regarding the suitability and quality of the items used to this researcher. It should be noted, however, that, in an email to me dated 24/11/11, Marsh stated that he would have included more items in his instrument, had this been possible at the time of his initial work. With this in mind, the decision not to trial new items was a difficult one to take but one that was justified by the

109

need to develop a scale for use with younger students (16 - 18 years old), for whom a longer instrument may have led to fatigue or resistance.

Table 4.1 below shows summary statistics for those students and teachers involved in the feasibility study.

Teachers 10

Students 161

Average survey response time 6 minutes, 53 seconds

Year 12 students (%) 57%

Year 13 students (%) 43%

Table 4.1: Summary statistics for students involved in the feasibility study

Table 4.2 below shows the number of survey responses by subject:

Biology 30 Chemistry 70 English 60 ICT 30 Languages 14 Maths 22 Physics 1

110 Table 4.2: Numbers of feasibility study survey responses by subject

The student survey was administered online using a commercially-available software package called Cvent43 _{and the URL sent to the Assistant Head at the feasibility study}

school, who acted as the coordinator.

4.14 Feedback

The value of feedback has been well-documented (Askew, 2000; P. A. Cohen, 1980; Gregory et al., 2011; Hattie & Timperley, 2007; Kluger & DeNisi, 1996); work has also been done in higher education contexts to support the interpretation and understanding of student evaluation of teaching ratings as a means of generating improvement in the effectiveness of teaching (Dominick et al., 1997; Marsh & Roche, 1993; Penny & Coe, 2004; Rindermann, Kohler, & Meisenberg, 2007). The mechanism for feeding back evaluation information to the ten teachers involved in this feasibility study project was a one-to-one meeting with each teacher, in which I presented a class-level graph of the student evaluation of teaching data, an example of which is seen below in Figure 4.1 (the complete feedback sheet is found in Appendix 3c). In trialling this approach, I wanted to assume the role of ‘trusted colleague’ which would be so integral in the main study. While, obviously, I was not actually employed by the school, and thereby not a ‘colleague’ in the pure sense of the word, my background in teaching enabled me to approach this task appropriately.

111 Figure 4.1: Example of a feasibility study feedback graph

Where teachers had multiple classes involved in the feasibility study, a single graph showing data for all classes was presented, as illustrated in Figure 4.2 below.

Figure 4.2: Example of a feasibility study feedback graph showing all classes for a single teacher 4 4.5 4 4 4 4 4.5 4.5 4.5 1 2 3 4 5

Teacher 3: Class 13E

13E 4 4 4 3.5 4 3.5 4 4.5 4.5 4 4 4 3.5 4 3.5 4 4 4 4 4 3.5 3.5 4 3.5 4.5 4 4 4 4.5 4 4 4 4 4.5 4.5 4.5 1 2 3 4 5

Teacher 3 (all classes)

12A 13B 13D 13E

112

Each meeting was scheduled to last one hour and followed the following format: 1. How did you feel about having your students evaluate your teaching? 2. What do you think the data presented on your graph mean?

3. How do you feel about the evaluation data your class has given you? 4. What do you think these data mean?

5. What might you do with these data?

Comments that had been made by students during the survey were also presented to the teachers without editing. Each meeting was recorded and analysis of the transcript data is presented in Chapter 5.

The consultation meeting was designed around goal theory (Locke & Latham, 2002) and solution-focused brief therapy (SfBT) (Gingerich & Eisengart, 2000), greater discussion of which is presented in Chapter 6.

113

Chapter 5: Analysis of the feasibility

study findings

The feasibility project set out to gather data and present both analysis and interpretation of them in accord with the following objectives:

1. An exploration of the feasibility of A Level students evaluating their teachers, and an investigation of the students’ attitudes towards the evaluation process;

2. Gathering teachers’ opinions on the suitability of the instrument and its items for evaluating teaching at A Level, as well as their perceptions about being evaluated by their students;

3. Trialling the online survey package (Cvent);

4. Trialling a consultative feedback intervention session.

This chapter provides analysis and interpretation of the data collected, and provides discussion of them in light of the above aims. Furthermore, as this study adopted ostensibly a confirmatory approach, reliability data in the form of Cronbach’s alpha is presented below both for the instrument as a whole and for each subscale, concluding as it did that no item or scale should be dropped from the trialled version of the instrument when it came to the larger-scale main project trial.