Overview of data collection methods

This section will introduce the range of data collected and methods used to address the research questions. Table 3.2 provides an overview of data collection methods, with each method explored more fully below. For more detail on each item and links to secure electronic folders of the raw data, see Appendices 4A, 5A and 6A. School B data was collected over a longer period of time because the focus for this case was longitudinal change over time (RQ3), whilst for School A the focus was on the school’s practice of

‘formative to summative’ (RQ2). Using a range of data from each school provided a range of lenses through which to explore the case, together with supporting the triangulation of data, as discussed further in Section 3.5.

Table 3.2 Overview of data collection methods

PSQM database School A June 2013 – June 2015 School B March 2013 – June 2016 Key data for RQ Documentation E.g. school documents collected on visits: policies, lesson plans, records, work samples, PSQM submission. 91 assessment reflections Items=91 6 school visits 1 PSQM submission Items=44 6 school visits 2 PSQM submissions Items=58 RQ1 RQ3(SchB) Non-participant observation

E.g. Lesson observation or observation of meeting/presentation - 4 lessons 1 meeting 1 presentation Items=9 3 lessons 1 meeting 2 presentations Items=11 RQ2 RQ3(SchB) Semi-structured (researcher-led) discussions or meetings E.g. interview/meeting, group discussion - 2 interviews 4 gp discussions Items=6 3 interviews 4 gp discussions Items=4 RQ2 RQ3(SchB) Written tasks (researcher-led) E.g. completion of

questionnaire, sorting activity, pyramid self-evaluation on TAPS development days.

- 6 development days Items=8 8 development days Items=13 RQ2 RQ3(SchB)

70

3.4.2 Documentary extracts from PSQM database

Extracts from the Primary Science Quality Mark (PSQM) database were selected to address RQ1 since they could provide information regarding assessment practices from across England. Round 4 submissions were the most recent submissions at the time, with schools working towards the Quality Mark in the year April 2012 to March 2013, and writing the final reflections in March 2013. Each teacher reflection consisted of around 200-400 words and described: practice within the school, changes across the PSQM year, their impact and possible next steps. The C2 reflection related to assessment practice so this was

downloaded from the PSQM server and anonymised for each of the 91 English schools in Round 4. The rest of the submission would have provided a range of data regarding

practices across the school, but it was only C2 which was purely focused on assessment, and so it was this reflection which was analysed.

All participating PSQM schools were informed that submissions may be used anonymously for research purposes and the most recent PSQM applications (Round 4) from all English schools (N=91) received an additional email regarding this study, providing them with the option to withdraw their data. Since the PSQM database was a pre-existing dataset, once access had been gained to the website, then data collection merely involved downloading the C2 reflections. Collection of data for the school case studies was much more varied and this will be the focus for the rest of this section.

3.4.3 Documentation and written tasks within case studies

Documentation is the written record of an event or process, on paper or electronically. These documents are relatively easy to collect, but may be selective in their representation of the context (Cohen et al. 2011: 236), together with only providing insight into the output rather than the process of construction. Nevertheless, the wide range of documentation noted in Table 3.2 is utilised in schools and can provide part of a rich bank of evidence. Collection of documentation largely took place on school visits. At the beginning of the TAPS project schools were given a list of assessment samples to provide, for example, policies, pupil assessments and tracking grids (for the full list see Appendix 3A). Photocopies or

71 photos of teacher planning and pupil work were also collected at each lesson observation. The documents were used to support discussions with the class teachers, and then scanned for inclusion in the electronic folder for each school. The aim was to collect a wide range of documents over time; although it should be noted that these would always be documents which were supplied by the SL or class teacher. This self-selection could provide a different picture to typical practice, for example, planning for lesson observations could be more detailed and pupil work may have been marked in a different way. However, this provides information about what the teachers think of as ‘best practice’, providing for another line of enquiry regarding teacher perception.

Both Schools A and B completed their own PSQM submissions during the case study period and this provided a wealth of data: action planning, SL reflections, CPD logs and a portfolio of evidence (in the form of a powerpoint). These documents were produced, largely by the SL, for the purpose of gaining an award, so they are again likely to represent what the SL viewed as ‘best practice’.

Documentation was also produced as part of the TAPS project, including an application to join the project, together with a range of group and individual written tasks from the project development days which supplemented the school-led data above. For example, on TAPS development day 1 the teachers were asked to record individually what they understood by formative and summative assessment (Appendix 3B) and then to complete a group card sort of assessment strategies (Appendix 6D). These researcher-led tasks directed the participants to explore the language of assessment in a way that the school-led documents did not.

In addition, focused questioning in the form of short questionnaires was also used later in the project, to support the teachers to reflect on whether assessment practices had changed (Appendix 3C). Such questionnaires provided a range of largely open-ended questions in order that teachers had the freedom to explain their thoughts rather than assign them to a pre-determined category (Oppenheim 1992), since the aim was for in- depth study rather than comparisons across the population.

72

3.4.4 Observations within case studies

Observation within this research involved watching teachers: lead science lessons, take part in school staff meetings or present examples of their practice to a teacher audience. It was important to see assessment practice in action, whilst recognising that the act of

observation will have an impact on the situation being observed (Bassey 1999). The practice will also be seen through one person’s perspective; there is no such thing as an objective observer. All observation is subjective; the observer reconstructs their observations to create their own interpretation of the context (Greene 2010). However, post-lesson discussions with teachers were held to support the development and sharing of interpretations of the observations.

In order to examine practice in action, a number of lessons were observed. This provided the opportunity to explore in-class processes, together with the relationship between reported and actual assessment behaviours, the ideal and the real (Angrosino 2012). The observations were largely non-participant in a seat at the back or the side of the room, however, there were some interactions with teacher and pupils when either invited me to take a closer look or join the activity. Observation both ‘disturbs’, ‘shapes’ and is ‘shaped by’ what is observed (Lincoln and Guba (1985: 98). The presence of an observer would affect the lesson whether non-participant or not, with perhaps both teacher and pupil attempting to present the ‘ideal’ lesson since I had been introduced as a visitor looking at science, however, the aim was not really to judge the typicality of practice, it was more about trying to describe what is possible in assessment.

Observation is more than just looking, the gathering of ‘live’ data can be both systematic and selective (Cohen et al. 2011). The decision was made not to use recording equipment since this would be much more intrusive and would accentuate the feeling of ‘being watched’ rather than building a quality relationship between the ‘knower and the known’ (Lincoln and Guba 1985) in the collaborative nature of research promoted by the DBR approach. Field notes are necessarily selective, with only part of the lesson being recorded. However, in order to provide a more complete picture, field notes were supplemented by lesson documentation (planning and pupil work) together with pre and post lesson

73 discussions with the teacher. The way teachers used assessment was the key line of enquiry, thus there was a focus on the practice of the teacher during each lesson observation.

Open field notes were taken during the lesson observations. This was supported by an observation schedule completed after the lesson (Appendix 3D) which contained

assessment features from Harlen (2013). The assessment features were included in the observation schedule in order to ‘test’ the categories in practice, as part of the DBR process. Many of the Harlen (2013) assessment features became the basis for the details within TAPS pyramid layers (Davies et al. 2014). The observation began descriptively, then became more focused and selective (Flick 2009), by organising the notes at the end of the lesson with the developing TAPS pyramid categories, for example, teacher elicitation, observation or discussion (Harlen 2013). Making descriptive notes on the lesson events was not a

completely inductive approach because the TAPS pyramid categories were in mind, but by recording the chronology of the lesson before categorising, I hoped to reduce the bias or ‘expectancy effect’ inherent in looking for behaviour to match categories (Newby 2010, Cohen et al. 2011). This also meant that for each lesson there were two sources of observer data: chronological field notes and lesson events mapped onto the TAPS categories. The observation schedule developed over the course of the project, as the TAPS pyramid

evolved; for example, by March 2014 the pyramid had an additional pupil layer (Short 2014), and so these categories were added to the observation schedule (Appendix 3E).

3.4.5 Discussions and interviews within case studies

A key method for exploring the relationship between formative and summative assessment was talking to participants. Discussions with teachers took place on development days and on school visits, ranging from a brief pre or post lesson conversation to a more formal semi- structured interview. Where possible the interviews were audio recorded, but the majority of discussions were informal and in public places, for example, in the staff room or at the reception desk in the case of one head teacher, and so field notes were taken during many of the discussions. For all interactions it was essential to build a rapport with the

participants, a trust that would be conducive to open discussion and sharing of ideas,

experiences and thoughts (Mears 2012). This required open listening, but also a recognition that these interactions are social encounters, the ‘inter-view’ as an interchange of views

74 (Kvale and Brinkmann 2009). Such social encounters involve interaction on both sides, thus I found that the conversation flowed more easily when there was not a strict script to follow or an audiotape being used. Semi-structured interviews are designed to be flexible, with key open-ended questions aimed at supporting the participant to speak freely (Cohen et al. 2011) and additional follow up questions planned but not necessarily used. This was fit for my purpose because I was looking to stimulate in-depth responses, rich data to understand a case (Newby 2010), rather than comparison of answers for particular questions across the population (Oppenheim 1992).

The informal discussions were related to issues arising from the lesson observation or other activity. Abrahams and Millar (2008) suggest that a combination of observation and

interview enhances ecological validity because the interviewee is likely to be more anchored to reality rather than ‘rhetoric’ if the interviewer has observed the practice. This perhaps suggests that there is likely to be a closer match between espoused and enacted practice, but it also may be that in conceptualising their practice, the teachers are able to draw upon a shared experience of an observed lesson to exemplify and explain their approach to the interviewer.

The questions for the semi-structured interviews were compiled from the developing theories as part of the DBR process: initially using the layers from the Nuffield model (2012) for the first school visits in November 2013 (Appendix 3F) and later using elements from the developing TAPS pyramid. The final SL interview for School B explored experiences of the TAPS project (Appendix 3G) and was carried out by a different researcher so that the SL could speak to someone other than their link tutor, with whom they had been closely involved for three years. This meant that the SL needed to explain their assessment approaches afresh to a new person, this both guarded against SL concerns about repeating themselves, since data given to their link tutor may have reached ‘saturation point’

(Angrosino 2012), and could provide a point of triangulation as they explained from scratch.

A large amount of data, using a range of methods was collected, an essential ingredient of a case study as a basis for exploration of the significant features of the case (Bassey 1999). Whilst all the data helped to create a rich picture of each case, not all of it was directly

75 relevant to this study of the relationship between formative and summative assessment. Therefore, during the data analysis phase of the research, coding focused on the RQs, with some of the later lesson observations for example, receiving less attention because at that time the TAPS project focus had moved onto the development of assessment activities.