Research Methods - Questionnaires to year 7

Cycle 2 Questionnaires to year 7 - 11 students

4.7 Research Methods

4.7.1 Questionnaires

Questionnaires are a useful tool to obtain responses from a large sample (Cohen et al 2018, Newby 2010, Gillham 2007). Views were gathered from 287 students, so

questionnaires were considered the most effective method for this sample size. Using a questionnaire enabled a large amount of data to be gathered quickly (Cohen et al 2018, Menter et al 2011) which could be analysed using statistical techniques. Opie (2004)

advised questionnaires must be designed carefully so their appearance is inviting, and the instructions clear to the engage the participants.

Year 8 questionnaire. A blank copy of the questionnaire can be seen in Appendix 1. The aim of the questionnaire was to gather the views of students in year 8 about the

feedback they received from teachers and peers. Whilst interviews are usually an effective way of eliciting participant opinions and views, the large cohort size prevented this approach. Cohen et al (2018) and Newby (2010) warned limitations of

questionnaires can lead to insufficient data collected due to unsuitable design, participants not completing them in full, no option for the respondent to ask for

clarification about the questions and researcher bias in the design of the questionnaire.

To minimise the possibility of these problems, the questionnaire was structured with closed questions, which did not allow the respondent leeway in personalising their input (Newby 2010) but did ensure that it was quick and easy to complete (Gillham 2007).

The questions required students to choose and tick relevant responses, which avoided putting too much strain on the respondents as this can lead to poor quality responses (Cohen et al 2018). Newby (2010) warned this approach can introduce bias as the categories are based on the researcher’s perceptions, which may not be representative of the students’ views. However, the questions were based on literature about

assessment and feedback, particularly Black et al (2003)’s research and the Assessment Reform Group’s principles of Assessment for Learning (ARG 2002). This ensured the question content was not based on my pre-conceptions, but on previous research. The questionnaire was structured to aid the analysis of large quantities of data (Cohen et al 2018, Newby 2010). Open response boxes were added where respondents could add their own views if they felt they were not adequately captured, although these were rarely used.

Planning and piloting the questionnaire helped to ensure it was designed to obtain the data required and was straightforward to complete. Section 4.7.1 (i) explains the process of piloting the questionnaire. The questionnaire was short, only two sides of paper forming two parts, one on each page. The paper was double-sided to prevent two pages becoming detached (Newby 2010). Cohen et al (2018) advised providing

instructions for completion where they are needed rather than all at once, so information was given at the start of each part rather than all at the beginning. The first part of the questionnaire required participants to tick selected responses, the second part required students to rate the aspects of feedback they had received. These responses required a greater level of thought than the first part and was considered better to be placed second. Cohen et al (2018) and Newby (2010) advised that questionnaires should start with straightforward questions with more thought-provoking ones later as this engages

the respondents and encourages their commitment in completing the questionnaire. Only a few of the returned questionnaires were partially filled out, suggesting the participants found them simple to complete throughout.

The first part of the questionnaire sought to establish the form in which students believed feedback most commonly occurred (written in exercise books or worksheets, verbal through questions or as a grade or report), why teachers gave feedback; what was useful about feedback and what would make it more useful. The questions were designed with multiple choice responses with discrete categories (Cohen et al 2007), students were able to select more than one. By providing categories for students to select, I was imposing my view of the choices on the students, rather than leaving them free to provide their own response, thus it could be considered there was a bias in the selection of the response criteria. However, due to the large number of students participating, for ease of analysis and to ensure that the questionnaire was quick to complete, multiple choice questions were believed the best way to capture this data.

Students were also asked if in general they liked to receive feedback, this question was dichotomous with only two options (yes/no). This was chosen to force students to make a choice to gauge their overall view (Cohen et al 2007). When the questionnaire was completed during the English/Law survey, a small proportion of students chose to add a category ‘sometimes’.

The second part of the questionnaire sought to capture student views on written, verbal and peer feedback in eight of their subjects. These are the main areas of feedback determined by the Assessment Reform Group (Gardner 2006). Specifically, the

questionnaire sought to gain student views on whether they received feedback on their achievement (known in the school as WWW: What Went Well); on development points (known as EBI: Even Better If); peer feedback; whether written and verbal feedback was explained, was easy to understand, was useful and if they had opportunities to use it.

The subjects included in the survey were: Maths, English, Science, Geography, History, French, Technology and Art. Technology was comprised of: Graphics, Resistant Materials and Food Technology in which there were a range of different teachers. These subjects were all compulsory in KS3. They were chosen in agreement with the Head Teacher as she felt these subjects would represent a range of different pedagogical approaches across the school.

Students were asked to give a response to the questions, for each of the eight subjects, using a Likert scale. A four-point scale was used to avoid students selecting a middle option. The four points were: Always, Mostly, Sometimes, Never. This range was felt to be sufficient to capture the potential variations in views. A Likert scale was used as this is an existing robust method (Newby 2010). Normally five points are used in a Likert

scale, but I chose to use four so respondents were forced to have an opinion (Gillham 2007). Newby (2010) argued not having a mid-point does not affect the integrity of the measuring device, however, Cohen et al (2018) claimed forcing respondents to have an opinion may make them have a view on something they do not have, and they should be given the opportunity to be neutral. In this research, by seeking students’ opinions of the feedback they received, I believed they would already have an opinion through their lived experience, so a four-point scale was suitable. Gillham (2007) argued there are weaknesses in using scaled responses as participants rarely use the whole scale, they tend to lean more towards the positive stance, even if this is not their true belief and the responses do not give reasoning behind their opinion. Despite these limitations, I

believed a four-point scale with the chosen categories was sufficient to capture an overview of the students’ views and kept the process of completing and analysing the questionnaire simple.

The ordering of the questions was important, so they were meaningful and easy to understand (Cohen et al 2018). The first questions focused on gaining the students’

perceptions of what constituted feedback. This was intended to get them thinking about feedback and its purpose before moving onto evaluative questions. Similar questions were grouped together to avoid the questionnaire being confusing for respondents.

Cohen at el (2018) advised researchers to ensure all questions are relevant to all participants, unless it is clear if they do not need to answer some. In the year 10 questionnaire, not all the subjects applied to all students as some were options in KS4 (Geography, History, Art, Technology, French). I chose to include these subjects in the questionnaire to maintain continuity from the Year 8 questionnaire, however some students commented on their irrelevance. It had not been made clear on the questionnaire to only answer applicable questions.

4.7.1. (i) Piloting: When preparing the questionnaire, a pre-pilot (Gillham 2007) was undertaken by having a discussion with two year 9 students to explore their views of feedback and determine how the data could be collected from a large cohort of year 8 students. Through the dialogue with these two students, I identified the types of words they used and clarified them to ensure my understanding was the same as theirs. This terminology was used in the questionnaires ensuring it was age appropriate for the target cohort (year 8). Words used on the questionnaire can affect the responses (Newby 2010), so it was important to use familiar words that the participants would understand (Cohen et al 2018). To achieve this, the potential questions were discussed with the two students to ensure they made sense and could capture the information intended. The questionnaire was then created and then piloted by four students from the

target cohort to check the clarity, validity and to gain views on the response system (Cohen et al 2007).

The purpose of piloting a questionnaire was to check on the instructions and layout, any ambiguity of wording, how well it could be understood by the participants, the types of questions and format, and particularly, the rating scale (Cohen et al 2018). There were two males and two females in the pilot. These students did not participate in the final survey. Piloting enabled me to ascertain the ease with which students could complete the questionnaires, the length of time it would take and the clarity of the instructions and questions. This was particularly important as I was not present when the

questionnaires were distributed. As a result of this process, I established the

questionnaire took between five and ten minutes to complete. The students said the instructions and questions were clear with the exception of one question which was ambiguous and one which was repetitive. I discussed suitable re-wording with the students and removed the repetitive question. They felt the questionnaire was simple to complete and looked appealing. Preparing and piloting the questionnaire in this way enable it to be refined and ensured it was an effective tool to collect the data required.

This gave it construct validity as I had confidence it would measure what was intended (Yin 2018, McNiff 2017, Swanborn 2010).

The students who participated in the planning were willing to co-operate and unlikely to represent a full cohort. This potentially failed to ensure that the questions were worded so all students could understand them, thereby losing the voice of pupils without the linguistic capital to express their voice (Arnot and Reay 2007, Noyes 2006). To overcome this, the questionnaire was kept simple to complete (Gillham 2007). Out of all the

questionnaires returned, only 3% were spoiled.

Table 6 explains the process of the questionnaire design. It details the stages, how it was undertaken and the timeframe.

Stage Action Time

Questionnaire Preparation Research literature on the topic to develop understanding to establish the focus of the research and suitable lines of questioning

January – June 2015

Familiarisation with research methods to

determine the most suitable methods. May – June 2015 Questionnaire Design Consultation of literature on

questionnaire design to ensure the tool was robust and effective

June 2015

Pre-pilot

Piloting

Discussion with two year nine pupils about their views of feedback to

determine appropriate question type and terminology

June 2015

Four year 8 students piloted the questionnaire to determine the ease of completion and suitability

July 2015

Amendments made to the questionnaire July 2015 Distribution Distributed blank copies of the

questionnaire to the school for completion

July 2015

Collected completed questionnaires for

analysis July 2015

Table 6: The process of questionnaire design

4.7.1 (ii) Year 8 Questionnaire distribution: Paper versions of the questionnaires were distributed to the students. These were administered by a gatekeeper, who was a teacher in the school, rather than the researcher. The attitude of the gatekeeper could have introduced a bias by influencing the student responses and therefore affecting the student view (Arnot and Reay 2007). However, having an unknown person administer the questionnaire may have been uncomfortable for the students, so a familiar member of staff managing this process was preferred. The absence of the researcher prevented an opportunity for any clarification regarding the tool to be made (Cohen et al 2018). To overcome this, clear instructions were given to the gatekeeper regarding the completion of the questionnaire to ensure that it gathered student views as accurately as possible, although her attitude could have determined the seriousness of its completion (Cohen et al 2007).

4.7.1(iii) English/Law Questionnaires: Following a request from Kirstie to be involved in the research, I provided her with a copy of the original Year 8 questionnaire to gain her views on the suitability of its use for her students. We agreed it would be appropriate to use the same questionnaire with the removal of the range of subjects in part 2. A copy of this amended questionnaire can be seen in Appendix 2. Kirstie distributed the questionnaires, which were completed during her lessons.

4.7.1 (iv) Year 10 questionnaires: The original Year 8 questionnaire was adapted to reflect changes in the school and issues emerging from earlier research which I wanted to capture when students were in year 10. However, it was important to keep some similarities with the original questionnaire to explore the views of students as they progressed through the school. To keep the questionnaire short and simple (Newby 2010, Wellington 2015) some original questions were omitted. Table 7 explains the changes made to this questionnaire. A copy of the Year 10 questionnaire can be seen in Appendix 3.

Table 7: Changes made from year 8 questionnaire to year 10 questionnaire

Year 8 questionnaire Year 10 questionnaire Reason for change

I think that feedback is:

• Written in exercise books

• Written on worksheets

• A grade

• Teachers asking questions

• Teachers answering questions

• Constant monitoring report

This question was removed The question was removed as it was considered the students’ views of what feedback was had been established in the original questionnaire. The removal enabled additional questions to be added.

Why do you think teachers give you feedback?

• To give me a working grade

The terminology was changed from

‘grade’ to ‘level’:

To show what level I am

This reflects the change in the terminology adopted by the school, which used the terms levels rather than grade by December 2016.

What do you find useful about feedback? No change This question was considered to be relevant to year 10

What would make your feedback more

useful? This question was removed This question was removed as it had been established in the original questionnaire. The removal enabled additional questions to be added.

What part of feedback do you value the most?

This question was added due to the change in

Government policy in 2014 to drive schools away from focusing on grades and levels to more qualitative feedback (Department for Education 2014). The students’

views regarding levels and comments were sought in this survey.

Generally, do you like to receive

feedback on your work? This question was removed Question was removed as their views had been established in original questionnaire. The removal enabled additional questions to be added.

What types of feedback do you get in class on your learning?

Question was changed due to a strong focus in the school about increasing the provision of feedback since 2015.

The rewording of the question sought to establish if the feedback was having a positive difference in learning, rather than being just provided.

Is written feedback explained to you? This question was removed These questions were removed as they were addressed in the question below and their removal enabled additional questions to be added.

Is verbal feedback explained clearly to

you? This question was removed

Is your feedback easy to understand? Feedback is easy to understand Is the feedback that you get useful to

you?

Feedback helps me to think about my own work

The students’ views gained through the interviews referred to feedback being a mechanism to think about their work. This was important to capture from the year 10 students. In the preliminary analysis (Chapter 5), the term ‘useful’ was shown to be confusing and open to interpretation, so it was avoided in the Year 10 survey.

Do you have opportunities to use written

feedback to improve your work? These 2 questions were combined to: Feedback helps me to improve my work

It was considered this was a simpler way to collect this information and kept the number of questions to a minimum.

Do you have opportunities to use verbal feedback to improve your work?

I understand the success criteria in

my subjects Teacher and student interviews as well as published literature highlighted the importance of students understanding success criteria (Ruiz-Primo and Brookhart 2018, Swaffield 2011, Hattie and Timperley 2007, Harlen 2007 Black et al 2003), I wanted to ascertain the students’

understanding of this across the eight subjects.

I am confident in assessing my own work to identify strengths and areas for improvement

As peer assessment was not regarded as useful in the Year 8 questionnaires, this question was added to identify the students’ confidence in self-assessment.

Self-assessment is an important aspect of Assessment for Learning and can support peer assessment (Berggren 2014, Lawson 2013, Willis 2011, Black et al 2003). It was also raised by the students in the Art interviews.

4.7.2 Interviews

Interviews can provide the researcher with new knowledge about the respondents’

experiences through an interaction between them and the interviewer (Kvale 2007).

Interviews were chosen to elicit more information from a small sample of students with regards to aspects of teacher and peer feedback. Student interview data was intended to supplement the questionnaires by enhancing the understanding of the social actions and process (Menter et al 2011) regarding the issue explored. The interviews had two

purposes: to gain student views specifically in one subject (Art) to help the teacher develop her practice; and to establish student views about the feedback provided by another teacher (English/Law) to review her practice and identify what could be learned from her approach. Kvale (2007 pg.11) explained interviews ‘allow the subjects to convey to others their situation from their own perspective and in their own words’. The perspective from every student could not be captured, so a sample of students were chosen. It is acknowledged these cannot be assumed to be representative of the whole student cohort.

In addition, interviews undertaken with teaching staff were designed to ascertain how the student perceptions had been used both at a local level (subject teachers) and at a school level (SLT and DHT). They also provided an opportunity for the respondents to discuss and reflect on the data and identify any further action to be taken. The

interviews enabled aspects that emerged from the questionnaire data to be explored in depth and provided a greater understanding of the student experience (Kvale 2007).

4.7.2 (i) Planning the interviews

Prior to the student interviews, I met with the subject teacher and we planned the focus of the interviews. Initial analysis of the Year 8 questionnaires indicated there could be difference in opinions in male and female students, so it was decided to interview them separately to allow for a different line of discussion to emerge, if necessary. The focus of the interview depended on what the teacher wanted to achieve, but the purpose was to build on knowledge gained from the questionnaires. To obtain views from a range of students, the teachers identified individuals with varying attainment grades and invited

In document Feedback for learning: exploring the views of secondary school students about feedback and how they can shape classroom practice (Page 93-106)