9. Pilot Study
9.6. Feedback
9.6.8. Teacher Comments
I also welcomed the feedback by the teacher. I was not present in the administration process and therefore I requested analytic feedback by the teacher who assisted with the pilot study. Even though it might be argued that in the first administration, I should be presented, I firmly believe that the pilot study should follow the exact same process as the actual study. ‘As Oppenheim remarks, everything about the questionnaire should be piloted; nothing should be excluded, not even the type face or the quality of the paper’ (Cohen, Manion & Morrison, 2007). For this reason, despite the fact that the
178
pilot study showed that adjustments should have been made in the assessment forms, the same administration process was followed exactly as the trial. I decided to post the assessment forms in order to follow the exact same process as the actual study.
In the teacher sheet for each form there was a question which asked the teacher whether there were words that the students did not know. In spite of having previously examined the readability scores, this questioned aimed to further explore to what extent the carrier language was appropriate for this age group. Carrier language is the language which is used to set the task (Chartered Institute of Educational Assessors, 2008). In other words, carrier language is the question which is set, and it looks for the answer. If the question is not explicitly set, then the answer is more difficult to be given. In high readability items the variable which is measured is not only the mathematical ability, but also other variables, such as the reading ability (Hewitt & Homan, 2003). The phrase that the students did not know according to the teacher comment was only one (F5); ‘take for granted’. This means that the Readability test used to reassure the appropriateness of the language according to the age of the student was successful.
After the administration process, I contacted the teacher to ask further feedback. She kindly responded to a short questionnaire I sent to her. The questions in the questionnaire aimed to cover important functions of the pilot study. The feedback form completed by the teacher revealed that there were no problems during the administration and the students enjoyed the assessments. The teacher explained that the assessments took place during the last two hours of the school day and the students were tired. However, the students yet had enough time to complete the assessments, which took them approximately 15 minutes. Also the teachers said that the students “understood most instructions. They had difficulty in understanding the sentence ‘Take for granted that what is said in the box is true and try to reach the correct conclusion”’. This was the first time they had to do an activity of this kind and they kept thinking of alternatives to the scenario or imagining subtext, which altered the ‘take for granted’ instruction.
The teacher also referred to the instructions for the administrator: ‘I thought that the instructions to the administrator were too long and sometimes unnecessarily complicated’. This was a particularly interesting feedback, because it would not be considered that the problematic part could be the administration instruction. Initially, I chose to have more complicated instructions, but reassure that the process in all the
179
schools will be standardised. The aim of the analytical instructions was the exclusion of any potential ambiguity and vagueness in the process because it is crucial for all the schools to follow accurately the same process. Nevertheless, the feedback sent was extremely significant. Teachers are usually busy, and it is crucial to provide them with simple instructions to follow. For this reason, I decided to get further feedback for the language and the possible wordiness of the instructions. I asked two external people to judge the guides and I rephrased the instructions to make them simpler.
9.7. Chapter Summary
To sum up, based on the pilot there were slight changes made at the measurement tools. Based on the functions that Cohen, Manion and Morrison (2007, p. 341-342) suggested, this pilot study achieved its goals. Particularly:
It ensured that the questions were suitable for the students’ experience. For this reason, the measurement tools were piloted in a group similar with the targeted group.
It led to the practicing of the coding system of data analysis. As an additional benefit of the pilot study was the improvement of the grading system for the second creativity activity.
It provided data for distractor analysis. For each item, I checked how many students answered each of the three options A, B and C and therefore I considered whether there is a specific distractor who confused the students, because it was too believable or tricky.
It suggested that some items might have low discrimination.
It confirmed the clarity of the instructions and the items. Comments were provided by the students and the teacher and led to the reduction of the vagueness or difficulties in wording. In order to reassure this, I asked the teacher to right on the answer sheet the wording for which the students asked clarification. Furthermore, the students provided their own feedback written on the forms. There was just one phrase that was judged problematic ‘taken for granted’ and it was decided to be replaced by ‘what you read is definitely true’.
It gave the opportunity to receive comments on the type of questions and its format. The students did not have problems with the format of the questions. However, the lack of a thinking cloud on the last page made a student wonder and therefore I
180
decided to also include a cloud also in the last page. Unexpectedly, in this category I had the feedback regarding the administration guide. The instructions were judged as too complicated. This led me to the decision to redesign the administration guide by simplifying it but ensuring the standardised process.
It provided a realistic image for the appropriateness of the questions. There was no missing data in this case. However, the missing data, as it has been explained before, could have been revealing concerning the item difficulty. Furthermore, based on the teacher comments the questions were appropriate for the majority of the students. However, the teacher clarified that the SEN students found difficult to reply to the complicated thinking problems.
It highlighted omissions or irrelevant information in the forms. The students’ comments revealed the omissions in some problems and there were phrases that were amended.
It provided feedback on the attractiveness and appearance of the questionnaire. I received positive feedback by the teacher regarding the reaction of the students for the appearance of the questionnaires. For this reason, I decided not to make any changes in the appearance.
It revealed how much time the questionnaire requires to be completed and whether it is too extensive or too short. I decided not to be present in order to pilot it in the exact same way as it will be implemented in the actual study. Therefore, the classroom teacher gave me feedback on the time which is needed for the forms to be completed. The allocated time was 30 minutes. However, the students needed 10-20 minutes to complete it. Additionally, the students commented that the time was too long. When the questionnaire needed less time than expected to be completed, there were two possible options. The one would be to add some questions in the questionnaire. The other was to change the suggested time from 30 to 20 minutes. Concerning the first option, I included a second problem-solving question. I recognized that the ability of the students to solve problems might be context- dependent. Thus, I included a second problem to enable the students to demonstrate their problem-solving skill in two different contexts. The addition of more items was not justified when the utility of the assessment was concerned. Secondly, the test demands already by the students to think. I thought that adding more thinking problems will be unreasonably demanding by the students. I rejected the second
181
option, because I thought that maybe students in a different cohort might work on a different pace. It would be better for teachers and students to have the pleasant surprise of finishing earlier, rather than having students who are rushed to finish because of the limited time. Moreover, the specific measurement tool does not aim to measure the thinking speed of the students and hence the time is not a factor.
182