Trials and crosschecking

A working draft of the observation schedule was created using all input from the advisors and classroom reflections. It was then iteratively used and refined over ten classroom trials across four schools. Factors used to select trial classrooms included: the schools’ socio-economic status and ethnic composition (to match those of the study schools); teacher skill and experience (to match those of study teachers); and the existing professional relationships between the researcher and the trial teachers.

The importance of incorporating the classroom trials in the development process is best illustrated through examples of the rationale for alterations to the observation schedule and the consequential changes (Table 3.3). The rationale for each change and suitable modifications were deliberated through an informal process including discussions between the researcher and cultural advisors and between the researcher and other teacher educators and education researchers across the trial phase.

Table 3.3: Modifications to the observation schedule as a result of classroom trials Rationale for modifying the

observation schedule

Additions to the schedule (Appendix 3)

To be able to record further detail regarding teacher care for students’ mathematics progress

one-to-one teacher-student interaction (mathematics content)

acknowledgement of students’ mathematical success

persistently maintains students’ focus on lesson To explore possible links between

teacher care and class engagement

recording the proportion of the class highly engaged at four roughly equally spaced points through the lesson

recording whether or not the class was engaged to (apparent) teacher expectations

To include a further measure of teacher care for individual students

one-to-one teacher-student interaction

(unrelated to the lesson’s mathematics content) To include a measure of the quality of

teacher-student relationships

student-initiated interactions with the teacher (e.g., use of humour, asking questions) Completed observation schedules for

different lessons for the same teacher could vary greatly depending on the nature of the lesson and learning experiences.

a brief overview of the topic and style of the lesson

The classroom trials also led to practical data gathering decisions important for the eventual consistency and quality of the study data. For example, decisions regarding the start and end time for the observations could be considered, trialled, and made. Some trial teachers required students to line up outside the classroom prior to the lesson and therefore for these teachers the first interactions with students were conducted outside the classroom. For others, the first interactions were as the students arrived into the classroom from their previous lessons. The trials showed that despite the importance of capturing the initial teacher-student interactions, data gathering outside the classroom before the lesson was comparatively intrusive and therefore less likely to generate representative data. For consistency in approach across all study observations, it was decided the data collection period would begin when students first entered the classroom.

Similarly, it became apparent that data collected during the minutes in the classroom after the lesson had concluded were also important as it is often in this time that students initiate informal conversations with the teacher, offer farewells, questions, comments, and use humour. Therefore each observation continued until all students had left the classroom. Exceptions to this pattern were made when teachers held students back for disciplinary discussions at which times student privacy was given precedence over data collection.

Other examples of changing the schedule through the classroom trial phase which allowed further specificity in data to be obtained included: recording acknowledgement of ‘effort’ as well as of ‘mathematics success’; recording whether students’ questions were about or unrelated to mathematics; and indicating the level of application of observation aspects (Appendix 3).

Very few events were observed during classroom trials in the area of teacher care for students as culturally located individuals and hence rather disappointingly there was little opportunity to revise this section of the schedule. This difficulty highlights the importance of ensuring trial classrooms can provide the necessary environments to inform all aspects of the schedule.

Trialing of the draft student and teacher questionnaires and interviews was also carried out within the six months immediately preceding initial data collection. The general themes and specific questions of the questionnaires and interviews (Appendices 4-11) were drawn from theory described in Chapter Two. Questions adapted from those used

within the ‘Tripod Project’8 (which explores similar themes in the area of teaching care for students as individuals) were included in questionnaires where suitable.

The first student questionnaire (Appendix 4) and all student interview questions (Appendix 10) were trialed with Year 9 classes at a school similar to the study schools. As trialing was carried out late in the school year, Year 9 students were chosen rather than those then finishing Year 10, as they were closer to the stage the study students would be at the beginning of data collection. An iterative process of trial and refine was used. In each iteration one entire class completed the questionnaires; two individual students from the class filled out their questionnaires voicing their thinking to the researcher as they responded, in order that the researcher could enhance the content and face validity of the questions (S. Fontaine, personal communication, 16 May, 2005); and the same two students then responded to the interview questions. This process was repeated twice more with different classes within the same school over a two week period. The main changes made to the questionnaires and interviews were clarifying wording of questions and including further options in questions using lists.

Teacher questionnaires (Appendices 7-9) and interviews (Appendix 11) were trialled with teachers geographically distant to the study in order to avoid exposure of the study teachers to the questions before the study began. Again, questionnaire responses were voiced to the researcher allowing issues of clarity and relevance to be identified. Again three cycles were used to ensure the questionnaires and interview questions were fit for the purpose.

After all trials were complete, all data gathering tools were discussed and refined using the trial results and the advice of the cultural advisors and mathematics education colleagues. A complete crosscheck was carried out to ensure that relevant data for all of the key aspects of the study were being gathered across each of the three data collection methods (Appendix 12). As a result of the crosscheck further questions were included in the third student questionnaire to ensure a fuller set of data could be gathered on students’ perceptions of barriers to their learning and of their teachers’ use of real life contexts. Consistency of coverage was confirmed through the crosscheck process providing confidence that the study had a sound methodological triangulation foundation (Denzin, 1985). All student questionnaires were then translated into Māori and Samoan (Appendices 4-6) enabling students proficient in those languages to select whether to use the

8

translated or the English-language questionnaires. To maximise the integrity of the translations, the translators were nominated as experts by the respective cultural advisors and were fluent speakers of te reo Māori and Samoan respectively. A further check on the suitability of the translations was provided by the cultural advisors.

Further additions were made to the second and third sets of questionnaires and interviews during the first year of the study in response to data gathered earlier in the study, further reading, and consultation with cultural advisors. The observation schedule and the questionnaires remained constant for the second year of the study to facilitate comparison of data across both years.

3.2.3 Discussion

The evidence of changes made to the data gathering tools due to all development stages as described in Section 3.2 indicates the importance of using a range of ways to inform the development of rigorous data gathering tools that can be used with confidence for multicultural classroom contexts, and indicates that development by theory alone or through a combination of theory and trials may not provide tools fit for the purpose. Every aspect of the development process contributed to the final form of the observation schedule, questionnaires, and interviews thus enabling further detail of a broader range of relevant data to be captured including aspects less well documented in the literature. Use of such a process is important in order to enhance the credibility of findings particularly amongst teachers and researchers involved with, and of, indigenous and minority cultures, often the very groups performing less well within traditional achievement measures and therefore those for which we most need suitable research data, analysis, and findings.

Rigorous development of data gathering tools can place considerable demands on timeframes and budgets. However, quality research is dependent on quality data and the use of a comprehensive process to ensure data gathering tools allow the full scope of relevant data to be captured and to sensitise the researcher/s to the research context is essential. For small or large-scale classroom-based investigations in multicultural contexts attention must clearly be paid to seeking and integrating stakeholders’ (in this instance, students, teachers, and cultural experts) skills and knowledge.