CHAPTER 3 RESEARCH METHODOLOGY
3.3 Research Instrument Construction for Data Collection
This section describes the research instruments used for data collection, in three phases, namely a pre-lesson phase, a while-lesson observation phase, and a post- lesson phase. This section provides a discussion on the question of validity in qualitative inquiry instruments and explains how I promoted the validity of research tools.
3.3.1 Instrument Construction Procedures
Prior to data collection, I applied various practical procedures to construct research instruments. Figure 3.2 below illustrates the procedure for instrument construction.
Figure 3.2: Instrument construction procedures
I started the process by reviewing various methods of lecturer cognition research to understand possible instruments. Later, I focused on the instruments that positively met my research aims. For the next activity, I evaluated relevant research instruments based on the criteria: appropriateness, practicality, suitability and feasibility. These instruments had to be useable in the field, and they must suit the participants to yield data. Also, I had to ensure that I had sound competence in the use of these instruments. Next, I constructed research instruments based on the nature of particular instruments. Moving to the next activity, my supervisors evaluated all research instruments and gave feedback on certain points for further development. Full details regarding validation are given in Section 3.8.1. The next activity was a pilot study, in which I tested the constructed instruments with volunteers (see Section 3.2.3 for more details). I finalised the instruments prior to data collection.
3.3.2 Pilot Study
Piloting is commonly used with all types of research designs to enhance the legitimacy and trustworthiness of the data-gathering instruments (Morrel & Carroll, 2010). Piloting is a valuable tool in the study as a pilot study can stimulate foresight prior to actual use. Thus, Bell and Waters (2014) recommend that all data collection tools ought to be piloted. They also clarify that a pilot study aims to trial how long
Instrument review
Critical appraisal
Selection Elaboration
Validitation by experts
each tool takes informants to complete; to check that all questions and instruments are well-defined; to remove the bugs from the tools; to confirm that the responses address the intent of the questions; to address any problems or confusion; to eliminate any items that do not yield useable data; and to add new questions that might be related. Ideally, researchers should trial proposed data-gathering tools with pilot volunteers similar to those that will form the population of the actual study, but if, in practice, that is impossible, they can pilot with the available resources and volunteers (Bell & Waters, 2014; Morrel & Carroll, 2010). The researchers need to trial the actual methods and tools with the pilot volunteers so that the genuine participants will not encounter difficulties while giving data in the main study.
There were two phases to the pilot studies of research instruments. The first phase was in New Zealand. I piloted semi-structured interview schedules with seven Thai students (three PhD and four MA). For piloting an observation check-list, due to limited relevant classroom resources, I tested the check-list by employing CLIL lessons on Thai Teachers TV and YouTube. This channel provided lessons that used MOI in Thai and English, as there were no available resources of this type in New Zealand. Figure 3.3 displays an original resource used in testing the check-list.
Figure 3.3: CLIL lesson
I selected this CLIL lesson because a teacher had English proficiency, but students lacked English skills. I also used other CLIL lessons on YouTube in order to pilot my instrument with different teaching practices. For example, I focused on how the teachers gave lectures and structured activities (kobwit piriyawat, 2011) and classroom interactions. In regard to piloting schedule/topics for focus group discussions, I piloted the schedules and topics with the same group of Thai students. Afterwards I developed all research instruments based on comments from the volunteers and my assessment made during the pilot study.
In this pilot study, I gained more confidence in using the research instruments and operating data collection in the actual research field. Although initial research instruments had been piloted under available resources in New Zealand, I was aware that the piloted instruments still needed to be adjusted to suit local practice in the field.
The second phase was in Thailand prior to a data collection stage. I used the piloted instruments from the first phase with EMI lecturer populations, who were not the main participants. Figure 3.4 presents pilot study processes of semi-structured interview schedules.
Figure 3.4: Pilot study processes
In step 3, I paid attention to the Thai language used in the instruments due to EMI lecturers’ preference. They agreed that using Thai could help them express themselves more easily and deeply in terms of critical points, as Thai is their first language. I was able to understand that they preferred topic discussions rather than questions and answers. During the interview, the EMI lecturers preferred ‘going with the flow’ rather than ‘fixing a sequence of the topics.’ Having a chance to listen to their perspective, I was also able to add new topics to be discussed with actual lecturer participants. In step 4, I had incorporated core interview topics into the final products of the interview schedules. These schedules were not fixed and might change upon particular circumstances during actual data collection.
Piloting interview schedules with volunteers in NZ