Pilot Study

Pilot study refers to a small-scale trial of a study conducted before the full-scale trial (Gay et al., 2011). The primary purpose of piloting is to ensure that the research instruments could be used effectively in the actual study (Bryman, 2012). In addition, piloting helps to easily identify items that contain ambiguity or are unlikely to be of interest within the study as they might not form a variable (Bryman, 2012). Therefore, it is essential that the instruments are piloted in order to establish content validity (Creswell, 2014). For the present study, the pilot study involved administering the questionnaire, the speaking test and the interview, after Ethical Approval was obtained (see Section 3.9) and informed consents were gained from the participants.

109 3.7.1 Questionnaire

The questionnaire was piloted several months before the actual study. The participants were selected through convenience sampling to enable a higher response rate (Bryman, 2012). Cohen et al. (2018) suggested that gathering sufficiently large-scale data allows the researcher to calculate “reliability levels (alphas), item difficulty and item discriminability, and identify commonly misunderstood or non-completed items” (p. 583) more easily. Therefore, 115 students were recruited for piloting the questionnaire. These students were asked to answer the questionnaire via the Google Forms application; the link to the questionnaire was sent through their social media such as Facebook and Twitter. To maximise the acceptability of the questionnaire, reliability testing was used. Reliability refers to the degree of consistency of what is being measured (Gay et al., 2011). Reliability was assessed by measuring internal consistency using SPSS version 24 to calculate Cronbach’ Alpha Coefficient. Cohen et al. (2018) suggested that a scale is considered reliable and acceptable when α > .70. The reliability test result showed that the questionnaire was reliable and appeared to have good internal consistency, α = .905. All the items were worthy of retention and no changes were made after piloting. Although the greatest increase in α would come from deleting item 3 (α = .907), and item 14, 16 and 17 (α = .906), removal of these items would increase the alpha only by .002 or .001. This increase is not dramatic and the items were considered crucial to be asked in the questionnaire. Therefore, the alpha value reflected a reasonable degree of reliability.

3.7.2 Speaking Test

Two sets of the speaking test which consisted of two questions were adapted from the previous MUET speaking test. The two sets were obtained directly from an English teacher who had experience as both MUET teacher and examiner for more than 10 years. Although the previous MUET speaking test had been vetted by experts, piloting was considered necessary for this

110 study as some of the items were altered to suit it. The two sets were Set A, self-reported or personally related to students’ experience (e.g. how to overcome tiredness and which is the best way overcome tiredness) and Set B, where questions were more factual and cognitive demanding (e.g. major causes of climate change and how to overcome it, and reasons children should be banned from gadgets). However, both sets were allocated the same time length and were the same type of decision-making tasks in which students had to describe and justify their decision. Both sets were piloted by the researcher’s colleague in Malaysia on four ESL undergraduate students. Their speaking tests were audio-recorded and sent to the researcher to be analysed. Students were also asked to give feedback on the two sets of speaking test. Students were able to comprehend and answer both items. However, the major concerns were the length of the test and the language produced. In set A, the results showed that students were able to produce approximately 350-450 tokens. For Set B, however, the results showed that students produced approximately 200-300 tokens with a high production of disfluency. As for the students’ feedback, the results showed that both sets were comprehensible and they understood each question clearly. However, students mentioned that the time length given for each test was insufficient especially in Set B where the questions were more factual, critical, demanding and required a high level of general knowledge. The present study investigates the effects of two different forms of language teaching (TBLT and TSLT) on students’ speaking performance rather than the effects of task complexity, hence Set A was selected for the pre- and post-speaking test rather than Set B. Moreover, Set A also provided richer data of a sufficient amount to be analysed.

3.7.3 Semi-structured Interview

The semi-structured interview was piloted with three undergraduate ESL students by the researcher himself a month prior to the actual study. During the interview, students were informed about the

111

interview and that it would be audio-recorded. The interview lasted approximately 40-45 minutes. The researcher then listened to the audio and transcribed the interview. A few questions were found to be ambiguous which led students to hesitate and to answer repetitively which resulted the interview to be lengthy. Therefore, the researcher decided to disregard these questions. Questions that prompted students to elaborate their answers were kept. Only a few questions needed to be reworded as they were found to be unclear to the students.

3.7.4 Speaking Intervention

The intervention was piloted to evaluate its suitability and practicality before the actual study. It worth noting that only one lesson was piloted due to limited time available and also the researcher’s availability. One teacher, who is the researcher’s colleague, agreed to conduct the pilot study. A copy of one of the lesson plans and also feedback forms for students and the teacher were given via email to the teacher to be employed in a Malaysian university setting. Before the pilot study, the researcher explained the procedure to the teacher and requested that the classroom be video recorded. Through convenience sampling, 24 ESL undergraduate engineering/science students who were enrolled in the FoEL course were selected to participate in the pilot study. These students were current students in one of the classes taught by the teacher herself. The researcher received the video-recoding and feedbacks from both students and teacher two weeks later. Based on the observation from the video, students reacted positively towards doing the speaking tasks. The speaking tasks were conducted in groups of four. Students were able to participate and interact according to the demands of the task. Following the observation, the majority of the students gave positive feedback towards the speaking tasks. However, the teacher commented that pair work was not conducted frequently due to limited time and group work was considered sufficient. In addition, the teacher commented that due to time constraints, it was impossible to do individual presentations.

112 Accordingly, the results of the pilot study indicated that the lesson was feasible and workable with pair/group work discussion and group presentation only.