Data collection procedures in the main study

Before describing in more detail how the data in the main study were collected, an overview of the data collection methods employed to answer the research questions is provided in Table 3.6.

Research questions Participants Data collection RQ1 What listening processes do ESL test-takers

engage in while performing (adapted) PTE Academic listening-to-summarize tasks? a. Are there any differences in the processes

and strategies when compared between tasks with different language modalities, namely speaking and writing, and performance levels? Group A (12 Participants) Stimulated recall data  Summary content Listening notes Perception questionnaire RQ2 What are ESL test-takers’ perceptions of

(adapted) PTE Academic listening-to- summarize tasks and task difficulty? a. What is the relationship between the

perceptions of listening task difficulty and task performance?

Group B (60 Participants) Perception questionnaire Task performance scores Table 3.6: Research questions and an overview of data collection methods

To answer the first research question and its sub-question, qualitative data on processes and strategies performed to complete the tasks were collected from a total of 12

108

participants (Group A). More precisely, the data comprised verbal report data, gathered by means of stimulated recalls, and language data, taken from notes written down by the participants while listening and summaries produced after listening. To answer the second set of research questions, concerning the perceptions of tasks and listening task difficulty and the relationships between the perceptions of listening task difficulty and test-takers’ performance, quantitative data were collected from a total of 60 participants who were not involved in the stimulated recalls. The data comprised responses to perception questionnaires and task performance scores.

Figure 3.3 gives an overall picture of how the tasks were delivered to the

participants and how the data were collected. As indicated in section 3.2, the participants were separated into two groups, A and B. In both groups, the procedure began with completing a background questionnaire and then completing four listening-to-summarize tasks on a one-to-one basis. After each task performance, the participants in Group A were asked to complete a perception questionnaire and then a stimulated recall, whereas in Group B only a perception questionnaire was administered.

109

The following explains the data collection step-by-step.

1) Completion of the background questionnaire

To begin with, a background questionnaire was administered to each participant. The questionnaire (see Appendix 2) consisting of 10 items aimed to collect bio-data on the participants’ first language, gender, educational background, overseas experience, and their English ability.

2) Completion of the listening-to-summarize tasks

Although indicated earlier that eight tasks were used in the study (see 3.4.2 ), each participant was asked to perform only four tasks with different listening passages. That is, Figure 3.3: An overall picture of the main study’s data collection procedures

110

they performed one of the task groups presented in Figure 3.4, Task Groups 1-4. The grouping of the tasks was based on the listening passages and the skills involved in task performance. To investigate whether different modalities (speaking and writing) affect listening comprehension processing (RQ 1a), it was important that each participant performed both listening-to-speak and listening-to-write tasks. However, to avoid that they heard each listening input more than once so that their experience of producing a summary in one modality would affect a second summary, each task had to be associated with a different listening input. Consequently, in each task group, no listening passage was used more than once. Task Groups 1 and 3, as shown in Figure 3.4, were made up of the same tasks but presented in a different order, and Task Groups 2 and 4 were also made up of the same tasks but presented in a different order. This was in order to minimize the potential effects of task sequencing on task performance. The difference between Task Groups 1+3 versus Task Groups 2+4 was the modality of each task, i.e. listening-to-speak tasks in Task Groups 1+3 were listening-to-write tasks in Task Groups 2+4 (and vice versa).

111

* 3participants from Group A and 15 participants from Group B **L-S = Listening-to-speak tasks & ***L-W = Listening-to-write tasks

Figure 3.4: Task delivery and data collection

Before completing the four actual tasks, the participants were asked to do two sample tasks (one listening-to-speak and one listening-to-write), in order to familiarize themselves with the item type and to reduce test anxiety. Participants were assigned to a Task Group in such a manner that each Task Group included participants with different levels of language ability (as based on their self-reported IELTS scores).

112

The perception questionnaire was administered immediately after the participants completed each task. Thus, since the participants performed four different listening-to- summarize tasks, each participant completed the perception questionnaire four times (two times for listening-to-speak and two times for listening-to-write tasks).

4) Participation in stimulated recall (for Group A’s Participants)

In Group A, where 12 students participated, a stimulated recall was conducted immediately after the participant completed each task and the perception questionnaire. As each participant was asked to perform four tasks, four stimulated recalls were obtained from each person. The participants were offered a choice of using English and/or Thai for the stimulated recalls. This was to compensate for the impact of language ability on expressing thoughts or opinions. As all the participants used Thai, translations are given in the quotes in the findings chapters (4-6). Prior to the actual data collection, the participants were provided with an opportunity to practice the stimulated recalls on the sample tasks.

For the purpose of stimulated recalls, the 12 participants from whom verbal protocol data were collected were video-recorded whilst completing the tasks. Immediately after they completed each task, they were asked to watch their video- recording and encouraged to describe what they were thinking about or paying attention to while listening to the texts. Similar to the pilot study procedure, the recording was played at least twice in order to obtain insightful data, depending on the participants’ abilities to report their own thoughts. In the first round, the participants were in control of pausing and explained whatever they wanted to share. In the second round, the researcher

113

stopped the video recording systematically at the same points for all participants and asked the participants to explain their thought processes. For a couple of participants, the video was played three to four times if they appeared to be quiet during the first and second laps of video playing.