Reflections on the Pilot Study

The purpose of the pilot study was to test out the data collection methods that I would be using in the main study, and to find out whether these data collection methods enabled me to collect data which answered my research questions. In the pilot study, the research question which I sought to answer was ‘how do higher education (HE) students engage with the teaching materials provided?’. Admittedly, it was still a very broad research question, and therefore after the pilot study, I narrowed down the scope of the thesis and was able to figure out a more specific research question for the study, which is “how do learners use resources in their repertoire to make meaning when learning Chinese in an online, self-directed context?”.

As regards the data collection methods, a combination of screen recording, thinking-aloud, learning diary, structured and semi-structured interview worked well together as they complemented each other. However, from the point of view of participants, the technical aspect of the study might have been an obstacle.

The use of Camtasia was manageable. The difficult part had been the file transfer, which took an hour to complete each time. In the main study, this was still an issue. Participants recalled spending hours trying to upload the videos. This was unavoidable due to the large file size of the recording. In the main study, instead of using Dropbox, I used the UCL Dropbox

The use of the thinking-aloud technique was successful in the pilot study. As I watched the recording, I felt that I existed in the same space with Anne and that she was directly talking to me. It was as if I was in the same room observing her in real time. The use of a learning diary was meant to be a tool to help Anne reflect on her learning experience after each lesson that she took. The use of a learning diary, in the case of Anne, did not work well as most of what she wrote was the same as what she said in the ‘thinking-aloud’. This was also a problem that she pointed out a few times in the semi-structured interview. After

reflection, in the main study I decided to make the learning diary an optional component as it did not provide extra information, and yet it added to the workload of the participants.

In the pilot study I used the stimulated recall technique to clarify specific events occurred in the recording. I thought it would be a useful tool to help Anne recall what she did in the lesson. However, after reviewing the literature, I realised that it would be problematic in the main study when I was going to have weekly interviews with the learners because the accuracy of the recall would decrease as there would be a longer than expected interval between the event and the recall (Bloom, 1954; Yinger, 1986; Cotton and Gresty, 2006). Therefore, I had decided not to use the stimulated recall technique in the main study.

In terms of the number of interviews, initially it was planned that three interviews would be conducted during the four-week study. One before the study began, one in the middle of the study, and one at the end of the study. However, it was found from the pilot study that three interviews were not necessary, and this was cut down to two in the main study. It has to be clarified that as the pilot study and the main stage of data collection were not linear, separated events, the reflections made from the pilot study were operationalised gradually in the main stage of data collection. As a result, I still had conducted three interviews with the participants that I recruited in the early stage of the main study.

Every care was taken to minimise my effect in the screen recording process. I asked Anne to install Camtasia and recorded her lessons at home so she could

do her lessons in a natural setting. This strategy worked well because I gave minimal instructions on what she should do, and I stayed out of the process as much as possible, giving Anne a lot of autonomy on what she wanted to do. However, on a practical level, it has to be admitted that my absence in the process did not help much in terms of putting Anne at ease and enabling her to behave naturally. She mentioned in the semi-structured interview that “I knew that you were there, so I was talking to you”. It is clear that whatever she did, she was aware that I was watching, although not in real time, and that she would adjust her behaviour because of this. In practice, it is not entirely natural behaviour, and so a slight distortion of data was inevitable. The ‘observer’s paradox’ is discussed in greater detail in section 4.9.1.

One of the major changes that I made after the pilot study was instead of doing face-to face-interviews, I opened up the study so that people could participate through Skype. Although the pilot study was conducted face-to-face, I soon realised that it could also work in an online context. I could use the same format with people participating through Skype. It was a major breakthrough for me because at that point I had difficulty finding enough participants who I could meet face-to-face. This challenge is elaborated in section 4.9.2 of this Chapter.

The pilot study also helped me discover some of the practical issues that I had to be aware of when carrying out the main study. For example, I realised that more explicit instructions had to be given to participants on how to use

Camtasia to record the screen and the face, and therefore in the main study I created a simple guide to help participants with the software (see Appendix 2). Also, I sent weekly reminders to participants to keep them on track, and to allow them a chance to ask any questions regarding what they had to do and to help them solve technical problems. This is particularly important when the

participants were located in different parts of the world. In the pilot study, I realised that when Anne pointed at the screen with her finger, it was not

captured in the camera, as the camera could only capture her face, but it could not capture Anne’s finger pointing at the screen. Therefore in the main study, I encouraged participants instead of pointing at the screen with their fingers or using deictic expressions, it would be better to also use the cursor to indicate

the part that they were referring to because the pointing action and the deictic expressions lost meaning if they are not captured in the screen recording.

Most importantly, after the pilot study, I decided to narrow down the number of focal OLLPs from three to one. This was informed by the results of the pilot study, my observation of the three platforms, as well as practical issues in terms of recruiting enough volunteers. As mentioned in the previous section, the reflections from the pilot study were operationalised gradually, and therefore data related to the other platforms were still collected at the early stage. The chosen OLLP, Memrise, allowed learners more flexibility to use their linguistic and semiotic resources creatively, and it had more users than the other two platforms, Livemocha and ChinesePod. Also, I had only two participants from those two platforms who volunteered to join the study. Their data was collected but it was later on discarded, and thus it was not reported here. In fact, I

discovered at a later stage that Livemocha was no longer in operation. Those were the reasons why I decided to change the focus of the thesis, from

comparing how three platforms are designed, to focusing on how learners learn Chinese using different resources in one platform.

4.5.7 Fieldwork –Chinese Classes for staff at Institute of Education (IOE)