After-Project Questionnaires

The last data collection instrument employed in this investigation is after-project questionnaires. I used this type of questionnaire with a qualitative intention in mind. This means that its purpose was, in some cases, to help develop other instruments; while in others it was to substantiate the explanatory power of other qualitative data (see Riazi & Candlin, 2014, for a discussion of the complementarity and development purposes of research methods). In other words, I used after-project questionnaires to get a first impression of the ability of the projects to achieve ASEC engagement along with the contextual factors that contributed to increasing or diminishing it; data which I would in turn use, not only to better interpret the information obtained by means of the after- project group interviews, but also to accurately design and conduct these interviews.

Thus, the after-project questionnaires offered me an opportunity to rapidly obtain a parallel account of events that would serve as the baseline for the development of future data collection instruments, and the better understanding of their findings. This satisfied the purpose of the qualitative triangulation proposed for this study (cf. 4.4), which sought to make every attempt to increase the validity and reliability of the findings of this investigation — and herein lies the reason for using this instrument in addition to the others. The Table below gives an overview of the questionnaires administered in this

Table 4.3: Questionnaire types and characteristics

Questionnaire type Participants Aim

After-project

questionnaire n.1 17

Explore the impact of the first project on students’ ASEC

engagement (if any) and learn about what facilitated or diminished it.

After-project

questionnaire n.2 17

Explore the impact of the second project on students’ ASEC engagement (if any) and learn about what facilitated or diminished it.

Regarding administration, the after-project questionnaires were handed out at the end of the lesson right after the project was over, and along with a brief explanation of the purpose and significance of the questionnaire, both for them (i.e., benefits of reflection for learning) and for me (i.e., helping me to improve education). Following

Oppenheim’s (1992) suggestions on questionnaire group administration to minors, I asked another adult (i.e., Pamela) to be present (and even help me with the distribution) to ensure that the students would fill out the questionnaire by themselves without chatting or copying. To increase honesty in student responses, since they felt more at ease when dealing with an outsider like me (as the interview data will show in Chapters Six and Seven), before starting the questionnaire I reminded the students that I was the only one who would access the data, just in case they thought that the presence of Pamela implied otherwise (see Dörnyei 2003, for a discussion on emphasising confidentiality). In fact, to illustrate my point, towards the end of the questionnaire I asked Pamela to step outside the class so that I could collect the questionnaires, and answer any questions the students might have regarding the data they had provided. That being said, I have to say that this approach to increase the quality of participant

responses in the questionnaire was in fact my backup plan. Originally, I had wanted to bring a member of staff from outside the area of languages to give out the

questionnaires, and then put them in an envelope, which would be signed by a student (Clément et al., 1994). I had thought at the time that such a course of action would release the students from any influence that Pamela might have on their responses, and might also make the participants provide more honest information as a result of the perceived anonymity (Dörnyei 2003). Sadly, on both occasions (i.e., project one and project two) no staff member was available, so it was necessary to resort to the strategy mentioned above.

Regarding the questionnaire design, I paid attention to several format characteristics. Firstly, as recommended by Dörnyei (2003; 2007), I kept it as short as possible. This was an important aspect to consider since I did not want to produce long questionnaires that would discourage participation (Oppenheim 1992; Sudman & Bradburn, 1982). With this in mind, I generated questionnaires that did not exceed four pages and would not take more than 30 minutes to complete. In fact, I designed the questionnaires so that they could be answered in less than ten minutes.

In terms of the layout, I tried to produce questionnaires that would look professional and attractive because, as Dörnyei (2003: 89) notes, “respondents are normally willing to spend time and effort on a questionnaire if they believe that they are contributing to a serious investigation”. In practical terms, this meant that, on one hand, I opted for a

bookletformat that would not only make the questionnaires look serious and

professional, but also shorter (Dörnyei 2003). On the other hand, I tried to be balanced when using page-full questionnaires so as to avoid looking crowded (ibid.). Finally, to further contribute to improving the respondents’ impressions of the questionnaires, I focused on aspects such as providing an orderly layout with well-marked sections or using coloured paper (Dörnyei 2003). With the sequence marking I wanted to convey a sense of visual structure and one of progressing, while with the coloured paper my intention was to further increase the attractiveness of the questionnaires. In addition to

this, I also included some decorations and pictures to make the questionnaires more visually appealing to further encourage respondents to participate (see Appendix F, for a sample).

Focusing now on item design, I drew heavily on established questionnaires guided by the fact that questions that have been empirically validated are more likely to have already gone through refinement process (Dörnyei 2003; Sudman & Bradburn, 1982). Thus, for the after-project questionnaire, I drew on a variety of sources to compose the questionnaires, especially on questionnaires that had been previously used in studies to measure engagement at the school and classroom levels (e.g., Appleton et al., 2006; Finn & Zimmer, 2012; Fredricks et al., 2005; Greene et al., 2004; Reeve 2013; Walker et al., 2006; Wang et al., 2014). The reason why I leaned on established questionnaires employed to measure engagement at the levels of the school and the classroom is because validated scales at the activity level are still in their early stages (see Greene 2015, for a discussion). As Fredricks and McColskey (2012: 766) point out:

“[engagement] measures are rarely worded to reflect specific situations or tasks, making it difficult to examine the extent to which engagement varies across contexts”.

Moreover, the after-project questionnaires encompassed different questionnaire items and types. They consisted of close-ended items, which “[did] not require the respondents to produce any free writing; instead, they [were] to choose one of the alternatives, regardless of whether their preferred answer [was] among them” (Dörnyei 2003: 35). The after-project questionnaire encapsulated 35 items while the second one featured 34; these ranged from rating scales to checklists. The rating scales, whose characteristics were explained in the previous section, were mainly used to collect evaluative information on student ASEC engagement and the contextual aspects influencing it. They were assembled into a four-response option to avoid students adopting a neutral position and therefore obliging them to make a choice. I did so because, unlike the open-

response questionnaires, I did not have qualitative data to interpret middle answers in either a positive or a negative light. As for the checklists, which consist of “mark[ing] the items on the list that apply to a particular question” (Dörnyei 2003: 46), they focused of gathering information about student learning, which I considered useful in terms of exploring the relationships between student engagement and subsequent learning.

At the final stage of the questionnaire construction, similar to the open-response questionnaires (cf. 4.4.3), I did not pilot the after-project questionnaires. It was

impossible to find a sample group similar to the target group, as only the participants of this investigation did the projects. Nevertheless, this did not stop me from conducting some trial runs with my PhD colleagues that would help towards making alterations and improving the final versions of the questionnaires. In the case of the second after-project questionnaire, I also drew on the experience gained from the administration of the first after-project questionnaire to help me modify the former. Moreover, despite omitting the piloting stage for both questionnaires, their qualitative nature ensured they were still highly suitable instruments. Simply put, since the purpose of the after-project

questionnaires was to lay the groundwork in the design of other instruments, as well as provide a parallel, (merely) informative perspective of the social phenomena under study — for the sake of data complementarity — it is my view that the absence of a formal pilot should not be considered a serious threat towards the validity of the data obtained by means of these questionnaires. By the same token, other limitations often associated with questionnaires, such as the superficiality and unreliability of responses (see Dörnyei 2003, for a discussion), should not pose a threat to this study because of the non-quantitative aim of the questionnaires.