Open-Response Questionnaires (or Reflection Sheets)

employed a third qualitative research instrument, the open-response questionnaire(s) (Brown 2009), which I often refer to as the reflection sheet across this study for reasons that I specify below (see Appendices D and E, for samples). This is a type of

questionnaire that includes items “where the actual question is not followed by the response options for the respondent to choose from but rather by some blank space (e.g., dotted lines) for the respondent to fill” (Dörnyei 2003: 47). Thus, I opted for this

particular data collection instrument because I wanted to gather evidence of student ASEC engagement throughout the project, in addition to documenting such evidence at the end of the project by means of the after-project group interviews (cf. 4.4.1.2) and after-project questionnaires (cf. 4.4.4). Furthermore, I wanted to obtain richer

information than just quantifiable data (i.e., after-project questionnaires, which I address later on), in a way that would not be too time and effort-intensive but allow for regular data collection.

In addition to the primary objective of the open-response questionnaires, their secondary aims included learning about the problems and challenges (as well as the strengths) that the project, and thus the implementation of the ProE model, posed to the students in terms of ASEC engagement. This would better help me “determine the actual source of engagement [or disengagement], how engagement is [was] related to context, and how engagement changes [changed] if conditions are [were] altered” (Fredricks et al., 2004: 86), which is an area that is often unexplored by engagement researchers (Fredricks & McColskey, 2012). It can be argued that, although much progress has been made regarding which aspects of the context influence engagement and to what extent, to date there is still limited empirical evidence regarding the nature of such interactions (see Christenson et al., 2012, for a discussion). This is explained, in part, by the fact that

measures of engagement often rely on quantitative methods that can hardly go beyond mere description of causality (ibid.). Moreover, I also used open-response questionnaires as a tool to bolster student engagement through self-assessment and feedback, as

recommended in formative assessment research (see William 2011, for a discussion). Accordingly, on the one hand, it is known that when students (a) assess their own work, (b) identify discrepancies between current and desired performance, and (c) they identify and implement courses of action to improve their understanding or skills (Cauley & McMillan, 2010: 207) they engage with their work more deeply. This is partly related to the fact that they feel more involved with the learning process (Nicol & Macfarlane- Dick, 2006). On the other hand, feedback, especially that directs “attention to what’s next rather than focusing on how well or badly the student did on the work” (William 2011: 128) helps close the gap between students’ current and desired performance and, as a result, contributes to enhance engagement.

The respondents individually completed the questionnaires three times across the course of each project; the first questionnaire was given out in the initial stages of the project, the second in the middle stages, and the third towards the project’s completion. This implementation strategy was kept for both projects one and two. Also, regarding questionnaire administration, prior to submitting each open-response questionnaire the students were always reminded of its purposes and why it was worth answering — namely, the benefits of reflective practices for learning (see William 2011). I must admit that, initially, I had planned to have students complete an open-response questionnaire every week for the duration of the project to collect as much data as possible. However, I was unable to do so, since we frequently ran out of time and at times the class

dynamics (e.g., misbehaviour, tiredness, etc.) simply did not allow for reflection time.

The format of the open-response questionnaires was designed in a friendly way to encourage participation. In other words, the questionnaire was kept relatively short (i.e.,

both sides of a page for the first project and only one side for the second project), using a booklet format (in the case of the questionnaires for the first project), reducing the density of the questionnaire as much as possible, and using coloured paper to enhance its attractiveness. These aspects are discussed in more detail in Section 4.4.4 where I address the after-project questionnaires. Additionally, as far as time-effectiveness is concerned, the questionnaire was laid out so that it could be answered over a ten-minute period at the end of the class. I felt that a longer reflective engagement after a one-hour class (two hours on some occasions) would jeopardise the quality of the answers given. Consequently, the questionnaires only featured a select number of sentence completion items (i.e., approximately six in each questionnaire), which encouraged the respondents to write a response of more than sentence and up to a paragraph in length. Amongst the reduced number of open-ended questions, I also included one close-ended item: a rating scale, which “[required] the respondent to make an evaluative judgement of the target by marking one of a series of categories organized into a scale” (Dörnyei 2003: 36). This was so that I could evaluate the students’ overall satisfaction with the lesson (i.e., emotional engagement). Although it can be argued that the inclusion of such an item did not lead to obtaining meaningful data, the truth is that the rating scales allowed me to observe a rise or a drop in student satisfaction during the project, the reason(s) for which would be potentially explained by the responses in the open-ended items.

Following in this vein, the design of the questionnaire items was approached from two perspectives. For the first project, I adhered to Brown’s (2009) considerations.

Therefore, as with the interviews, I spent some time reflecting on and building my own knowledge about the topic so as to create the best possible questionnaire to achieve my goals. This phase involved creating an item pool from which I would retrieve the final questionnaire items (see also, Dörnyei 2003). I then established the item sequence. This was set from easy to difficult, meaning that the cognitive effort required to answer the third question was higher than the one needed for the first. I did so in the hope of giving

the students both a pleasant first impression of the questionnaire and a sense that the subsequent questions could be completed as rapidly as the first. In stark contrast, for the second project I used an adapted version of William’s (2011) learning log. I decided to do this because I realised that the amount of data obtained from the open-response questionnaires used in the first project started to decrease over time, with responses including shorter, not-so-thoughtful answers and even unanswered questions. Therefore, I found myself obliged to generate a friendlier, open-response questionnaire that would be quicker to complete while still allowing me to obtain thoughtful and relevant

responses, especially at the emotional level, which is the dimension of engagement that I most wanted to tap into because it is less observable than the other dimensions (see Fredricks & McColskey, 2012, for a discussion). In this regard, William’s proposal seemed an appealing alternative due to the fact that it encourages an autonomous approach to reflecting on one’s learning by letting students choose three prompts they feel inclined to answer (see Appendix E, for a sample).

It must be noted that both open-response questionnaires underwent a series of changes over each project, not just from project to the other. Some of these changes affected the wording of questions while others comprised an alteration to the structure itself, including the order, nature or number of questions. To illustrate, in the first project I found that some students, in the three-point rating scale provided, were inserting a fourth option because they did not want to sit on the fence. This motivated me to shift to a four- point scale to see how the students would react. They reacted well, as there were no more instances of students drawing an extra option.

Another example of this is in the open-response questionnaires used in the second project. I realised that most students were choosing positive prompts (e.g., what they liked) rather than negative prompts (e.g., what they did not like). This made me think about what would happen if they had been forced to choose at least one negative

prompt. Interestingly, I thereafter realised that a significant number of students did not feel negatively towards the project at that particular time and for that reason they were only choosing the positive prompts. This alleviated my concern.

As a final example, in the case of the open-response questionnaire for the first project, I noticed that the feedback dialogue that I attempted with the students, in response to their questionnaires, did not work efficiently. That is, providing written feedback to each student for each questionnaire was a laborious task for which I would not obtain much in return. Generally speaking, the students would read my comments and answer my questions but this did not seem to have a significant impact on their performance. Moreover, in most cases, I would end up writing the same comments for each student because they all reported similar concerns in their reflections.

Regarding the questionnaire items, these were concise and succinct. Also, I formulated them so that they could easily be understood, with the hope that the students would not be discouraged from answering (Brown 2009). Hence, I stayed away from two-part questions, negative questions, and technical language. Furthermore, I avoided including biased questions. As mentioned above, the questions were not completely open-ended but consisted of some prompts to help the students answer the questions. This decision was partly made due to the low level of reflection I witnessed over the first term; it seemed to me that the students sometimes did not know what to say when they were asked to reflect on something.

On a final note, as this questionnaire was specifically designed for those who had been through the project experience I was unable to pilot the questionnaire. Nevertheless, before distributing it I trialled it with some of my colleagues, focusing on the

characteristics cited by Dörnyei (2003: 64), amongst which are: the number, difficulty, clarity and length of the questions, appropriate wording, and the overall appearance of the questionnaire. From the trial runs I obtained valuable feedback regarding these

aspects, and I was also able to see the kind of data that could possibly emerge from the questionnaires — since I asked my colleagues to write out their responses. All this was very helpful in refining the instrument. Similarly, the on-going administration of the open-response questionnaires also allowed me to treat each administration as an opportunity to pilot the questionnaires and continue to improve them. The next sub- section describes the last research instrument I employed in this study, namely the after- project questionnaires.