Chapter 4. Methods
4.4. Development of Data Gathering Instruments
This section explains the development of the questionnaires and interview schedules. Also included is an explanation of how the instruments were piloted.
4.4.1. Development of the Questionnaires
Four different types of semi-structured, written questionnaires were developed for this longitudinal study: the two general questionnaires, the four pre- and the four post- achievement standard questionnaires, and another questionnaire which focused on
14
It was not possible to gain information about the ethnicities of the potential pool of 232 students to establish the degree to which the sample reflected the ethnic makeup of the cohort from which they were drawn.
extra-curricular activities. The rationale for developing and employing semi-structured questionnaires has been discussed in the previous chapter. Each of the four types of questionnaires served a specific purpose. However, a number of common questions were repeated across three of the four types of questionnaires. The fourth type of questionnaire, the Outside Class Activities Questionnaire, was markedly different and very short. In total there were 11 questionnaires. These questionnaires are listed in Table 4.2. A brief description of the focus of each questionnaire is also provided. (Also seeAppendices B1-B5 for examples of the questionnaires.)
Table 4.2: Name and focus of each of the 11 questionnaires
Name of Questionnaire Focus
Initial Questionnaire Motivation in NCEA level 1 English and predictions about performance in NCEA level 2 English. Pre-Achievement Standard 2.1
Questionnaire
Post-Achievement Standard 2.1 Questionnaire
Motivation for the creative writing achievement standard 2.1 (internally assessed).
Pre-Achievement Standard 2.2 Questionnaire
Post-Achievement Standard 2.2 Questionnaire
Motivation for the formal writing achievement standard 2.2 (internally assessed).
Pre-Achievement Standard 2.3/2.4 Questionnaire
Post-Achievement Standard 2.3/2.4 Questionnaire
Motivation for the extended text achievement standard 2.3, or short texts achievement standard 2.4 (externally assessed). Students could choose one or the other, depending on which they believed they were most likely to complete in the external exams, as not all students completed both in the exams.
Pre-Achievement Standard 2.6 Questionnaire
Post-Achievement Standard 2.6 Questionnaire
Motivation for the unfamiliar text achievement standard 2.6 (externally assessed).
Outside Class activities Questionnaire Regular activities students are engaged in outside class (e.g., sporting commitments, part-time work)
Final Questionnaire Motivation over the year in NCEA level 2 English.
The development of questions for each of the questionnaires occurred after a wide ranging literature review was undertaken to identify various theories that were likely to be applicable to the research objectives of this study. While self-determination theory was the overarching theoretical framework guiding this study, a number of closed and open-ended questions were also developed around other specific motivation theories and constructs, such as self-efficacy, interest, and attribution theory. In addition, closed and open-ended questions were incorporated, which focused on social and contextual
factors that might potentially influence students’ motivation. Moreover, general open- ended questions were included to allow students to list factors that they identified as being most influential in their motivation to achieve NCEA level 2 English overall and the four specific English achievement standards. In the Final Questionnaire students were asked to rank their responses from the most to the fourth most influential factor influencing their motivation. This ranking process enabled the most salient factors to be identified.
The majority of closed questions used Likert-type response scales (e.g., a five-point scale from very unhappy to very happy). Each of these closed questions was usually followed by an open-ended question, to allow participants to explain, qualify, or justify why they selected a particular choice in the preceding closed question (Hall, 2008). The pairing of open-ended questions with closed questions was aimed at enhancing the meaningfulness of the data gathered and the accuracy of interpretation. For example, if students indicated that their parents influenced their motivation a lot, then the open-ended question helped to obtain an understanding of how the students believed their parents influenced their motivation, and if that influence was positive or negative. In turn these qualitative data were then quantiized to determine which parental influences on students’ motivation were most salient.
In some instances, filter questions were used to gather the most useful data. For example, if students indicated that their friends had influenced their motivation a lot, they were directed to the contingent question to explain why, whereas if they answered
a bit or not at all, they were directed to skip the contingent question. In such instances only factors that were perceived to be influential were required in keeping with the focus of this study.
One key open-ended question involved students identifying up to four factors they believed had the most influence on their motivation to achieve NCEA level 2 English. The question that immediately followed this question asked students to then rank their responses from the most influential to the least influential (see Question 11 (b) in the Final Questionnaire in Appendix B5). The ranking was important for determining the most influential factors overall.
Care was taken with the wording and sequencing of questions, instructions, and layout, to enhance the response rate and consistency of the responses (Fowler, 2009). The length of the questionnaires and the cognitive demands of the questions were also
taken into consideration to limit respondent fatigue and satisficing (Ben-Nun, 2008; Krosnick, 1999). Advice on all these aspects was sought from the pilot group (see subsection 4.4.3). For instance, the pilot group recommended that examples of possible answers be provided for a number of open-ended questions, thus helping participants understand the type of response they might give, and thereby lessening the cognitive effort required. (There was a concern that participants might simply copy these, but this rarely occurred.)
The issue of social desirability was also considered. At the top of each questionnaire was a statement that stressed there were no right or wrong answers and emphasised the importance of giving answers that really reflected each participant’s views, thoughts and feelings.
The questionnaires were designed to be self-administered in groups, in a face-to-face context, under my supervision. The inclusion of all the instructions within the
questionnaires ensured a degree of consistency across different situations and time. Furthermore, designing the questionnaires to be self-administered enabled students to complete the questionnaires at different rates, and catered for those students who arrived late, or needed to complete a questionnaire at home.
4.4.2. Development of Interview Schedules
Semi-structured interviews were selected as a suitable tool for extending the knowledge and understanding gained from the questionnaires. The purpose of the interviews was to ask 16 of the 107 participants to elaborate on particular responses that they had made in the 11 questionnaires to more fully understand the factors influencing their motivation to achieve NCEA level 2 English and specific English achievement standards. Given that the purpose of the interviews was to gather qualitative data, the interview questions were all open-ended.
Each set of interview questions began with the same opening question, “Can you tell me what it was like for you doing NCEA English last year?”Each interview ended with the same two closing questions, “What advice would you give teachers on how to help students feel motivated to do well in Year 12 NCEA English?” and “Thinking about your
motivation in NCEA English are there any other points you think it would be good for me to know about?” The questions in between the opening and closing questions were specific to each of the 16 participants. As the questions arose from a review of each
student’s responses from their 11 completed questionnaires, 16 individualised interview schedules were created. However, it was expected that other questions would be asked during the interview when the need arose to seek greater depth and breadth of understanding.
4.4.3. Pilot Study
In October 2009, a request was made to a secondary school principal to pilot the questionnaires and trial the interview process for this study with six students (three males and three females) from Year 12 A Stream English classes. This school was one of the two schools from which students were drawn for the main study in 2010. NCEA level 1 English results were used to ensure that the students selected varied in ability from “average” to “very able”. Each student was sent a letter inviting them to be part of the pilot group, and outlined the purpose of the research, why their input was important, and gave assurances that the data they provided would only be used for the purposes specified. Face-to-face meetings were subsequently arranged with the group during their lunch breaks.
Initially the students were asked to answer the questions in a prototype of the Initial Questionnaire, and to make notes of any specific words, questions, or instructions they did not understand, and questions they found frustrating to answer. They were also asked for advice on the layout of the questions. At the conclusion of this process the students shared the points they had noted and discussed possible solutions to issues that were raised. This feedback was then used to make changes to the prototypes of the Initial Questionnaire, the pre- and post-achievement standard questionnaires and the Final Questionnaire. The review process was repeated three times with successive improvements being made to each type of questionnaire. The only exception was the Outside Class Activities Questionnaire, which was developed later when it became apparent participants were often involved in other activities that might potentially affect their motivation. This iterative review process was concluded when students
considered the questionnaires were ready to be administered to participants.
Members of the pilot group were also asked to identify all the factors they believed had an important influence on their motivation in NCEA level 2 English. These factors were discussed and compared. Based on their feedback, a list was produced, reviewed by the pilot group, and then finalised. The list became the basis for Question 7a in the pre- and post-achievement standard questionnaires and Question 6a in the Final
Questionnaire. The interview process was also trialled with two members of the pilot group with individualised interview schedules developed from their responses to the prototype questionnaires.
Advice was sought from the pilot group as to whether participants should complete these questionnaires electronically outside school hours. They were unanimously against this idea on the grounds that students were unlikely to complete the questionnaires outside school hours because of other activities or commitments. Instead, they recommended that participants complete paper-based questionnaires during their lunch breaks. This advice led to a request to the ethics committee for approval to amend the original proposal to allow participants to complete
questionnaires at school during their lunch breaks.
The piloting process was valuable and informative. The significance of the pilot group’s contribution was evident in the fact that nearly all the participants in the main study completed all the questionnaires, the questions elicited the types of responses that would have been expected, and participants rarely asked for clarification when completing the questionnaires. As a result of trialling, the interview process with the participants also went smoothly. “Thank you” letters were sent to each member of the pilot group and the principal.