Research Methods

The focus of this section is on the qualitative and quantitative methods employed in this study. Specifically, it examines the rationale for developing and employing semi-

structured questionnaires and semi-structured interviews as data gathering instruments in this study. These two different methods were chosen because, as Denzin and Lincoln (1994) have noted, it is difficult for a single method to fully capture the richness and diversity of people’s experiences. Also discussed are the limitations and potential issues associated with the use of these two instruments that were considered when undertaking this study. Chapter Four describes the development and the use of these two instruments in this study.

3.3.1. Questionnaires

Questionnaires were selected as a key data gathering instrument in this longitudinal, mixed methods study because they are highly efficient and versatile instruments for gathering data over a period of time from a large group of participants (Check & Schutt, 2012). The decision was made to develop semi-structured questionnaires comprising a mixture of closed and open-ended questions, thus enabling both quantitative and qualitative data to be gathered through one instrument (Cohen et al., 2011).

There were several reasons for developing semi-structured questionnaires, rather than relying on existing questionnaires. Most importantly, a wide range of closed and open- ended questions were required to collect the quantitative and qualitative data needed to answer the research questions in this mixed methods study. No existing

questionnaire provided the range of questions that were needed to capture the complexity of students’ motivation within the NCEA environment. Furthermore, most pre-existing questionnaires are structured questionnaires, providing only quantitative data. While structured questionnaires can enhance the manageability of data and the generalisability of the results, this can be at the cost of appreciating the variety of meanings that students attribute to their educational experiences (Dowson &

questions and the questionnaires also needed to be appropriate for the context in which data were being sought and for the longitudinal nature of this study. As a result, four types of purpose-designed questionnaires were developed to be administered at different points across the 12 month data gathering period.

When developing questionnaires for this study, a consistent focus was maintained on both the research objectives and population from which participants would be drawn, to ensure valid data were gathered (Check & Schutt, 2012). To enhance the response rate and quality of the data gathered, attention was given to the quality, clarity, and sequencing of the questions, the clarity of the instructions, and the layout of the questionnaires (Cohen et al., 2011; Fowler, 2009). Questions were pretested with a pilot group to ensure they functioned effectively to achieve the research objectives. The details of the piloting process are outlined in Chapter Four.

The length of the questionnaires and the cognitive demands of the questions were also taken into consideration in an effort to limit respondent fatigue and satisficing (Ben- Nun, 2008; Krosnick, 1999). Satisficing occurs when respondents answer questions superficially, rather than thoughtfully (Krosnick, 1999). Superficial responses can greatly affect the validity and reliability of the data being collected. To further minimise satisficing, Krosnick (1999) has suggested that participants are reminded that their responses are valued. While social desirability can also be an issue (Check & Schutt, 2012), it was not considered to be an issue in this study, given the focus of the research questions.

Also of concern is the impact of questioning participants using the same or similar questions administered at different times through repeated questionnaires in longitudinal studies. Participants’ responses to questions that they have previously encountered may be affected because they are encountering the same question again (Ruspini, 2008). However, the importance of using some of the same or similar

questions to aggregate responses and allow comparisons to be made outweighed this concern.

3.3.2. Interviews

Interviews feature strongly in qualitative research, because they are one of the most effective and direct ways to access “people's perceptions, meanings, definitions of situations and constructions of reality" (Punch, 2005, p. 168). Furthermore, they are

versatile and flexible (Sarantakos, 2005). However, as Richards and Morse (2007) have noted, conducting an effective interview “requires extraordinary concentration”, as it is “the nature of the questions asked and the attention the researcher gives the participants and to detail that determine the quality of the data collected” (p. 109).

In this study, semi-structured interviews were considered to be the most appropriate means of eliciting the type of information sought from participants; that was asking participants to elaborate on or clarify their questionnaire responses from Phase One. Semi-structured interviews are inherently flexible. The opportunity to vary the questions asked in an interview was an important consideration when seeking to understand students’ motivation in greater depth (Bryman, 2012; Cohen et al., 2011).

The interviews were conducted face-to-face, as this approach followed naturally from the personal contact established during Phase One. Face-to-face interviews have the added advantage of drawing on multi-sensory channels of communication: verbal and non-verbal, spoken, and heard (Cohen et al., 2011). Other aspects were also given due consideration, such as where the interview would be conducted, the importance of establishing a positive rapport, and ensuring the interviews were conducted

professionally and sensitively (Kolar & Kolar, 2008; Mertens, 2010; Sarantakos, 2005).

During the interview care was taken to maintain the dynamics of the interview; in particular to keep the interview flowing and focused on the research objectives (Cohen et al., 2011). Throughout the interview efforts were made to listen attentively, observe the interviewee’s body language, show an interest in what was being said, and to not lead the interviewee to give a particular response (Cohen et al., 2011; Richards & Morse, 2007). Given the cognitive demands associated with comprehending a question, recalling information, and constructing a response, time was allowed for interviewees to think about their responses (Sarantakos, 2005). Providing wait time has been found to increase the quantity and quality of responses given (Rowe, 1974, 1986). For the same reasons time was taken to consider an interviewee’s response before proceeding with the next question (Rowe, 1986).

Digitally recording the interviews enabled full attention to be given to the interviewee, the interview questions, and the dynamics of the interview (Cohen et al., 2011). Having an interview schedule also helped to ensure key areas were covered, and that greater attention was given to the interviewees and their responses (McCracken, 1988).

While interviews are very versatile and useful for gathering data, they also have their limitations and potential issues. Some of these issues are the same as those identified in relation to questionnaires, such as respondent fatigue, satisficing, and social

desirability. Efforts were made to minimise these issues by keeping the interviews relatively short and focused, by closely monitoring interviewees, showing an interest in their responses, ensuring they knew that their contributions were valued, and not becoming overly friendly (Check & Schutt, 2012; Fowler, 2009; Krosnick, 1999).

Inherent within the interview process are other potential issues. With semi-structured interviews there is a greater risk that questions may be poorly phrased, but equally there are opportunities to rephrase, clarify, or elaborate. Verbal and non-verbal responses, manner, voice, attentiveness, and ability to probe appropriately can all influence an interviewee’s responses (Cohen et al., 2011; Fowler, 2009; Gomm, 2008; Richards & Morse, 2007). While it is impossible to eliminate these issues, considerable care was taken to minimise their impact.

Transcriptions of digitally recorded interviews can also be problematic. Any errors made in the transcribing process can affect the analysis and interpretation of what has been said (Bryman, 2007). Also, transcriptions do not capture everything that occurred during the interview. In particular, they fail to capture the nuances of the social

interactions that occur, such as the tone and inflection of the voices, pauses, and body language, which all impact on the way in which meaning is interpreted (Cohen et al., 2011). However, in this study these concerns were somewhat negated by the fact that I undertook all the interviews, and thus was able to analyse the data with a recollection of what occurred during the interviews.