Data collection instruments

A paper-based questionnaire (Appendix F) was used for data collection in the first phase of the study. A questionnaire, according to Malhotra (2006), is a formalised set of questions to achieve information from the respondents. Mathers, Fox, and Hunn (2007) highlighted that a questionnaire is “a very convenient way of collecting useful comparable data” from a large number of individuals (p.19). The questionnaire in this study was designed with the research questions and the characteristics of the target respondents in mind. It is comprised of three sections. Section A requires the participants to provide their demographic information, followed by section B aiming to capture the participants’ perspectives on institutional accreditation. Section C examines their views on the programme accreditation process. All participants were requested to answer Section A and to select the appropriate section (B, C) based on their experience.

The questions employed in sections B and C are almost the same. The sole point of difference between sections B and C is that while section B includes questions to extract the participants’ views on key policies and practices of the follow-up, those questions were excluded from section C. This difference is due to the fact that the programme accreditation in 2012 was a pilot and therefore the results were not approved and published.

In each section of B or C, the questions were categorised into the two following sub-sections:

Sub-section 1: Staff perceptions of the key policies and practices of accreditation

As there has not been prior empirical research examining the perceptions and experiences of different stakeholders of some key policies and practices of EQA, most of the quantitative items in this sub-section were developed by myself based on literature on the key elements of accreditation and my experience as a practitioner in the field. Some items relating to the self- study course were developed based on those employed in the studies on teacher training course evaluation of Wong and Yeung (2003) and Uysal (2012). Due to the limited scope of the study as well as the practical conditions, only main aspects concerning key policies and practices of accreditation were taken into consideration in designing the quantitative items. Each quantitative item is a positive statement with respect to a specific policy or practice of accreditation, using a 5 point Likert scale (1=strongly agree, 5=strongly disagree). The respondents were also asked to respond to the open-ended questions denoting key limitations of accreditation policies and practices.

Sub-section 2: Staff perceptions of the purposes and impact of accreditation

Similar to sub-section 1, this sub-section includes both closed-ended and open-ended questions. Mathers, Fox, and Hunn (2007) noted that in selecting between pre-existing questionnaires and the self-developed one for the study, the priority should be given to the former. One of the benefits is that the pre-questionnaires have already been validated and tested well for reliability. Thus, the quantitative items employed in this sub-section were adapted from Stensaker et al. (2011) as the well-known scholars in the area of educational QA. Although their study only used a closed-ended questionnaire to collect data, it was seen to be successful in drawing a comprehensive picture of how different stakeholders within universities and colleges in Norway perceived the impact of different methods of EQA. The questionnaire in their study covered a range of key issues in respect of EQA impact as documented in the literature.

Based on the Stensaker et al.’s (2011) study, the quantitative items in this sub-section are comprised of Likert-type response items focusing on three issues including: (1) the perceived purposes of accreditation, (2) the perceived overall impact of accreditation, and (3) the perceived impact of accreditation on different areas of institutional practices and institution’s reputation. The participants were also requested to give open feedback with regard to the purposes of the accreditation, the benefits gained as well as the problems and challenges they faced when implementing this process. They were also asked to recommend what could be done for more effective accreditation.

In short, the questionnaire employed in this study was carefully designed so as to obtain a wide range of data from the participants. While the closed-ended questions allowed information to be captured from the predetermined categories, the open-ended questions allowed the respondents to answer in the manner they wished, thereby eliciting their true feelings and opinions of the research problems (Gillham, 2008).

Semi-structured interviews

Interviewing is a powerful tool for eliciting people’s views and attitudes (Gray, 2009) and also “the main road to multiple realities” (Stake, 1995, p. 64). The second phase of the study involved the one-on-one interviews with eight ‘key’ staff members. In their study, Colton and Covert (2007) underscore that a survey may bring about meaningful results but a researcher needs to gather some additional information not readily assessed by predetermined items. Therefore, the combination of questionnaire and interviews in this study would lead to a more in-depth understanding of staff’s perspectives on the two accreditation processes.

The decision to conduct individual interviews instead of group interviews in the second phase was made as the former seemed to be easier to arrange and manage than the latter. The staff members in the colleges were often very busy, which made it difficult to arrange a meeting that suited them all. Moreover, the individual interviews, according to Tomal (2010), could draw out true feelings that might not be obtained in a group setting. The type of interview selected for this study was semi-structured face-to-face interviews. The reason is that this type of interview permits probing and exploring within the predetermined inquiry areas to obtain additional information about the phenomenon under investigation (see Gall, Gall & Borg, 1999).

After the initial survey data analysis, eight different interview protocols were prepared to serve as the guides to the ‘conversations’ with the eight interviewees. Each interview protocol included the common themes used for all the interviews and the additional specific ones according to the responses of a specific interviewee in his or her returned questionnaire. The common themes in all the interview protocols not only referred to the significant survey findings but also the general perceptions and experiences of accreditation.