Research Instruments

Yin (2003) asserted that in doing case studies the researcher “should immediately note that no single source has a complete advantage over all the others. In fact, the various sources are highly complementary, and a good case study will therefore want to use as many sources as possible” (p. 85). He further listed six sources that can be used in a case study: document, archival record, interview, direct observation, participant-observation, and physical artefacts. Therefore, for the current research, the instruments selected to be used in executing evaluations and collecting related data are interviews, observations and document reviews. Interviewing will be employed as the dominant data collecting tool, whereas observations and documents will be employed to support and supplement interview data.

77 The current research uses the MP3 player as a voice recorder; notes were also taken by the researcher during the interviews. When and how the instruments were used is explain further in the following sub-sections.

3.5.1 The Interview

Bogdan and Biklen (2007) citing Morgan (1997) describe an interview as “a purposeful conversation, usually between two people, but sometimes involving more”. They further asserted that it “is directed by one in order to get information from the other” (p. 103). In qualitative research, according to Bogdan and Biklen (2007), there are two ways of using interviews. “They may be the dominant strategy for data collection or they may be employed in conjunction with participant observation, document analysis or other techniques” (p. 103). Additionally, Newby (1992) noted that an interview could achieve better response level, and allows for unexpected kinds of questions to be pursued. An interview, according to Newby (1992, p. 96), is appropriate for the purpose of, among others:

a) “exploring learners’ responses to particular training designs or learning methods b) assessing the extent to which training aims and content are perceived to be relevant

to the learners’ job activities

c) examining the extent to which training content has been applied to work practices d) gaining information about learners’ feelings and attitudes, and

e) as a preparatory aid in the drafting of questionnaires”.

Hence, for the current research, interview is the dominant tool for collecting data. The researcher takes down field notes, and records the interview using the MP3 player as a voice recorder, so that no part of the information is missed. This research adopts one- on-one in-depth semi-structured interviews. The semi-structured interview allow deep

78 probing. One-on-one interview is decided on for easy management and control over probing. The researcher does not have to go through a difficult time discriminating individuals’ voices from audio recording, as compared to a focus group interview where the researcher would have difficulty controlling the focus group, and may have difficulty discriminating between voices recorded. To avoid leading and dichotomous questions, this research ensures that the interview questions are open-ended. The Appendices A-2, A-3, A-4, A-5 and A-6 illustrate the interview open-ended questions. These questions have been created and classified into five categories according to the participants in the current research.

3.5.2 The Observation

Newby (1992) extended the term observation as “behaviour observation”. According to him, “the main methods of observation are direct analysis of live behaviour, indirect analysis using video recording, and indirect analysis using sound recording” (p. 197). For the current research, direct analysis is mostly adopted as it is less time-consuming and non-intrusive, and allows better control over an observed situation. Therefore, the researcher’s role is as a nonparticipant observer. While observing, the researcher takes down field notes. The Appendix A-7 shows the use of the “Observation Guide” form used by the researcher who observes trainers and trainees in the selected training programs, and during visits to the trainees in their work context.

Even though an observation may give a superficial account, it may be a beneficial support to interviews. As mentioned by Fraenkel and Wallen (2008, p. 440), “Certain kinds of research questions can be best answered by observing, how people act or how things look”. An example from the current research on determining participant’s motivation level, apart from interviewing them this study could observe how the participants behave during training, whether they look attentive or otherwise.

79 The respondents as a whole may not act naturally when they know or realize they are being monitored. Ways to minimize this problem will be explained in the observation procedures section. The following Figure 3.2 summarizes the varying observation approaches adopted by this research taken from Fraenkel and Wallen (2008, p. 442).

Role of the observer: Observer is an outsider

How the observer is portrayed to others:

Participants know that observations are being made and they know who is making them.

How the purpose of observation is portrayed to others:

The purpose of the observation is fully explained to all involved.

Duration of the observations: Multiple observations; long-term duration (e.g., months).

Focus of the observations: Broad focus: Holistic view of the activity or characteristic

being observed, and all of its elements are sought.

Figure 3.2 Role of observer.

3.5.3 Document Analysis

Creswell (2008, p. 231) asserts that “documents represent a good source for text (word) data for a qualitative study”. He adds that documents “are also ready for analysis without the necessary transcription that is required with observational or interview data.” On the other hand, he argues that “documents are sometimes difficult to locate and obtain”. Also Yin (2003) notes that the most important use of the document is “to corroborate and augment evidence from other sources” (p. 87). In the case of the current study, some relevant documents to be obtained will be classified in accordance with the aims of the study. The available documents used in the current research include training materials, annual training reports from 2009, 2010 and 2011, the training official plan 2009, 2010

80 and 2011 -- objectives, content outline, delivery method, budget, assessment methods, existing questionnaires, performance appraisal, each level of evaluation’s findings and any relevant documents related to the training and evaluation. The Appendix A-8 illustrates the “Relevance Document Summary” form, and the Appendix A-9 illustrates the “Document Review Guide” form employed in this research.