Chapter 4: Data Collection
4.2 Preparation of Interview Questions
4.2.2 Pilot Study for the Interview Questions
The main challenge for employing interview questions as a data collection procedure is the need for instrumentation rigour and bias management (Chenail, 2011). A pilot study is a common practice that is employed to assess and refine the interview questions, and discover any potential researchers’ bias to ensure that the questions make sense and interviewees understand the questions being asked (Chenail, 2011; Jacob and Furgerson, 2012; Creswell, 2013). In brief, a pilot study allows the investigator to determine which questions are effective and which are not. Although a pilot study may not promise a successful data collection during the interview, it may increase the success rate of interviews and improve the quality of the data collected.
In social science research, there are two types of pilot study, (1) a test run during planning for the main study and (2) a feasibility study, which is a small scale version (Chenail, 2011). This study has employed the first approach. A pilot study may involve asking the interview questions and having someone to answer them. The study could start up with friends or someone that we know (Jacob and Furgerson, 2012) and then follow up with people who are close to those that will participate in the actual study (Turner, 2010; Jacob and Furgerson, 2012). Pilot cases are typically selected based on convenient access and proximity on a geographic basis (Creswell, 2013). In the event where piloting is not practical, interviewing the investigator approach could be adopted in which the investigator undertakes the role as a research participant to be interviewed by someone, but in this event, the investigator may be blinded by their biases and unable to foresee the difficulties during the real interview session (Chenail, 2011). Thus, Chenail (2011) recommended the use of interviewing the investigator approach as a pre-piloting technique before conducting a pilot study.
a. Pre-pilot Study
During the pre-pilot study, the interviewing the investigator approach was employed. I undertook the role as a research participant (i.e. student) to be interviewed by a friend, and both roles were switched around afterwards. During this process, some questions were found to be too challenging and might be beyond a student’s comprehension. Then, we switched position – my friend acted as a teacher or parent to be interviewed by me. Some questions were found to be redundant, similar and not significant. During the interview process, mistakes were discovered while going through each of the interview questions in addition to new issues being raised. This
79
approach was good, but comparatively, more information was acquired for being an interviewer than playing the role of a participant because I could not be a real research participant. I knew the questions to be asked and I had my own agenda.
After conducting the pre-pilot study, interview questions had been reduced from an average of 75 questions (i.e. 74 for students; 83 for teachers; 66 for parents) to 48 questions (i.e. 51 for students; 50 for teachers; 43 for parents) without affecting the objective to the study and some questions were rephrased for better understanding and practical reasons. An average time to answer the 75 questions was about 2 hours. Even being an interviewer myself, a large number of questions had also caused exhaustion, loss of concentration and boredom. It exceeded the interview best practice of an hour to 90 minutes of interview (Hatch, 2002).
As a result from the first round of pilot study using interviewing the investigator approach, the interview questions were tested again with me being the interviewer and my friend being the interviewee (i.e. acting as a teacher). After practising for more than three times, the time required for completing the 50 interview questions had been reduced to 45 minutes. This was due to my improved familiarity with the questions asked. We did not practise the interview questions for students and parents once again because the questions were 90% similar to the teachers’ interview questions.
b. Pilot Study
To conduct a pilot study, participants who were similar to the actual sample of study were recruited - three students, three mathematics teachers, and three parents. They were the people whom I knew personally, including my former teacher, friends, colleagues and their sons or daughters. Although it was a pilot study, all the participants were very helpful in providing constructive feedback, opinions and suggestions to improve the process of the interview. Each conducted interview lasted approximately 1 hour. People are less likely to agree to an interview if it lasts longer than an hour (Jacob and Furgerson, 2012; Mulder and Yaar, 2006).
i. Mathematics Teacher
During the pilot study, each interview took 50 to 60 minutes. The teachers had approximately 11 to 24 years of teaching experience at secondary schools. Feedback obtained from the pilot test had identified five weaknesses which were discussed below. After the pilot test, the interview questions for mathematics teachers were reduced from 49 to 42 questions.
Redundancy
Some questions were found to be redundant. Learning from this, other similar questions were combined into one, with the first question being a general question
80
and the subsequent questions would serve as probes. A general question allows the respondents to answer freely in any direction whilst the probes are used to obtain additional information or to clarify responses (Johnson and Christensen, 2008). One of the examples is shown below.
New question: Tell me about the approaches you used to teach mathematics. Do you
use group discussion, games, etc.? [General] Have you ever used computer technologies such as the Internet or any software to teach mathematics in the classroom? [Probe]
There were a few explanations for the mistakes made. When I designed the interview questions, I failed to foresee all possible responses in each interview question. I assumed that all teachers would respond according to what I had anticipated. Apparently, their responses were unpredictable and what was going through the teachers’ mind was a matter of conjecture. As mentioned in Jacob and Furgerson (2012), unexpected data and surprises were expected in a qualitative interview. I realised that probe questions were useful to direct the participants to the required path. In addition to that, prompts could be used to allow further justifications of claims and beliefs with appropriate examples and facts. Prompts are useful to get a specific information that respondents did not mention (Jacob and Furgerson, 2012).
Jargons
When all respondents (i.e. students, teachers, and parents) were asked the following questions, they did not understand the meaning of digital games and educational
computer games.
Have you ever played digital games?
Could you name a few educational computer games that you know?
It was important for the research participants to understand the questions asked so that they could generate a rich description and authentic data. Here, I initially made an assumption that all the respondents could understand and be familiar with the technical terminologies. In some cases, technical terminologies could be translated into the every day more easy going language (Cohen et al., 2007). However, translating the terminologies could alter or impair the actual meaning. Furthermore, these terminologies, digital and computer were rated as common computer terms (PC.NET, 2014). Similarly, the word computer was rated as daily used basic term, and digital was rated as frequently used basic terminology (TechTerms.com, 2014). Experience gained through piloting phase had supported the needs for providing brief definitions or examples in the new version of interview questions.
81
Ambiguity
When the following question was asked to all respondents, they elicited undesirable responses because they were confused with the term - technology skills.
What kinds of technology skills are you good at? What kinds of technology
skills are you weak at?
In the new version of interview questions, multiple choices with a list of technology skills were presented. These technological skills were developed based on the literature (GO-Gulf, 2012; Laura, 2005; Donaldson, 2013; Andrade, 2012).
New question: Based on the following technology skills, how you would rate yourself
with a rating of 1 to 5 (1 would represent novice, and 5 would represent expert). Typing skill
Word processing (e.g. Ms Word) Spreadsheet (e.g. Ms Excel) :
One of the important guidelines during the interview is to avoid ambiguity and imprecise questions (Arksey and Knight as cited in Cohen et al., 2007). Sometimes, it is inevitable as many meanings that are clear to one person will be relatively unclear to the others (Cicourel as cited in Cohen et al., 2007).
The teachers responded to abstract and difficult questions in two ways: (1) asked for clarification, (2) answered based on their personal understanding. The first scenario was evident, but the second scenario was very much dependent on my personal justification. I had noticed this problem during the pre-pilot test when my friend did not response accordingly to some interview questions. Suspecting that she could not understand the interview questions, I verified with her after the interview session. She admitted that she did not understand some of the interview questions and chose not to ask. She explained further that she felt embarrassed to ask for clarification. This is the knowledge aspect of the interview, in which interviewee must not feel threatened by the lack of knowledge (Cohen et al., 2007).
Learning from my mistakes, I pre-briefed all pilot test respondents before the interview, and informed them that they should not worry if they encountered any difficulty or confusion during the interview as I would be there to clarify their doubts and they were free to ask for help. It is important to establish the trust and good rapport between the interviewer and interviewees as this would enable the interviewees to provide information about their inner world (Cohen et al., 2007; Fraenkel and Wallen, 2006; Mertens, 2010; Johnson and Christensen, 2008).
82
Hypothetical
When the teachers were asked the question, “Recently there has been the general opinion that students are falling behind in mathematics. What do you think about this?” all teachers agreed on this statement. However, this question is a hypothetical or speculative question that should be avoided (Asksey and Knight as cited in Cohen
et al., 2007). A new question was designed with a proper fact and evident.
Original question: Recently there has been a general opinion that students are falling
behind in mathematics. What do you think about this?
New question: Recently, Malaysian students showed poor rankings and average
scores in both Trends in International Mathematics and Science Study (TIMSS) and Programme for International Student Assessment (PISA). What do you think about this?
Limitation of Multiple Choices
When all the respondents were asked “What do you normally use a computer for?” a list of activities were given with a multiple choices selection of yes or no. The responses given might be inaccurate because they were grey areas that existed between yes and no. For instance, some of them responded sometimes, rarely and
never, instead of yes or no. This indicated that the multiple choices (yes/no) given
were inadequate to accommodate all the possible responses. Besides that, when this question was asked, they related it to the utilisation of the Internet. Therefore, instead of asking about computer and the Internet separately, both usages were combined into one question. To rectify this problem, the frequency of usage was added to the list of activities – always, sometimes, rarely or never.
New question: What do you normally use a computer for? Based on the following
activities, tell me your frequency of use – always, sometimes, rarely or never.
always sometimes rarely never
Typing documents
Analyse data
Prepare presentation slides :
Interview questions designed with fixed-alternative items confine the respondent to choose their answer from two or more alternatives (Cohen et al., 2007). My intention was to achieve a greater uniformity of measurement so that the responses were coded more easily. The superficiality of multiple choices selection that was thought to be ideal had exposed some weaknesses during the pilot study.
83
the possibility of forcing responses that are inappropriate, either because the alternative chosen conceals ignorance on the part of the respondent or because he or she may choose an alternative that does not accurately represent the true fact (Kerlinger as cited in Cohen et al., 2007, p.357).
The limited fixed-alternatives may also irritate the respondents who find that none of the alternatives is suitable for them (Kerlinger as cited in Cohen et al., 2007). This problem is more manageable in an interview because an interviewer is more flexible than a survey questionnaire.
ii. Students
The three students interviewed were between 14 to 16 years old. Each interview took 43 to 52 minutes. After the pilot test, I realised that the most challenging aspect of interviewing children was their level of understanding. Two of the students were proficient in English, and yet they had difficulty in understanding some of my questions. After the pilot test, the interview questions were reduced from 51 to 45 questions. The weaknesses found during the pilot test with the students are discussed as follows.
Limited vocabulary
Other than the terms digital games, educational computer games, and technology
skills which were discussed earlier, the students also could not understand the
following terminologies - engineering, simulation, strategy, role-play, land navigation,
problem-solving, using search engines, social networking, and web browsing. In the
revised interview questions, a brief definition or examples were given. One of the revised questions is shown as follows.
New question: In digital game, do you think you learn _______ skill?
multitasking (i.e. handle more than one task at the same time) land navigation (i.e. use of compass, map and direction)
team work (i.e. working collaboratively with a group of people) :
The biggest mistake was using the vocabularies that the students could not understand. I initially disregarded the guideline given by Arksey and Knight (as cited in Cohen et al., 2007), in which attention must be paid to the vocabulary to be used and to try to keep it simple. In this case, limited vocabularies prevented students from understanding some of the interview questions. On a few occasions, the questions had to be rephrased with simpler words; some terminologies had to be explained and
84
a few common examples were quoted. Before the pilot study, I thought the students’ interviews were similar to adults’ interviews.
Although secondary school students are teenagers aged between 13 to 17 years old, they are presumed and associated with children in many past research studies (Punch and Hyde, 2011; Greene and Hogan, 2005; Nordström et al., 2014). Children differ from adults in the cognitive and linguistic development, and it is important to use straightforward language (Arksey and Knight as cited in Cohen et al., 2007).
Complicated question
A complicated question in this context meant that the question could not be fully understood by students at their level of understanding. Some of the interview questions were revised from experience gained in the pilot study. An example of a revised question is shown as follows.
Original question: Would you recommend the use of computer games to learn mathematics in the school?
New question: Imagine that one day your mathematics teacher is using a computer
game to teach in the classroom, how would you feel?
The use of role-play (i.e. imagination) question was effective. This approach is also suggested by Mertens (2010). An investigator should ask the same question in different ways to check whether the children understand what the investigator has been saying (Fraenkel and Wallen, 2006). For children’s interview, it is important to use child’s language (Arksey and Knight as cited in Cohen et al., 2007). Furthermore, I found that complex and compound sentence structure confused the students which confirmed the work of Fraenkel and Wallen (2006) who had suggested to avoid questions which elicited multiple points.
From the pilot study, my rule of thumb for children’ interview questions was to keep them short, simple, straightforward and easy to understand. This rule not only helps to obtain rich and valuable information from the children but also to gain their trust and confidence to talk and share their thoughts with ease.
Lack of Probe Question
In the pilot study, two of the students were shy to express themselves and more probe questions were required to lead the students to provide more detailed information.
Original question: Have you ever struggled in a mathematics exam?
New question: Have you ever struggled in a mathematics exam? What do you find
most challenging about mathematics? [Probe] In your opinion, what are the most difficult topics in mathematics? [Probe] Do you go to any tuition class? [Probe]
85
Interviewing teenage students is entirely different from interviewing adults. Students may feel intimidated during the individual interview due to generation gap between the investigator and students, their level of confidence and they may feel threatened by the lack of knowledge. Children differ from adults so it is important to establish trust and to make them feel enjoyable (Arksey and Knight as cited in Cohen et al., 2007).
To encourage them to talk during the interviews, I tried to learn the language which they were using and acted as their friend in order to encourage them to share their views and experience comfortably. Furthermore, projecting a friendly and professional attitude through body language and voice tone were very helpful in boosting their openness and encouraging them to share their thoughts.
iii. Parents
The three parents interviewed were between 38 to 42 years old, of different ethnicities (Malay, Chinese and Indian) – a Ph.D. candidate, a clerk, and a physicist. Each interview took 42 to 71 minutes. The most interesting aspect of interviewing the parents was their diverse perceptions and viewpoints. To refine the interview questions, some probe questions were added as inspired by the parents responses and some unnecessary questions were removed. After the pilot test, the interview questions for the parents were reduced from 43 to 39 questions. The weaknesses found during the pilot study are discussed as follows.
Ambiguity
When the parents were asked “What do you expect the future school should be?” they looked a little puzzled. Nevertheless, they expressed their expectations towards the school and education system. I should not assume that parents were knowledgeable and aware of the schools’ future plans like educators. Instead, I should change the question to ask for their expectations or suggestions of future schools.
New Question: As a parent, what is your expectation from the school education
system? Do you have any suggestion for the school?
Mathematics teachers have the similar educational background, working environment, and perceptions towards education. Parents, however, are people coming from various educational backgrounds with different types of occupations and diverse life experiences. Diverse life experiences shape divergent views or perceptions. Being an investigator, it is important to accept that the interview may provoke new insights (Kvale as cited in Cohen et al., 2007).
86
A pilot test with the parents led me to discover that they did not see and view education through the lens of an educator. They gave me new, interesting and creative ideas which helped me in the design of several new probe questions.
Ill-informed question
When the parents were asked “Recently, there has been the general opinion that students are falling behind in mathematics. What do you think about this?” they seemed to be ill-informed on this subject. Most probably, they had less knowledge of the overall statistics of mathematics examination as compared to the teachers. I made a mistake for asking an ill-informed question because I assumed that the parents knew the answer. Thus, this question was removed. Although the parents did not mention