Will a pilot interview be conducted? There are two purposes for pilot interviews: one is to conduct a practice interview for the interviewer, and another purpose is to get feedback from

the respondent, which could be negative or positive. I used a pilot interview to get started, and received both negative and positive feedback. I selected a former female student with whom I felt comfortable, but who was no longer in my class. I had reservations about interviewing a young woman alone in my office, so I met her in the school cafeteria. I also had reservations about using a tape recorder because I thought it would be too intrusive, so I planned to take notes. The negative feedback was clear and immediate. At our first meeting in the cafeteria, the noise level was so high that I could see her lips moving, but could not hear what she was saying. In

_________________________________________________________________________________

Griffee 164

addition, since it was a public place, a friend of hers asked to join us, soon followed by a faculty friend who wanted to chat. For the next interview, I changed our interview location to my office. After two interviews during which I took notes, I decided that I was losing too much information, and for future interviews I needed to record and fully transcribe each interview. The positive feedback was that my pilot interview supplied me with a major hypothesis for the evaluation, one of which I was previously unaware, namely that students were actively discouraged from speaking in class. This was news to me because the official position of the school is to run most of its classes on a seminar style. The pilot interview not only gave me feedback about the noise level and interview location, but also supplied me with a rival hypothesis that I was eventually able to confirm in interviews with other students.

What kind of data typically result from an interview?

The most typical data are the words that convey the thoughts and ideas of the respondent. But various kinds of questions can produce other kinds of data. For example, it is possible to collect numerical data by asking a series of true-false questions, “how many” questions, or “how often” questions. It would also be able to count certain linguistic features.

How is interview data typically analyzed?

There are currently several assumptions regarding interview data. One assumption is that interview data exist in a complex relationship; they are not simply a “product” from the respondent. In other words, interview data are not just the words the respondent tells us. Imagine you are conducting an interview. If it were the case that the interview consisted of only what your respondent said, we would have to pretend that you did not exist. In fact you do exist, your assumptions exist, your biases exist, and your questions exist. What constitutes the interview is co-created by you and the respondent. It is not just the words coming from the respondent and recorded and transcribed.

Another assumption underlying interview data is that the words from the interview constitute raw data. Raw data alone do not tell us anything, they must be interpreted. Hitchcock and Hughes (1995) describe two strategies for analyzing interview data, including several specific steps. These strategies are: 1) Become very familiar with the data, and 2) Create meaning by using analytical categories. The first strategy, becoming familiar with the data, occurs—depending on how the data was collected—by going over notes many times, listening to recordings repeatedly, or constantly reading and rereading the interview transcripts. One creates meaning by the use of categories. There is some difference of opinion (perhaps a difference of approach would be a better way of putting it) about how these categories are best created.

One approach is to become very familiar with the data, and as a result, categories “emerge” or become apparent. As we look at the data, we begin to see that our respondent was talking about theme A, theme B, and so on. Pondering these themes, we finally come to understand (interpret) that our respondent is talking about X. In this approach, the categories are “grounded” in the data; that is, categories or themes emerge from the data and reflect the data. We don’t impose our will (biases) on the data, but rather let the data speak to us.

_________________________________________________________________________________

165 Data from Interviews

The second approach is to create the themes and categories before the interview takes place. In discussing how to conduct good interviews, Wolcott (1995, p. 115) says that behind every question should be a hypothesis. That is, we are not just asking questions randomly, rather we have some idea of what we are asking and why we are asking it. This is especially true in the case of research and program evaluation. The more exploratory your research is, the more you hope for grounded categories to emerge from the data. The more you know what you are looking for, the more you will rely on categories chosen prior to the study. For example, in the study reported, I had previously decided that I was looking for both students’ and teachers’ opinions on particular categories, such as the role of the textbook in learning. In each interview, I included a question on what the respondents thought of the textbook. But even with my interest in preselected categories, additional categories that I had not anticipated and was not looking for emerged from the data. I followed up on them in subsequent interviews.

Here are some examples of data analysis, based on Miles and Huberman (1994, p. 55). One problem in interview analysis is moving from a fairly large amount of raw data (the interview transcripts) to the meaning of what has been said. This is not only a process of data analysis, but of data reduction. We want to go from pages and pages of words to what is important.

Step one. Listen to the recording and transcribe the interview.

Step two. Read the transcripts several times to familiarize yourself with what is being said. Step three. Code the interview. Coding entails reading the transcript until certain themes become apparent. Identify each theme with a short word or phrase. This word or short phrase is the code. After you have your codes, define them so you can be consistent in coding across multiple interviews. For example, in coding a teacher’s interview, I used several codes including “G” and “B” which stood for “grammar” and “block.” I defined grammar as “references to grammar and syntax as a goal of the class or object of classroom teaching” and block as “any reference to what is bothering or hindering the teacher.” Go through the transcript and mark or circle places in the transcript where the respondent discusses the theme, and write the code in the margin. I use colored markers so I can see the themes quickly. After doing this for the entire interview transcript, you have coded the interview transcript.

Step four. Write a summary of the coded data. For example, on a piece of paper (or word processing document) write the code, and under each code list what the respondent said. For example, under the code “grammar” I put two comments, one of which was “Grammar is the main context of the course.” Under the code block I put seven comments, one of which was “Course grammar book not related to academic writing.” I then had reduced several pages of transcribed interview data down to one and a half pages of comments under various codes. I also knew what I believed to be the number of comments made by the teacher under each code. So, for example, I knew that the teacher commented on grammar twice, but on blocks seven times.

Step five. Write a memo to yourself. Miles and Huberman (1994, p. 72) suggest writing yourself a memo that not only summarizes, but ties together the themes and compels you to write what you think it means. This last step was the most important, because what I wrote in the memo turned out to be what I learned from the interview.

_________________________________________________________________________________

Griffee 166

How to calculate interview reliability

One definition of reliability is stability of data. According to LeCompte and Goetz (1982, p. 35), reliability refers to the extent to which studies can be replicated, According to Kvale (1996), reliability refers to how consistent results are. Some qualitative researchers have developed a parallel vocabulary. For example, Guba and Lincoln (1989, p. 242) adopt the term dependability, which they maintain is parallel to reliability. Dependability is concerned with the stability of data over time.

Interview data often take the form of words, and ideas that can be coded for content by someone called a rater. A second rater can look at the same content and code it. It is then possible to compare the consistency of the two raters and refer to this rater agreement as reliability. Miles and Huberman (1994, p. 64) offer the formula: reliability equals the number of rater agreements divided by that number of agreements plus the number of disagreements.

Reliability = agreement agreement + disagreement

For example, in a needs analysis study, six ESL instructors in a summer program were interviewed. The researcher sought a general sense of how the ESL instructors understood students’ problems. The researcher audio-taped the interview, produced transcripts, and coded places in the transcript where the researcher thought the instructors were identifying student problems. These problems, 23 in all, were then coded by the researcher as language problems, teaching problems, or cultural problems. Their transcripts were examined by a second rater who coded the same problems for the same three categories. On 18 of the problem areas the raters agreed, but on 5 they did not. According to the reliability formula supplied by Miles and Huberman (1994), the reliability was 18 divided by 18 plus 5, or 18 divided by 23 for a reliability of .78. Expressed in the earlier formula:

18 = .78

18 + 25

The two raters discussed the five disagreements and finally agreed on three of them resulting in a formula of 21 agreements and 2 disagreements for a final reliability of .91, which is a high level of inter-rater reliability.

Sources of unreliability

Cohen, Manion, and Morrison (2000, p. 121) suppose that sources of unreliability reside in the interviewer, the questions, and the respondents. This list can be expanded to include other parts of the interviewing process such as interview location, environmental factors, status equality, length of the interview, and topic threat. These eight possible sources of unreliability are discussed here, along with possible strategies to improve reliability.

1. The interviewer The interviewer, either individually or as part of a team, comes to the interview