INTERVIEW ANALYSIS METHODOLOGY
Subjects in this study were interviewed for one hour about their understand- ing of a wide range of concepts in digital logic design. Due to time con- straints, each participant was interviewed on only a portion of the selected concepts. The interview questions resembled problems that the subjects may have encountered previously in a digital logic course.
4.1
Subjects
In Spring 2008, Fall 2008, and Spring 2009, I interviewed nine undergradu- ate students, six undergraduate students, and eleven undergraduate students, respectively, at the University of Illinois at Urbana-Champaign. Consistent with standard qualitative research practice [86], more interview subjects were added until interviews ceased to reveal more misconceptions. All students were recruited from two large, three-credit digital logic courses: one each in the Department of Computer Science and the Department of Electrical and Computer Engineering. Both courses were taught by instructors who had taught their respective courses for multiple semesters and have been rated highly by their students, used the same textbook [87], and used similar syl- labi. Each course had about 200 students per semester. Both courses were lecture based and administered online homework assignments, weekly paper- based homework assignments, simulation labs, two midterm exams, and a final exam. All interviewed students were traditional age (18-22) under- graduates majoring in computer science, electrical engineering, or computer engineering who had just completed one of the digital logic courses and had earned grades of B or C (from 1.7 to 3.3 on a 4.0 scale). Most interviews took place shortly after students’ final examinations. These students were selected because their understanding was likely to be less complete than stu-
dents with higher grades (i.e., more likely to have misconceptions). Pilot interviews confirmed these expectations.
4.2
Interview Process
Students were interviewed for one hour. Interviews were conducted in a modified “think-aloud” format: Students were instructed to vocalize their thoughts as they solved problems and responded to questions [75]. Prior to the interview, students were briefed on the study’s goal of understanding how they think through various topics in digital logic. They were told to not expect feedback during the interviews about whether their answers were correct, but to expect frequent requests to elaborate on what they were doing [75].
All interviews were recorded using a document camera (which recorded only what the student wrote) and microphone. The audio tracks of the interview recordings were transcribed verbatim, the students’ gestures were included in the transcript, and every piece of paper the student wrote on was scanned electronically. Quotations presented in this dissertation have been “cleaned-up” to remove excessive “likes,” “ums,” and repeated phrases. Clean-up was performed only when removing these artifacts did not change the content of the statement. For example, the quotation “State? State is [pause] like, the state in a circuit is [pause] where you’re currently at. What values [pause] so if you have [pause] like in, some sort of [pause] datapath or something, you have certain values,” will be presented as “The state in a circuit is where you’re currently at. So if you have some sort of datapath or something, you have certain values.”
Students were paid for their participation, and all students gave writ- ten consent to be interviewed under IRB approval (University of Illinois at Urbana-Champaign number 07026).
4.3
Interview Questions
All students were asked questions that spanned four main topics in digi- tal logic: number representations, Boolean logic, medium-scale integrated
circuits, and state and sequential circuits. Each semester, I interviewed stu- dents using a slightly different set of questions based on their analysis and findings from the previous round of interviews. The interview questions for each topic will be introduced in their corresponding chapters.
4.4
Data Analysis
Grounded theory is a research paradigm that is intended to facilitate the generation of theories through rigorous, inductive analysis grounded in the data rather than in established theories [88]. Researchers first analyze the data independently without a predetermined coding scheme, and develop a coding scheme based on the data. This independent analysis allows theories to emerge from the data without an a priori theoretical framework that in- appropriately influences the observations. Forgoing an initial coding scheme also allows for fuller descriptions of what subjects did correctly or incorrectly. Later, the researchers collaborate to check the reliability and validity of the coding schemes.
To our knowledge, there have been no formal, systematic investigations on students misconceptions about digital logic concepts. Hence, we cannot assume that existing learning theories and research apply directly to student learning in digital logic. Grounded theory’s emphasis on the analysis of data without a guiding theoretical framework provides an appropriate research paradigm for the project.
The interviews were analyzed using a four-step interpretation of grounded theory and qualitative data analysis as described by Kvale [89], Strauss and Corbin [88], and Miles and Huberman [86]. Four researchers helped to ana- lyze the data at various points in the project: a former instructor of a digital logic course (Michael C. Loui), a colleague with content knowledge in digi- tal logic (Craig Zilles), a researcher with extensive experience in qualitative research methods (Lisa Kaczmarczyk), and I.
Step (1) - To avoid bias, we analyzed all interviews regardless of per- formance. The analysis of the interviews was divided into four sub-studies — one for each topic. For each sub-study, we analyzed only the portions of the interviews that pertained directly to the topic for the sub-study. For example, if a student talked about Boolean expressions when asked about
medium-scale integrated circuits, we evaluated the statements only as they pertained to medium-scale integrated circuits.
Step (2) - All researchers analyzed the interviews independently without a predetermined coding scheme, as prescribed by grounded theory [88].
Step (3) - Three or four researchers met and discussed every annotation and observation that they had made. To ensure the accuracy and complete- ness of our coding, a unanimous decision was needed for an annotation to be included for coding or rejected from coding. If a unanimous decision was reached, then it was counted as an agreement; otherwise it was counted as a disagreement. Preliminary code names and definitions were created for every accepted annotation.
Step (4) - After all interviews were discussed, I refined the preliminary code names and definitions with the assistance of Lisa Kaczmarczyk to fa- cilitate the identification of thematic patterns. The refined list of codes and definitions was given to all four researchers to identify the thematic elements of the codes independently. All researchers then met again to discuss the thematic elements that they had noted. A unanimous decision about the presence of a theme was needed for it to be included in the final list of themes.
This process was repeated for subsequent rounds of interviews. The themes identified during the earlier rounds of interviews were used to better inform our construction of interview questions and our analysis for the later rounds of interviews.
4.5
Codes
Through the process described in Section 4.4, we identified codes and themes for each of the four topics. The codes were divided into two primary cate- gories — Actions and Conceptions.
An inter-rater reliability of 95% was calculated as follows:
R = An/(An+ Dn), (4.1)
where R is inter-rater reliability, An is the total number of agreements, and
The conception codes are of most direct interest to this research, because they indicate the misconceptions that can be used to create the concept in- ventory. These codes also help to gauge the relative difficulty of different concepts. While the action codes will not be used to create the concept in- ventory directly, these codes offer additional insights about why the subjects had these misconceptions. Therefore, they provide guidance towards instruc- tional interventions to help students overcome their misconceptions. These codes also demonstrate the expertise level of the subjects.
4.6
Terminology
This section defines terminology that is used for the remainder of the dis- sertation. The term student describes any person who has recently learned digital logic or is currently learning digital logic. The term subject describes any student who participated in the interview portion of the study. All sub- jects are given pseudonyms such as “Subject 1.” Because each subject was not interviewed about every topic, the number of subjects that are described in each chapter varies. Consequently, subjects’ pseudonyms may change from chapter to chapter.
Additional terminology specific to each topic will be introduced in the appropriate chapters.