Study 1: Case Study

A case study of a reverse engineering situation was first conducted to understand and narrow down potential elements of conceptual and procedural knowledge and to identify where they originate. The study investigated the task of reverse engineering a program using the OllyDbg debugger [140] to determine the situational factors involved in reverse engineering an executable program.

A case study was needed as part of the overall methodology because the variables of interest as to how people make sense of executable programs were not clear at the outset of the research. The case study is also needed because there is no detailed examination of situational factors involved in making sense of executable programs found in the research literature (see Chapter 2). The case study presents analysis

of video and audio data captured from screen recordings of a reverse engineering task performed by the researcher, analysis of verbal data taken from the task, and an investigation of information and affordances in the task environment used by the researcher in the task.

3.3.1 Motivation for Using a Case Study Method. Case study methods are used often in exploratory research because they excel at helping researchers uncover variables and potential causal relationships in a domain of interest [211]. A case study is ideal for answering “how” and “why” questions and for gaining deeper insight into the nature of a problem and into complex issues [18].

Case study methods have a number of defining characteristics which set them apart from other research methods. Hitchcock and Hughes [89] characterize case study methods as those in which the researcher is “integrally involved in the case” and which provide “rich and varied descriptions of events” that involve the chronological relation of relevant events, the combination of descriptive and analytical views of the events, focus on the individual actor and the actor’s interpretation of events, and which indicate specific events in the case for attention.

Case studies attempt to portray the experience of being a part of the particular situation [78]. Benbaset, et al. [18] investigated case study methods in information systems and found that researchers use case study research to provide answers to problems “in which research and theory are at their early, formative stages.” Case study research was also found to be appropriate for “sticky, practice-based problems where the experiences of the actors are important and the context of action is critical” [18].

3.3.2 Relation to the Research Questions. The problem of understanding how people make sense of executable programs involves situational and contextual factors, and theory in this domain is at an early stage of development. The case study was chosen for the research problem in this dissertation because the problem involves

primarily exploratory research [48]. In exploratory research, case study methods provide tools to allow the researcher to develop and build theory at the same time [18] and to become as familiar as possible with the domain and the experience of the participants while maintaining an objective stance [48].

Solving the research problem of the dissertation involves understanding the con- ceptual and procedural elements involved in reverse engineering tasks. The verbal record and the model of the situation can be used to determine conceptual elements. The case study was the most appropriate method to generate this type of in- formation because theoretical models of mental processing in reverse engineering ex- ecutable programs do not exist (see Section 2.6 for a further discussion). Since the variables of interest and causal relationships had not been established in the liter- ature, it was not yet appropriate to attempt to apply experimental and statistical techniques [211, 18].

Standing on its own, the case study method would provide information that was useful in exploring possible ways that people can make sense of executable programs. However, the findings in the case study were also used to develop the organizational foundation for the rest of the dissertation. The case study data were used to develop the interview questions used in the semi-structured interviews (Section 3.4) and in the design of an observational study (Section 3.5). The case study data were also used to define the characteristics of the program the participants would reverse engineer and to determine what data to collect (see Section 3.5 for discussion of the design of the observational study).

3.3.3 Unit of Analysis. In a case study, a case can be a series of events or situational context [211]. In many case studies such as in Bryant [32], the case has an organizational context which provides the backdrop for decisions, motivations, goals, and more. In this case study, however, the context is a problem-solving situation rather than an organizational context.

Case studies can be characterizes as either exploratory, descriptive, explanatory, or improving types of designs [158]. Case studies can also be characterized as either holistic or embedded and comprising either single cases or multiple cases [211]. Holistic case studies are those in which the case comprises the primary unit of analysis, while embedded case studies are those in which multiple units of analysis are studied within a single case [211].

The method used in this study is an exploratory embedded case study design aimed at understanding the situational context in which reverse engineering is per- formed. In this embedded case study, the task environment, the task, and knowledge components are the embedded units of analysis within a single case. The case in this study is a situation in which a single participant (the researcher) reverse engineers a type of program called a crackme from assembly language instructions.

Data collection methods in case studies can comprise either first degree, or direct, methods which include examples and think-aloud protocols; second degree, or indirect methods, in which the researcher does not interact with participants; and third degree methods, in which the researcher studies available data compiled from elsewhere [118]. This case study involves first degree data collection through direct observations, a think-aloud protocol, and analysis of the crackme program and the OllyDbg debugger.

The data collection methods in this case study involved the following:

• One participant (the researcher) completed a reverse engineering task and ver- balized throughout the process of solving the challenge in a manner similar to a “think-aloud” protocol.

• Video data from one of the three sub-tasks in the reverse engineering task was analyzed to infer a sequence of task actions from observable information. • The researcher’s concurrent verbalizations during the problem-solving process

were recorded and transcribed into a text document.

• The task environment and crackme program were analyzed to provide additional information about the situation.

3.3.4 Analysis. Qualitative data analysis is used to analyze the results from this case study. Two types of qualitative data analysis are hypothesis genera- tion techniques and hypothesis confirmation techniques [176]. Hypothesis generation involves the coding, “memoing,” and classification of the data, followed by examina- tion of patterns and relationships in the coded data [163]. In hypothesis generation, the researcher should not define “too many hypotheses” before the analysis has be- gun [176, 163]. Instead the hypotheses are determined through an iterative process of analysis, pattern recognition, and interpretation. Hypothesis confirmation tech- niques, such as negative case analysis, triangulation, and replication, are then used to establish hypotheses that have been generated [176, 163].