Following the Model

Six root-cause studies that follow a research model like the one used by Endres’ are profiled and reviewed in the text that follows. See Table 2.2 for a summary of the studies.. As in Endres' case, the studies primarily examined data drawn from bug and modification reports filed by users and testers (Leszak et al., 2002; Perry & Evangelist, 1985, 1987).

Other studies drew data from in-process questionnaires (Basili & Perricone, 1984) and retrospectively administered surveys (Perry & Stieg, 1993). The study design in one case was experimental and examined purpose-built software (Schneidewind & Hoffmann, 1979). In the other studies, an empirical examination was made of software written for industrial environments, in a variety of languages and for different operating systems.

Taxonomies that represent the root causes of errors were the primary tool used in the analysis. Some schemes were theoretical, designed a priori by the researchers (Basili & Perricone, 1984; Schneidewind & Hoffmann, 1979) or developed in collaboration with members of the development team (Perry & Stieg, 1993). The taxonomy used in one study developed out of an analysis of error data (Perry & Evangelist, 1985, 1987). A second study used a scheme created earlier, adapting and extending it to represent additional information (Leszak et al., 2002). Developers were asked to classify errors using tax- onomies supplied by researchers in two cases. Basili and Perricone included their classifi- cation in a change report form completed by programmers. Perry and Stieg surveyed programmers responsible for closing modification reports asking them to classify the error into one of nine fault type categories and to indicate the phase of testing in which the fault emerged.

Classified collections of faults provided a lens for examining other features of software such as complexity, interface defects or environmental factors that influence software dependability. Complexity was found both to correlate to error frequency (Schneidewind & Hoffmann, 1979) and not to (Basili & Perricone, 1984). Application programming interfaces were found to have particularly high frequencies of errors associated with them (Perry and Evangelist, 1985, 1987). These and other root causes were interpreted according to the costs of finding and fixing (Basili & Perricone, 1984; Leszak et al., 2002, Schnei- dewind & Hoffmann, 1979).

Schneidewind & Hoffmann (1979) Basili & Perricone (1984) Perry & Evangelist (1985, 1987) Hypotheses/Aims Hypothesis: Program structure has a signifi- cant effect on error making, detection, and correction.

Aim: To find a com- plexity measure that can be used to guide program de-sign and resource allocation in debugging and testing. Aim: To analyze the relationships between environmental factors and errors reported during software devel- opment and mainte- nance.

Hypothesis: Interfaces are a source of prob- lems in the develop- ment and evolution of large system software.

Characteristics of Data 173 errors 64 errors deemed to be potentially relevant to complexity of structure 231 change report forms, created by pro- grammers over a peri- od of 33 months. Reports were verified by team manager, vali- dated by research team;

New development, but existing code re-pur- posed in some cases 94 randomly selected modification reports submitted by testers 85 contained sufficient data for the study Software evolution.

Characteristics of Software

Four projects un- dertaken by the same programmer Algol W for exe- cution on the IB- M360/67 Purpose-built code. Approximately 90,000 lines of code Primarily in For- tran for execution on an IBM 360 Aerospace (satel- lite planning stud- ies). 350,000 non-com- mentary source lines C programming language

Fault reports writ- ten against global header files. Domain unreport- ed, researchers af- filiated with Bell Labs and MCC.

Table 2.2: A summary of root-cause analyses. The research model established by Endres was also used in other root cause analyses. This table gives information for six such

studies, highlighting study aims, characteristics of study design, and the environment under investigation.

The studies, like Endres’ converge on one point: knowledge is one of the largest problems in software development (Perry and Stieg, 1993). However, the findings represent the notion of conceptual integrity in different ways. Endres described it as problems of understanding. The other studies conflate reasoning with constructs taken from software engineering. For example, Basili and Perricone found that roughly half of all errors related to requirement and functional specifications. Perry and Evangilist noted that 25 percent of the interface errors they studied were due to issues in design (Perry and Evangilist, 1987, Section 2 Background for the study).

The aim to go beyond the source code and to “get at” the reasoning process of develop-

Perry & Stieg (1993)

Leszak, Perry & Stoll (2002)

Aim: To determine general and application specific encountered during software evolu- tion.

Aim: To determine problems are found. Aim: To determine when problems are found.

Aim: To analyze defect modification reports; establish root causes. Aim: To analyze cus- tomer-reported modi- fication reports Aim: To propose im- provement actions to reduce critical defects and to lower rework cost

Total sample size unre- ported.

68% of surveys were returned in each of two surveys

Software evolution.

427 Modification Re- ports representing 13 domains (func- tional units of software) New development (51%) and evolution.

1,000,000 non- commentary source code lines, distributed real- time system writ- ten in C on UNIX Telecommunica- tions (AT&T). 900,000 non-com- mentary source code lines

Language and en- vironment unre- ported

Telecommunica- tions (Lucent)

survey for their case study that included a section for identifying the “underlying causes” of design and coding errors. Examples of categories included "Ambiguous design" and "Knowledge incomplete". All members of this category represented difficulties related to maintaining conceptual integrity (Perry & Stieg, 1993). Supporting Endres’ findings, their analysis indicated that lack of information dominated the underlying causes of the errors, while knowledge-intensive activities such as code inspections dominated the means of prevention.

Commentary about developer proficiency figures strongly, if indirectly in the studies. Schneidewind and Hoffman noted that their scheme was superior because it captured the

flawed ”mental processes” of the programmer in representing ideas within source code

(Schneidewind & Hoffman, 1979, p.282-283). Perry and Evangilist gave several causes for their error categories related to human performance, including several mentions of

inexperience (Perry & Evangelist, 1987). Leszak et al. reported that a mismatch between

the technical skills required and those available among workers is often the root cause of faults (2002). Echoing Hoare and the recommendations of Endres, Perry and Stieg concluded that process should be altered to include ”non-technological, people-intensive means of prevention” (Perry & Stieg, 1993).

In conclusion, on close reading the papers reveal that to fully understand why errors are made, information must be gathered about human understanding – where it is lacking, how it is coordinated and maintained (Leszak et al., 2002). The studies led by Perry and Leszak conclude with suggestions for follow-up work using methods to investigate the human element of errors, but only Endres’ discussed in any detail the generative qualities of error. As Endres argued, programming is a human activity shaped by an inner life of motivations and mental processes, of personal strategies developed to manage the work of program- ming. The sources of errors must, therefore, be considered not with regard to correction of

faults, but instead to intended implementations and subsequent outcomes (Endres, 1974, p. 329).