Research began with a survey of software engineering research and trade publications that treated concepts related to error and failure. The outcomes of this exercise, selections of
which are reported in Chapter 2 had three immediate, significant methodological implica- tions for the research design.
First, the methodological approach that was initially, naively, proposed was to be ethnographic in that it would examine the topic of error after immersive fieldwork within a single company. The survey of empirical research quickly revealed that with some exceptions (Prior, 2011), long-term, open, immersive access to developers is rare (Easter- brook, Singer, Storey, & Damian, 2008).
Second, at the outset, the starting point for analysis was assumed to be source code and records related to bugs because this is what developers produce and this is where errors are reified (Avižienis, Laprie, & Randell, 2004). It seemed reasonable to assume software could and should be read for evidence of the “tinkering” that goes on during development, to get at an understanding not only of how, but of why it works as it does (Mahoney, 2008).
This is an approach that has been used to good effect, demonstrating among other things how developers navigate within source code (Lawrance et al., 2013), how they engage with APIs in companies (de Souza, Redmiles, Cheng, Millen, & Patterson, 2004) or how programmers use comments to organise and communicate aspects of ongoing work (Storey, Ryall, Bull, Myers, & Singer, 2008).
However other studies demonstrated clear failings in the software records that are kept about error (Aranda & Venolia, 2009), that matched calls for future research consistently made in root-cause analyses. Root-cause analyses studies largely draw upon bug and maintenance reports. Errors that appear in early stages of a project, with less experienced programmers, or after a “hectic period of changes” (Endres, 1975, p.328) are not well represented. Studies have recommended that data about errors should be collected from
should not be collected too long an interval of time after events have passed (Perry & Stieg, 1993).
Third, the root-cause studies consistently suggested that future research should examine “human erring”, including factors such as problems of understanding (Endres, 1975, p.331), inexperience (Perry & Evangelist, 1987), lack of information (Perry & Stieg, 1993), and skill mismatch (Leszak, Perry, & Stoll, 2002). This call matches recent interest to counter technically “saturated” curricula in software engineering with examinations of of engineering process as a “human activity”. (Capretz, 2014).
This review led to three decisions:
• Fieldwork would have to be undertaken opportunistically, in multiple
environments.
• Examination should establish a fuller chronology for error by examining activities throughout the development cycle.
• In order to respond to the call to examine “human erring”, individual experience should be the focus of analysis.
4.1.1 The Ethical Impetus
Other concerns shaped the research design. Data is never “pure” (Hammersley & Atkinson, 2007), but contamination seemed to be of particular concern in the context of error. Developers might change their behaviour if they were watched (Hammersley & Atkinson, 2007). Spoken to after a period of observation, they might swagger or boast in their responses (Hammersley, 2003) and not be credible. Organisations might not grant access or treat developers who agreed to partake poorly after the fact.
To address these worries, the responsibility of beneficence as described by Vinson and Singer (Vinson & Singer, 2008) and vulnerable stirrings (Behar, 1997) provided the best guidance. Researchers need to consider potential harm toward companies, ensuring, for
example, that important trade secrets are not disclosed. This can generally be managed in the way findings are reported. Beneficence toward informants is not always so straightfor- ward.
Social research can change the environments in which it is conducted and it can have effects on the people and cultures that are examined (Hammersley & Atkinson, 2007). Outcomes can be put to uses after research is completed that researchers cannot control (Spradley, 1979). These factors were of particular concern during the early stages of this research. Depictions of “incompetent” developers in the software engineering research invoked the spectre of Reason, finger extended:
“For those who pick over the bones of other people’s disasters, it often seems incredible that these warnings and human failures, seemingly so obvious in retrospect, should have gone unnoticed at the time. Being blessed with both uninvolvement and hindsight, it is a great temptation for retrospective observers to slip into a censorious frame of mind and to wonder at how these people could have been so blind, stupid, arrogant, ignorant or reckless.” [Emphasis added] (Reason, 1990, p. 214).
Thus chastened, the aim was formed to find a way to perform an analysis of error that would keep ethical concern for developers at the fore. Credible sources of data were sought that would allow observations of practice and interviewing but in which in the research presence would not be considered a threat. Methods were sought to encourage developers to be open and straightforward in their behaviour, and also to ensure that they would not be censured by colleagues for doing so.
The next sections describe how these aims were met, first by establishing epistemologi- cal commitments to using ethnographic principles. Next a description is given of gathering material from multiple sites. The process of organising data into sets for analysis is described, and an overview is given of the methods used in individual studies to build up a