Acceptance Test Framework (Site C)

This project is extending an open-source acceptance test framework to allow users to specificy "literate" acceptance tests. Wiki-based and written in JAVA, the framework was designed to allow non-technical users to specify and run acceptance tests for software. The altered project will permit tests to be written that follow the Given-Then-When pattern (North, n. d.). Classes and packages will be named so that they can be parsed and present- ed to readers on webpages in a form that approximates natural language. Likewise, users of the framework will be able to create tests with names that are readable and easy to understand.

Development draws upon a JAVA interface written by Marcus some months prior to filming. That project had two aims: to test out ideas about writing human object oriented application programming interfaces (API), and to support the separation of roles and tasks in behaviour driven development frameworks. It included examples which demonstrated the use of the API in conjunction with the open source acceptance test framework that is being altered by the pair. The API is used directly at points, and examples included in its documentation are referred to and borrowed from.

6.2.1.1 Informants: Marcus and Joe

Marcus and Joe, identified in Table 6.1 below, perform all programming tasks together. The two are more or less evenly paired, each has been programming professionally for over ten years. Both programmers are familiar with the acceptance test framework, however Marcus appears to have more recent experience in developing within it.

Table 6.1: Informant demographics, Site C.

By contrast, Joe exhibits greater familiarity with the tools that are being used, in particular with the IDE and a continuous unit testing plugin for the IDE. Evidence is given that he takes the lead on performing upgrades on these tools between filming. There is also evidence to suggest that he is an advocate for using Linux or Unix variant operating systems, and that his most recent development work has been done in Cocoa.

Watchers

The episodes that were analysed were filmed in office environments, as depicted in Figure 6.1, and there are frequently people co-located in the room where development is happen- ing. Watchers predominantly follow along with the programming action, but also comment from time to time. At times, their input affects the work. Watchers differ between episodes; no single Watcher is consistently present.

Episodes were webcast using web meeting software and Watchers also participate via the internet. People drop in and out of the sessions, at times commenting or asking questions in real-time via chat. When this happens, a co-located Watcher brings the question to the attention of Marcus and Joe, or one of the developers notices that a question has been asked in real-time chat. In both cases, the question is addressed in the course of ongoing work.

6.2.1.2 How Practice is Organised

Marcus and Joe use the Eclipse integrated development environment (IDE), and create extensions to the wiki-based acceptance test framework. They also use the wiki to direct

Site

Site C

Name

Joe Marcus

Gender and Age

Male, thirties Male, thirties Experience Professional Con- sultant, 10 years Professional con- sultant, 10 years

their work, writing stories within it that define the functionality they intend to add to the framework. The wiki environment is viewed in Firefox.

Figure 6.1: Development sessions were held in offices.

Figure 6.2: Filming depicted a screencast. After episode 20, the screencast included video of the developers at work, prior to that, only the IDE or web browser output was visible. This figure also depicts a screen explaining test-driven development principles followed by

The developers take turns driving and navigating. One writes a unit test, defining proposed behaviour for a class, and the other implements the behaviour by adding necessary methods to the class. The driver often informally “thinks aloud” to indicate the actions he is taking. Likewise, the navigator often acts as narrator for the audience, explaining in broader terms what is being done, and how it is oriented within the larger goals for the project. In addition, the two interact with each other, discussing the work that is being performed.

Development is undertaken on a Windows laptop owned by Marcus (see Figure 6.2 for a representative image of the screencast depicted in video recordings). In the first and subsequent episodes that were analysed, the performance of this laptop distracts the developers and slows progress.

6.3 Findings

As described in Chapter 3, Section 3.3, error handling is generally described as a three- stage process (Brodbeck, Zapf, Prümper, & Frese, 1993). A person must know that an error has occurred, identify both what was “done wrong” and “what should have been done” and then understand how to “undo” the effects of the error (Sellen, 1994, p. 476). Handling unfolds in the course of “progressive” problem solving. An error is suspected or detected, and an evaluation is made to identify the source of the problem (Allwood, 1984). Environ- mental cues supply feedback to the problem solver by blocking forward progress (Norman, 1981), communicating about problems in system state (Lewis & Norman, 1986) or by circumstantially guiding a problem solver to recovery (Reason, 1990).

Following this rubric, features of error are illustrated in the findings that follow using statements and exchanges of dialogue drawn from error handling incidents. Forty-three incidents were examined in detail; a diagram depicting the spread of incidents across episodes can be seen in Figure 4.4. Taken as a group, incidents are notable for the time

they span: many are fewer than five minutes in length with clearly defined points of detection and resolution. Other handling processes took much longer, with the longest taking approximately fifty minutes and spanning two videos. In most cases, incidents are resolved on camera within a single video episode, though some span multiple films and several incidents are resolved off camera. For a fuller description of the methods used in analysis, see Chapter 4, Section 4.4.2

Incidents were selected for reporting because they represent a cross section of the various kinds of development tasks that characterise encounters with error. They also illustrate different aspects of handling. Finally, they are incidents that are brief enough to be presented in total or near total entirety, allowing the reader to form a sense of how error handling unfolds from start to finish.

Excerpts of dialogue from videos are presented to illustrate aspects of handling. Dialogue that does not pertain to the immediate topic has been removed for brevity and clarity. Exchanges are presented in italics, with the name of the speaker in a strong font. A catalogue of incidents is given in Appendix C.2. Full exchanges are given in Appendix C.3. For fuller transcription conventions, see Appendix A.

6.3.1 Slips of Action

Actions sometimes do not go as planned, or were not intended. They are often simple, routine, and are commonly detected in the act based on perceptions that arise while doing something (Sellen, 1994). Often described in software engineering in terms of backtrack- ing (Bowdidge & Griswold, 1997), they could also be described as slips of action (Norman, 1981). Selecting the wrong item from a drop down menu or improperly referenc- ing a variable are two examples:

Joe: No can't do that cause it's. Oh we can move it outside the [try block ]... (Ep. 7,

00:06:51)

In these cases, the full exchanges for which can be viewed in Appendix 6.3.1, each developer gives a clear indication that something is wrong. What should have been done is evident, and recovery is simple. It is likely that Marcus caught his error in the act. Detection is also commonly made by assessing outcomes, and Joe’s statement suggests that he may have responded to effects his actions had on the development environment.