Semi-Structured interviews

To test our intuition about the usefulness of simple screen-help awareness, viewing developers’ screen without the ability to pause and replay, and buffered screen-help awareness, replaying developers’ screens, in promoting help giving, we focused on the intern/mentor scenario and interviewed eight subjects - four mentor/intern pairs - in a large organization. The subjects had professional experience ranging from 3 months to 20 years. Some of the subjects were not employed as programmers, but programming played a major part of their job role. To determine the usefulness of simple and buffered screen-help awareness, we asked participants if they could give examples of simple and buffered screen-help awareness would have been useful in previous help giving interactions. To give participants a concrete idea of how simple and buffered screen-help awareness would be implemented, we showed them a prototype that replays the audio and video of meetings [34].

All subjects liked the general idea of simple and buffered screen-help awareness. For example, one intern reported that he liked that his mentor would be able to watch over his shoulder when he needed help. Similarly, one mentor reported that the tool would eliminate the overhead of scheduling multiple project status meetings with interns.

5.2.2 User Study

Encouraged by the results of the interviews, we decided to perform studies that evaluated simple and buffered screen-help awareness. A field study of simple and buffered screen-help awareness would give the most reliable evaluation, but requires robust implementations and long-term usage of these implementations. Therefore, we decided to do a lab study to motivate such implementations and usage.

There are many ways to simulate developers and observers in a lab study. One approach is to use “live” developers and observers, wherein the latter monitor developer activities as they are performed. This was the approach used in [11] in a game playing activity designed to support help giving. However, this approach has several problems when applied to traditional software development. First, as getting stuck is a rare event, this means an observer must wait or perform some other activity irrelevant to our study until the next stuck point. Enabling a subject to monitor the activities of multiple developers can ameliorate this problem, but the observers and developers must be scheduled at the same time. More important, the study must involve multiple developers and subjects to reduce the impact of subject and observer idiosyncrasies, and ways must be found to make them familiar with the developers’ tasks and give the observers some expertise so they can play a potential helper (mentor/teacher) role.

(a) (b) (c)

Figure 5.1: “Stuck Point” Video Observation Tool simulating two modes: (a) buffered and (b) simple with (c) answer interface.

An alternative simulation approach – the one we followed – is to perform a two phase study in which subjects play the role of developers in the first phase and observers in the second phase. In the first phase, developers’ screens are recorded and the stuck points marked. In the second phase, observers are shown screen recordings of other subjects using simulations of the simple and buffered screen-help awareness. All developers are given the same problems in the first phase. As a result, when they play the role of observers in the second phase, they are

familiar with the problems and have some expertise to solve them. As mentioned below, our developers used different languages and created different solutions for the same problems. Moreover, several months elapsed between the two phases. Thus, arguably, this approach simulates the teaching assistant/student mentor/protégé scenario where the former is helping the latter with a similar problem they have solved before.

Yet another issue is what it means for observers to determine if they can help. A reliable solution is for them to actually develop a working solution, but without the code base it is not possible for them to do so, and more important, this approach is overkill as our help giving model assumes that after making this determination, they communicate or work with the developer to help them get unstuck. Therefore, we asked them to instead outline the solution, and used two coders and I to check if it is correct. A problem with using only this approach in a lab study is that sometimes the observers lacked certain knowledge (such as the file API of a programming language with which they were unfamiliar) to solve the problem. Rather than have them search for this information, which is unrealistic if they were real experts, we asked them to also describe the problem the developers were having. If the coders found this description to be correct, then, we assumed that with the right knowledge, they were in a position to help. We built a special tool for these coders’ evaluation (Figure 5.2).

We used problems from the Mid-Atlantic ACM programming competition as in chapter 3. In the first phase, fourteen participants solved the three ACM programming problems while using the programming-activity difficulty-detection component. We recorded more than 40 hours of developer activities. I identified 16 stuck points in these recordings based on the inferences made by the help awareness mechanism and manual monitoring of developers’ actions, which were confirmed by two coders.

The second phase involved ten of the fourteen initial subjects who were able to participate again. To simulate screen and help awareness, we created a stuck point video

observation tool shown in Figure 5.1. This tool only shows a five minute video segment of each developer’s stuck point. We chose five minutes segments after doing some pilot studies because participants were able to determine the fix to a problem, on average, within five minutes.

Each participant viewed the stuck segments under one of two conditions: either without rewind and pause (simple) or with rewind and pause (buffered). Participants did not view their own stuck points. For each participant, we assigned conditions to tasks randomly. We trained participants using one of the 16 stuck points, leaving 15 total stuck points. Two observers did not view 11 stuck points because of time constraints.

Participants were given an unlimited amount of time to describe a problem and determine how to fix it. The help determination time was calculated based on the time users pressed a button to view the video minus the time users pressed a button to submit the fix and explanation (Figure 5.1).

Figure 5.2: Coder Agreement Tool.