Evaluating the Reading Techniques

The above design will provide answers to our questions for evaluating reading techniques (described in section 1.3.1) through the following analyses:

EQ1. Is the technique effective in the environment?

This question can be broken down into more specific questions that are directly answered by this study:

• Does the use of PBR improve fault detection over an ad hoc process?

• Does the use of PBR with error abstraction improve detection effectiveness over the use of abstraction alone?

• Does the use of PBR with error abstraction require more reviewer time than the use of error abstraction alone?

• Do team meetings with PBR have a different effect than team meetings for which it has not been used?

• Is error abstraction useful for defect detection (with and without PBR)?

• Is team discussion of errors (with and without PBR) feasible? Is it useful?

Unfortunately, the design presented in section 3.4 confounds the difference due to the techniques with any difference due to the level of difficulty of the two documents. That is, if document 2 is easier than document 1, then the improvement that we would expect to see for reviewers on the second document will be impossible to separate from any improvement due to PBR. However, this situation is unavoidable given the constraints of a classroom environment, as explained in that section.

We feel that this interference will actually be small because our experience on previous studies has shown the two documents to be very close in terms of review difficulty, as measured by the detection rates observed for each document. Over the two studies run at the University of Maryland, detection rates for reviewers using their usual technique averaged 24.5% and 21.5% on the PG and ATM documents, respectively. For subjects using PBR, the averages were 27.6% and 30.8%.

We can also use a variety of measures to detect if the error abstraction portion of the technique provides any benefit. Part of this will come from a qualitative analysis of the subjects’ experience, in order to detect if there were benefits added in areas we were not expecting, such as getting more familiar with the document, or being better able to repair it. We can also undertake an analysis of the number of faults discovered, to see if the experience with error abstraction uncovered new faults that had been missed by the fault detection procedure.

EQ2. Is the technique at the right level of detail?

We can obtain an answer to this question by comparing the quantitative and qualitative data for this experiment with the previous experiment in PBR, to determine if the more specific procedure does in fact provide any benefit. Since we used the same documents in both studies, we can perform a quantitative analysis of whether or not a different number of defects were found when the technique was made more specific. However, since we also collected some data in the previous study on subject satisfaction, we can also contrast this with the current subjects’ satisfaction. For example, we can address important questions

such as whether the procedure, in being more detailed, was actually too constraining and less enjoyable to use. (The data collection forms and questionnaires, with which we intend to measure subjects’ satisfaction with the new techniques, are found in Appendix E.)

This question is also pertinent to the error abstraction process. Because we view the abstraction process as an inherently “creative” one, in that reviewers have to make subjective decisions about what categories usefully describe the faults discovered, we do not provide a very detailed technique. We should analyze whether the use of the abstraction procedure leads to a common or similar set of errors for all reviewers, or whether the results are completely dependent on the individual reviewers (that is, every reviewer produces a significantly different list of errors). If it is the second case, then we should consider providing more detail in this process. Otherwise, one of the potential advantages of error abstraction (facilitating communication about requirements defects) would be negated.

EQ3. Do the effects of the technique vary with the experience of the subjects? If so, what are the relevant measures of experience?

As always, we will use questionnaires and background surveys to assess the experience of subjects in many related areas. Once the fault and error lists have been analyzed we will test for any correlation between these experience measures and the observed performance. (The background surveys are also included in Appendix E.)

EQ4. Can the technique be tailored to be effective in other environments?

We propose to address this question by continuing our usual approach of creating web-based lab packages, which have been described in some detail in section 1.2.4. An initial lab package has already been created and is available at:

http://www.cs.umd.edu/projects/SoftEng/ESEG/manual/error_abstraction/man ual.html.

Even though this is only a draft of the finished product, we consider this to be a promising lab package in that it has already supported replications of the experiment in a number of different environments (in Philadelphia, Italy, Brazil, and Sweden). As the analyses of these experiments are published, we will use them to build hypotheses about the larger area of reading techniques, as we have used the Kaiserslautern and Trondheim replications of the original PBR experiment.

As we continue to work on this lab package, we will use some of our lessons learned from the first lab package to improve this version. These lessons come from feedback that we received from other researchers concerning what they would have liked to see in the first PBR package. Potential changes may include:

• Specifying the experimental procedures in a way more easily used by other researchers in their own replications (by adapting the guidelines of the American Psychological Association, which has developed a standard approach for human-centered experiments in psychology);

• Incorporating the results from replications of the study directly into the lab package, so that the package can serve as a “clearinghouse” of all related knowledge on the subject;

• Including alternate file formats that are more easily modifiable by other researchers for their own replications;

• Including additional training materials that can be used to train subjects in the technique being investigated.