DEVELOPMENT AND EVALUATION OF VALUE-BASED REVIEW (VBR) METHODS. Keun Lee

(1)

DEVELOPMENT AND EVALUATION OF VALUE-BASED REVIEW (VBR) METHODS

by Keun Lee

---

A Dissertation Presented to the

FACULTY OF THE GRADUATE SCHOOL UNIVERSITY OF SOUTHERN CALIFORNIA

In Partial Fulfillment of the Requirements for the Degree DOCTOR OF PHILOSOPHY

(COMPUTER SCIENCE)

May 2006

(2)

UMI Number: 3237109

3237109 2006

UMI Microform Copyright

ProQuest Information and Learning Company 300 North Zeeb Road

P.O. Box 1346

Ann Arbor, MI 48106-1346

(3)

ACKNOWLEDGEMENTS

First of all, I would like to dedicate this dissertation to God. Even though I had been a Christian, I didn’t know God before coming to USC. God gave me a chance to meet him and let me know how the grace of God had led me to get this far. I finished this dissertation; however, I knew it would be impossible for me to achieve Ph.D. without God.

I would also like to thank my family and Good Shepherds members. My parent had supported me with encouragements and prays always. I refreshed when I had great time with GS members especially for SeungHyun, HaeReem, YoungKyung and SaeEun.

I am also deeply grateful to my Ph.D. advisor Dr. Barry Boehm. He has not been great mentor in my research but also be my role model of research attitude. I really appreciate his advice and look forward to continue relationship in the future. My gratitude also goes to my committee members, Nenad Medvidovic and Bert Steece. They gave valuable comments on my research so that I can update my dissertation.

Last but not least, I really thank to all my friends who prayed for my research including YuJung, SungA, and SeungWon.

(4)

LIST OF TABLES

Table 1-1 Comparative Business Cases: ATG and Pareto Testing... 5

Table 1-2 Comparative Business Cases: Value-neutral review and Value-based review... 8

Table 2-1 Example of checklist ... 14

Table 4-1 Differences between the LCO and the LCA... 38

Table 4-2 Assignments in the Experiment ... 58

Table 5-1 T-test results and percent Group A less than Group B ... 68

Table 5-2 Comparison of Group A and Group B: Average number of Concerns and Problems... 70

Table 5-3 Comparison of Group A and Group B: Mean Impact based on Concerns and Problems... 71

Table 5-4 Comparison of Group A and Group B: Mean Number based on Concerns and Problems of OCD ... 74

Table 5-5 Comparison of Group A and Group B: Mean Impact based on Concerns and Problems of OCD ... 76

Table 5-6 Comparison of Group A and Group B: Mean Number based on Concerns and Problems of SSRD ... 80

Table 5-7 Comparison of Group A and Group B: Mean Impact based on Concerns and Problems of SSRD ... 81

Table 5-8 Comparison of Group A and Group B: ... 85

Table 5-9 Comparison of Group A and Group B: Mean Impact based on Concerns and Problems of SSAD ... 87

Table 5-10 Comparison of Group A and Group B: Mean number of concerns and problems per hour ... 89

(7)

Table 5-11 Comparison of Group A and Group B: Mean Cost

Effectiveness of concerns and problems per hour... 92

Table 5-12 Comparison of Group A and Group B: Mean Number per hour based on Concerns and Problems of OCD... 94

Table 5-13 Comparison of Group A and Group B: Mean Cost Effectiveness based on Concerns and Problems of OCD... 96

Table 5-14 Comparison of Group A and Group B: Mean Number per hour based on Concerns and Problems of SSRD ... 97

Table 5-15 Comparison of Group A and Group B: Mean Cost Effectiveness based on Concerns and Problems of SSRD ... 99

Table 5-16 Comparison of Group A and Group B: Mean Number per hour based on Concerns and Problems of SSAD ... 100

Table 5-17 Comparison of Group A and Group B: Mean Cost Effectiveness based on Concerns and Problems of SSAD ... 102

Table 7-1 The P-values from the T-tests... 106

Table 7-2 The P-values from the T-tests of OCD ... 108

Table 7-3 The P-values from the T-tests of SSRD ... 109

Table 7-4 The P-values from the T-tests of SSAD ... 110

Table 8-1 Open_Days_IDFix on defects (Accumulated number of defects)... 115

(8)

LIST OF FIGURES

Figure 1-1 Pareto 80-20 distribution of test value ... 5

Figure 1-2 ROI: Value-Neutral vs. Pareto Analysis ... 6

Figure 1-3 value-neutral review vs. value-based review (Pareto Analysis)... 9

Figure 1-4 Result chain of VBR ... 10

Figure 1-5 Research Approach ... 11

Figure 2-1 General idea of DBR ... 16

Figure 2-2 General idea of PBR... 17

Figure 2-3 The process of developing scenario for PBR ... 19

Figure 2-4 UBR processes ... 23

Figure 3-1 VBR processes ... 27

Figure 3-2 The general Value-Based checklist ... 29

Figure 3-3 Example of artifact-oriented checklists... 30

Figure 3-4 OCD section 4.3 System Capabilities ... 32

Figure 3-5 Effectiveness metric; Issue metrics and optimality guidelines ... 35

Figure 4-1 Experiment Process ... 36

Figure 4-2 Independent Verification and Validation in CSCI 577A ... 49

Figure 4-3 The Process to collect Data ... 59

Figure 4-4 AAR form (Concern Log) ... 61

(9)

Figure 4-6 AAR form (Problem List) ... 63

Figure 4-7 Example of AAR form (Problem List)... 64

Figure 5-1 Number of Concerns and Problems by IV&Vers... 65

Figure 5-2 Impact of Concerns and Problems by IV&Vers... 66

Figure 5-3 Effort Comparison... 67

Figure 5-4 Number of concerns and problems found by IV&Vers in both groups... 70

Figure 5-5 Impacts of Concerns and Problems in both groups... 72

Figure 5-6 Average number of issues of OCD ... 73

Figure 5-7 Average impact of issues of OCD... 75

Figure 5-8 Accumulated Average number of OCD issues ... 77

Figure 5-9 Accumulated Average impact of OCD issues... 78

Figure 5-10 Average number of issues of SSRD ... 79

Figure 5-11 Average impact of issues of SSRD ... 80

Figure 5-12 Accumulated Average number of SSRD issues ... 82

Figure 5-13 Accumulated Average impact of SSRD issues ... 83

Figure 5-14 Average number of issues of SSAD... 84

Figure 5-15 Average impact of issues of SSAD ... 86

Figure 5-16 Average number of SSAD issues ... 87

Figure 5-17 Accumulated Average impact of SSAD issues ... 89

Figure 5-18 Number of concerns and problems found by IV&Vers per hour in both groups... 90

(10)

Figure 5-19 Cost Effectiveness of concerns and problems per hour in both groups... 92 Figure 5-20 Average number of issues per hour of OCD ... 93 Figure 5-21 Average impacts of issues per hour (Cost Effectiveness) of

OCD ... 95 Figure 5-22 Average number of issues per hour of SSRD ... 96 Figure 5-23 Average impacts of issues per hour (Cost Effectiveness) of

SSRD... 98 Figure 5-24 Average number of issues per hour of SSAD ... 99 Figure 5-25 Average impacts of issues per hour (Cost Effectiveness) of

SSAD ... 101 Figure 8-1 Definitions of Open_Days... 114 Figure 8-2 The percentage of accumulated number of defects by their

(11)

ABSTRACT

Reviewing is a key activity that can find defects at an early stage of system and software development. Since it is often cheaper to fix defects at an early stage, reviewing is a good technique for improving both product quality and project cost effectiveness. Currently, there are many review techniques proposed and many experiments have been performed to compare the review techniques. However, to data there have been no review techniques or experiments that have focused explicitly on the relative business value or mission value of the artifacts being reviewed.

In this dissertation, I provide Value-based review techniques adding cost effectiveness and value of each issue into review processes, and report on an experiment on Value-based review. I developed a set of VBR checklists with issues ranked by success-criticality, and a set of VBR processes prioritized by issue criticality and stakeholder-negotiated product capability priorities. The experiment involved 28 independent verification and validation (IV&V) subjects (full-time working professionals taking a distance learning course) reviewing specifications produced by 18 real-client, full-time student e-services projects. The IV&V subjects were randomly assigned to use either the VBR approach or the previous value-neutral checklist-based reading (CBR) approach that had been

(12)

used in the course. The difference between groups was not statistically significant for number of issues reported, but was statistically significant for number of issues per review hour, total issue impact in terms of criticality and priority, and cost effectiveness in terms of total issue impact per review hour. For the latter, the VBRs were roughly twice as cost-effective as the CBRs.

The dissertation also covers threats to validity and limitations of the experiment. Threats to validity were present but appear to have been adequately addresses. The main limitation of the experiment was its coverage of reviews by individuals as compared to groups. For reviewers by groups, it is likely that combinations of VBR and risk-driven forms of perspective-based review (PBR) approaches would be most cost-effective. This and other topics are attractive candidates for further research.

(13)

CHAPTER 1 OBECTIVE AND APPROACH

1.1 Motivation

1.1.1 Software Quality and cost effectiveness

Increasing software quality is a common objective for software engineers, however, the goal is not easy to achieve and there have still been many research efforts addressing how best to decrease defects and increase quality in software. Basically, there can be three main strategies for decreasing defects in software: defect prevention, detecting defects and fixing them, and reducing impacts of defects.

Automated analysis, human reviewing, and execution testing are the main

methods to detect and fix defects. The methods have their own characteristics, and based on the project situation, they can be used selectively or together.

Another issue for software engineers is cost effectiveness. There are four factors in developing software to be considered always: functionality, cost, quality and schedule [BIFF01]. Cost effectiveness is a primary concern of organizations in finding the best combination of four factors. Extensive research has been done on techniques to assess and to reduce the cost, but less has been done on techniques to assess and increase the benefits or cost effectiveness. However, when business

(14)

cases or stakeholder negotiations of software capabilities have been performed, information is available to assess benefits and cost-effectiveness.

1.1.2 Review techniques

The review is one of the main processes to find defects from the initial

development stage and increase the quality of the deliveries [BOEH01, FAGA76, GILB93]. Peer reviews have been used in requirements analysis, architecture, design and coding. Many research efforts have focused on formulating effective review processes to find defects [SHUL02, AURU02]. Studies have addressed review team composition and procedures, review preparation and duration, and criteria for focusing reviewers on sources of defects.

Initial approaches for focusing reviewers involved adding checklists in the review process [FAGA76]. With checklist-based reviewing (CBR), it is easy to

understand the process and CBR has become the most common review-focusing techniques in current practice.

Another approach to reviewing artifacts is perspective-based reading (PBR). This focuses on different reviewers’ perspectives, such as designer perspective and tester perspective [BASI96]. Different review perspectives help to find more

(15)

defects without overlaps. Another reading technique is defect-based reading (DBR), which focuses on different defect classes [PORT95]. Other review techniques proposed are functionality-based reading (FBR) [ABDE04], and usage-based reading (UBR) [CONR03, THEL03A]. A number of studies have compared the effectiveness of these techniques [SHUL02, AURU02, ABDE04, DENG04, BERL04]. They agree in finding that focused review methods do better than unfocused reviews; that methods’ cost effectiveness vary by the nature of the artifacts being reviewed; and generally, that the preferred method to use is

situation-dependent. However, with the exception of some uses of PBR and UBR, the cost-effectiveness metrics used for the methods and their evaluation have been value-neutral in that each defect is considered to be equally important. This causes much review effort to be spent on trivial issues like obvious typos and grammatical errors.

1.1.3 Comparison of value-based methods and value-neutral methods

Chapter 1 of the book, “Value-Based Software Engineering” [ARUE05] gives an example comparing the use of value-based and value-neutral methods for defect detection. The example describes a $2 million software project to develop a large customer billing system. The proposal of an automated test generation (ATG) tool from a vendor claims that:

(16)

1. ATG will cut the test cost in half.

2. Typical test cost is 50% of total development cost (or $1M). 3. Cost for using the ATG is 30% of typical test cost (or $300K). 4. The benefit of using the test tool is $500K - $300K = $200K.

The experience paper [PERS04] shows that ATG might not reduce the cost as much as the vendor proposed. The reasons for the failure are unrepresentative test coverage, too much output data, lack of management commitment, and lack of preparation and experience. An additional serious concern is that ATG tools, along with many other software engineering methods, processes, and tools, are value-neutral.

However, in many practical applications, the relative value of requirements, test cases, and defects have a Pareto distribution. The Pareto distribution shows that 80% of the mission value comes from 20% of the software components. Figure 1-1 shows the Pareto curve from the experience report (Bullock, 2000). In this application, each customer billing type tested improved initial billing revenues from 75% to 90% and reduced customer complaint rates. However, just one of the 15 customer types accounted for 50% of the business revenue, and the rest

(17)

result of value-neutral ATG-driven testing in which case each test is equally likely to affect business value.

20 40 60 80 100 5 10 15

ATG – all test have equal value Actual business value

Cumulative

Business Value (%)

Customer Billing Type

Figure 1-1 Pareto 80-20 distribution of test value ATG Testing Pareto Testing % of

Tests Cost(K) Value(K) Net Value(K) ROI Cost(K) Value(K) Net Value(K) ROI

0 1300 0 -1300 -1 1000 0 -1000 -1 10 1350 400 -950 -0.7 1100 2560 1460 1.33 20 1400 800 -600 -0.43 1200 3200 2000 1.67 40 1500 1600 100 0.07 1400 3840 2440 1.74 60 1600 2400 800 0.5 1600 3968 2368 1.48 80 1700 3200 1500 0.88 1800 3994 2194 1.21 100 1800 4000 2200 1.22 2000 4000 2000 1

Table 1-1 Comparative Business Cases: ATG and Pareto Testing

(18)

neutral ATG testing and value-based Pareto testing strategies. Figure 1-2 compares the ROIs.

2 1.5 1 0.5 0 - 0.5 -1 -1.5 20 40 60 80 100 ROI % Tests Run ATG Pareto

Figure 1-2 ROI: Value-Neutral vs. Pareto Analysis

The assumptions of the analysis are as follows:

z $1M of the development costs has been invested in the customer billing system by the beginning of testing.

z The ATG tool will cost $300K and will reduce test cost by 50% as promised.

z The business case for the system will produce $4M in business value in return for the $2M investment cost.

(19)

z The business case will provide a similar 80:20 distribution for the remaining 14 customer types.

Table 1-1 and Figure 1-2 show the comparative results of using ATG tool and Pareto testing. As seen in Table 1-1, and Figure 1-2, the highest ROI is at 40% tested point for Pareto testing, however, it is at 100% tested point for ATG. If testing focuses on higher values first, the highest ROI is much better, and is achieved at an earlier stage.

This example comprises the value-based method and the value-neutral method. Even though it is an example for testing tool, testing is another primary process to increase software quality. If the results of the example are applied to the

reading/review technique, similar results are produced.

I will now compare a value-neutral review technique and value-based review technique. The assumptions are as follows:

z $1M of the development costs has been invested in the customer billing system by the beginning of reviewing.

z The value-neutral and the value-based review techniques will cost 200K each.

(20)

z The business case will provide a similar 80:20 distribution for the value-based review.

Other assumptions are the same as with the testing comparison. Table 1-2, Figure 1-3 show ROIs for value-neutral and value-based reviews.

ATG Testing Pareto Testing

% of review

ed Cost(K) Value(K) Net Value(K) ROI Cost(K) Value(K) Net Value(K) ROI

0 1000 0 -1000 -1 1000 0 -1000 -1 20 1040 800 -240 -0.23 1040 3200 2160 2.08 40 1080 1600 520 0.48 1080 3840 2760 2.56 60 1120 2400 1280 1.14 1120 3968 2848 2.54 80 1160 3200 2040 1.76 1160 3994 2834 2.44 100 1200 4000 2800 2.33 1200 4000 2800 2.33 Table 1-2 Comparative Business Cases: Value-neutral review and Value-based review

Table 1-2 and Figure 1-3 show comparative results of using a value-neutral review and a value-based review. For a value-based review, the highest ROI, 2.56, is achieved between 40% and 60% reviewed point. However, the highest ROI, 2.33, for value-neutral review is achieved at 100% reviewed point. For a value-based review, beyond the 60% reviewed point, $80K of review investment produces only $52K in business value. This result shows that a value-based review can be worthwhile at an early stage of the review process.

(21)

2 1.5 1 0.5 0 - 0.5 -1 -1.5 20 40 60 80 100 ROI % Reviews Run Value-neutral Value-based

Figure 1-3 value-neutral review vs. value-based review (Pareto Analysis)

1.2 Objective

The primary objective of the research was to develop a value-based review technique and evaluate its relative cost effectiveness with respect to current value-neutral. An experiment was performed to evaluate the VBR with respect to the value-neutral checklist-based review (CBR) technique having been used previously.

(22)

Developing VBR process Developing VBR checklist Value-Based Review Increase cost effectiven ess More effort spent on

higher-value issues

Better understanding of the higher-criticality defects

Figure 1-4 Result chain of VBR

Figure 1-4 shows the result chain of VBR. The desired outcome is increasing cost effectiveness in the reading technique VBR. The contribution to this outcome of an initiative to develop a VBR process is to focus effort on higher–value issues. The contribution of investing in value-based review checklists to this outcome is to increase understanding of which defects are most critical to identify. In order to validate that investing in the initiatives would indeed lead to the desired outcome; an experiment was designed to evaluate VBR, by comparing it with the CBR guidelines that had been used in previous years.

(23)

1.3 Approach

Design Value-Based Review

Experiment Related Research

• Checklist Based Reading • Defect Based Reading • Perspective Based Reading • Functionality Based Reading • Usage Based Reading

• VBR concepts • VBR process • VBR checklist • Effectiveness Metrics • Experiment Design • Experiment Result • Experiment Discussion • Experiment Conclusion

Figure 1-5 Research Approach

The next few sections describe the research approach and results. In section 2, several common and popular review/reading techniques are surveyed including, reading processes, characteristics, strengths, pitfalls, and costs. In section 3, based on the survey, a VBR process and associated VBR checklists are developed. New concepts, main processes, and critical success factors are considered in this step. In section 4, the experiment that evaluates the VBR is described, and aspects of the results discussed in section 5-7. A summary of the research approach is shown in Figure 1-5.

(24)

1.3.1 Review techniques

To improve upon ad hoc reviewing, several review/reading techniques have been developed. Checklist-based reading (CBR) is a common reading technique that is used in many industrial fields. Perspective-based reading (PBR) is organized around multiple review perspectives; several empirical reports comparing CBR and PBR have been published. Some particular perspectives include Defect-based reading (DBR), functionality-based reading (FBR), and usage-based reading (UBR). These review/reading techniques are compared with each other and with value-based reviews in section 2.

1.3.2 Value Based Review

Based on research of review/reading techniques, a value-based review method is developed. It includes a basic VBR process and a set of value-based checklists. Its general concepts include priority and criticality. The priority and criticality are a basis used to calculate the value of artifacts. There are two checklists for VBR: the general value-based checklist covers general issues, and artifact-oriented checklists are prepared to review particular artifacts.

(25)

1.3.3 Experiment

The experiment was designed to compare the use of VBR using value-based procedures and checklists with a value-neutral CBR approach. The two approaches were performed by 28 randomly-selected professional software engineers taking a graduate software engineering project course by distance learning at USC. Their project assignment was to independently verify and validate (IV&V) the artifacts from one of 18 real-client e-services project applications that were being developed by the on-campus teams in the course [BOEH98].

(26)

CHAPTER 2 RELATED RESEARCH ON REVIEW/READING

TECHNIQUES

2.1

Checklist Based reading (CBR)

Checklist-based reading (CBR) is a reading technique using checklists to read artifacts. The checklist is provided as part of the evaluation process for Quality Assurance. The checklist helps ensure that the appropriate items have been covered for software elements. Usually, the checklist is applied to requirements, design, and code artifacts. Questions in a checklist are to review completeness, consistency, feasibility and testability issues. A checklist gives what reviewers should read but it doesn’t provide how to read the artifacts. Table 2-1 shows an example of a checklist used as a part of a value-neutral CBR in the experiment.

Architecture Design Checklist - Completeness Have all TBDs been resolved in requirements and specifications?

Are all of the assumptions, constraints, decisions and dependencies for this design documented?

Is the logic correct and complete?

Is the conceptual view of all composite data elements, parameters, and objects documented?

Questions

Is there any data structure needed that has not been defined and vice versa?

Table 2-1 Example of checklist

The strength of CBR is that it is easy to understand and thus additional efforts to train in CBR are not necessary. In addition, it is considered to be superior to an ad

(27)

hoc reading technique. Today, CBR is a common reading technique used in the industrial fields so that there are many experiments comparing other reading techniques with CBR.

The weakness of CBR is the passive attitude for reviewers to read artifacts. Generally, there are not formal steps or procedures to follow. Reviewers read artifacts following questions in checklists. And when the reviewer finds any issue which is not pass questions in checklists, (s)he report the issue. Therefore

reviewers do not need to read artifacts with active attitude to find issues.

2.2

Defect Based Reading (DBR)

Primary objective of Defect-based reading (DBR) is developing a strategy for identifying defects in requirement documents. DBR focuses defect classification to find defects in artifacts and a scenario is a key factor in DBR. The scenario of DBR is derived from defects classification. Based on defect class, a set of questions is droved from the defect class. The questions are characterizing the defect class, and also guiding the reviewers to read artifacts. The difference between scenario and checklist is scenario has more specific information and guidelines for how to read the artifacts. The figure 2-1 shows the basic idea of DBR.

(28)

Checklist Defect Class Scenario Defect-based reading Developing questions Designing Model

Figure 2-1 General idea of DBR

There have been empirical reports about comparing DBR and the other two reading techniques, CBR and ad-hoc reading [FUSA97, MILL98, PORT95, PORT98, SAND98]. Some reports concluded that DBR has better performance results than CBR and ad-hoc reading techniques [PORT95, PORT98]. Some results from the study by Porter et al. [PORT95] include:

1. The defect detection rate for DBR is higher than the other two reading techniques.

2. Scenarios lead reviewers to focus on defect classes specifically.

3. On the average, collection meetings contributed nothing to defect detection effectiveness.

Other experiments could not find the statistical evidence that DBR is better in finding issues than the others [FUSA97, SAND98]. However, it is clear that CBR

(29)

and ad-hoc reading techniques have no description given as to how the document should be read. DBR is more structured, distinct, and less overlapping than these other two techniques.

2.3 Perspective Based Reading (PBR)

Defect Class Roles Scenario Perspective-based reading Developing questions Identifying activities

Figure 2-2 General idea of PBR

Even though many observers are looking at only one object, the point of view of each observer is different. The interesting on the object is different and the attitude for looking is also different. Different interesting and attitude lead to focusing on different characteristics of the object. This idea is the basis of perspective-based reading (PBR). The assumption of PBR is each reviewer has different particular points of view to look at the artifacts. There are several roles in reviewing such as tester, designer, and user. Different roles cause different

(30)

scenarios for each role are provided, and the reviewer follows the scenarios based on their roles. The scenario includes a set of questions and activities to guide the reviewers on what to perform, and how to read the artifacts. While following the scenario, the reviewer reads the artifacts and finds issues. There are many empirical reports that PBR is more useful to find more issues than CBR, and without overlaps. The general idea of PBR is shown in Figure 2-2. PBR has the following characteristics:

• Systematic: The reader knows how he should read the document

• Specific: The reader is only responsible for his role and the defects that can be detected with regard to his particular point of view.

• Distinct: Each reader has his own role. I assume that there is a broader coverage between them as far as defects from different defect classes are concerned.

The key success factor of PBR is scenario. More practical and precise scenarios will lead to more efficiency of PBR so that the process to build scenarios is important in PBR.

The process for developing the scenario is shown in figure 2-3. First, the main activities of each role are abstracted. Next, based on abstraction, practical activities for the roles are identified. Last, detailed questions and guidelines for each activity are defined.

(31)

Abstraction the responsibility of role

Identification and description of expected activities for each role

Provide questions for each activity based on different defect classes

Figure 2-3 The process of developing scenario for PBR

Dr. Basili, developer of PBR, said in his paper [BASI96], "PBR teams were seen to have an improved (statistically significant at the 0.05-level) coverage of defects for both the generic and NASA documents. It was also shown that individuals using PBR performed better on the generic documents than non-PBR reviewers. There was, however, no significant effect on individual detection rates due to reading technique for the generic documents."

There are some other empirical studies comparing PBR with CBR and ad-hoc reading [CIOL97, SORU97, SHUL98, REGN00, LANU00, BIFF01, HALL01]. Some studies proved that PBR has higher performance results to find issues with statistical significance than other readings [CIOL97, SHUL98]. However, other studies could not prove this [SORU97, REGN00, LANU00, BIFF01, HALL01].

(32)

DBR is usually compared to PBR because of its characteristics. DBR is reading artifacts based on defect classes; however, PBR is based on different perspectives on artifacts, otherwise, the processes of reading are almost similar.

2.4 Functionality Based Reading (FBR)

Functionality-based reading (FBR) has different starting points than other reading techniques. FBR is developed based on framework reading. Framework is an artifact with a particular objective. For example, object-oriented frameworks are artifacts for reuse and extensibility. Currently, there have been many

developments of frameworks for many other domains.

There are empirical reports to compare reading techniques for frameworks. Three reading techniques, CBR, use case-driven reading (UCDR), and systematic order-based reading (SOBR), are compared in the experiment [ABDE04]. The results show there are no superior reading techniques, and the best solution is using the three reading techniques together based on practical situation.

FBR is proposed to trace framework requirements to produce well-constructed framework and review the code. Functionality types and rules are defined to extract the functionalities from the framework. Functionality types are provided as input to FBR by the top-down phase of “framework understanding.”

(33)

Functionality rules describe the objective, description of the functionality, location of the functionality, and correlated functionality. The process of FBR is first understanding the framework based on functionality types, and then

reviewing functionality rules to find defects.

The experiment was performed to compare FBR with two other OO reading techniques, CBR and SOBR. The experiment results show FBR seems to be significantly more effective and productive than CBR and SOBR.

The strength of FBR is that it is very applicable to function based artifacts. It gives formal steps to understand functions and key points to review in an artifact. However, due to the characteristics, it has limitation to apply to other artifact types like object-oriented artifacts.

2.5 Usage Based Reading (UBR)

The idea behind usage-based reading (UBR) is to spend reading effort to find the most critical usage-based defects in the reviewed artifacts. Defect criticality in UBR is prioritized in terms of the users’ perception of system quality. Potential users identify their perception of system quality based on prioritized use cases.

(34)

And other procedures like resource scheduling, meeting, and follow-up are important steps related to UBR.

UBR identifies use cases as to focus review effort. Also, use cases can tell reviewers about practical review plan, as test cases are helpful to tester in making plan for testing.

There are three steps in the UBR. The beginning of UBR is prioritizing the use case. This step is called ‘Before Inspection’. One of key ideas of UBR is prioritizing use cases as a vehicle for a review [THEL03A]. Based on potential users’ opinions, the use cases are prioritized to identify the users’ perception of system quality. Next is to understand design artifacts. Since UBR is performed based on use cases, the understanding on design artifacts is important. Through the step, the reviewers need to utilize the use cases to guide reading and

requirements documents. Requirement documents are used as references to verify the use cases. The last step is inspection, which is inspecting the design artifact. There are four small steps in inspection. Figure 2-4 shows all the steps of UBR.

(35)

Prioritizing Use cases

Understand Design artifacts

Select highest-priority use case

Trace and execute the use case

Review the artifacts pass the exit criteria

All use cases are inspected or time is unavailable? Inspection Design

artifacts

yes

no

Figure 2-4 UBR processes

UBR has some common and also different characteristics of PBR. The common point is utilization of the user perspective. While PBR uses different perspectives like tester, designer, and user, UBR uses only user perspective and prioritized use cases. Another difference is applicable artifacts. PBR is applicable to all artifacts if the scenarios for the artifacts are developed, while the scenario is developed

(36)

based on use cases. The scenario can be used for requirement artifacts, and code inspection in that project.

There have been experiments to compare CBR and UBR and the conclusion is that there are differences between the two reading techniques. It has been shown that UBR had better performances than CBR.

Based on experiments, it is proved that reviewers applying usage-based reading are more efficient and effective in detecting the most critical faults from a user’s point of view than reviewers using checklist-based reading [THEL03B, THEL04].

(37)

CHAPTER 3

VALUE-BASED VERIFICATION &

VALIDATION (VBV&V)

3.1 Concepts in VBV&V

The basic idea of VBV&V, as with other value-based software engineering (VBSE) activities, is to treat each V&V activity (analysis, review, and test) as a candidate investment in improving the software products [AURU05]. Earlier VBSE activities involve prioritizing the capabilities to be developed. This enables VBV&V activities to be sequenced by priority and criticality. Generally there are many stakeholders in a project. Each stakeholder has different win conditions so that priorities of each stakeholder are also different. Through negotiations and meetings with all stakeholders, priorities of system capabilities are determined. The priorities values determine values of system capabilities. The values of priority are High, Medium, Low or 3, 2, 1 in the experiment. Another factor to determine the value of a project is criticality. The criticality is how critical an undiscovered defect is to the project’s success. Generally, the values of criticalities are furnished by domain experts, but reviewers can determine the values in special circumstances. Since the Model-Based (System) Architecting and Software Engineering (MBASE) approach is used to develop artifacts in a project, MBASE experts determined the levels of criticality for the MBASE-related defects. The values of criticality are also High, Medium, Low or 3, 2, 1 in

(38)

the experiment. The numerical values of priority and criticality are used to guide V&V activities and evaluate their effectiveness.

3.2 Value Based Review

With the aid of other MBASE experts, I have developed an experimental set of value-based checklists for reviewing general and MBASE-specific specifications. These are used to test the hypothesis that review activities will be more cost-effective if review effort is focused on the higher-priority system capabilities and the higher-criticality sources of implementation risk [LEE05]. The USC Center for Software Engineering (CSE) spiral approach to systems and software

engineering used in MBASE emphasizes risk-driven activities, but CSE previous review guidelines made no distinctions in focusing limited reviewing resources on high vs. low priority and criticality review artifacts.

The revised approach to overcome the weaknesses in the previous approach includes the priority of system capabilities and criticality of sources of risk. Low-priority, low-criticality items are optional to review based on time availability. The basic process for the Value-based review can be divided into four steps.

(39)

1. First the reviewers need to determine which system capability will be reviewed first. A negotiation among stakeholders defines the priority of each system

capability.

2. The high-priority system capabilities will be reviewed first. If there are three levels of priorities like high, medium, and low, the system capabilities with high priorities will be reviewed first.

3. At each priority level, high-criticality sources of risk will be reviewed first, basically going down the columns in Figure 3-1 below.

4. After reviewing the issues with higher criticality, the reviewers can address the next lower criticality issues, as time is available. As shown in Figure 3-1, it is optional to go all the way down the columns for the lower-priority artifacts.

Negotiation Meetings Developers Customers Users Other stakeholders Priorities of system capabilities Artifact-oriented checklist Criticalities of issues General Value-based checklist Domain Expert Priority High Medi um Low Critic ality High Medi um Low 1 2 3 4 5 optio nal 6 optio nal optio nal Reviewing Artifacts

Number indicates the usual ordering of review*

* May be more cost-effective to review highly-coupled mixed-priority artifacts.

(40)

Figure 3-1 presents general processes of VBR. The priorities of system capabilities are determined through stakeholders’ negotiations and meetings. Domain experts decide the values of criticality of issues. The combination of two values of priority and criticality determines an order of review. The numbers in the table (right side) in the figure 3-1 shows orders in a review. Higher valued issues are reviewed first, lower one is reviewed later, lower one is optional if review time is not available.

3.3 Value Based Checklist

There are two checklists for VBR. The first one is a general value-based checklist shown in Figure 3-2 and the other is Artifact-oriented checklists. The main difference of Value Based checklists to normal checklists is that Value Based checklists include priority and criticality values in each question or statements in the checklists.

General Value-based checklist

The general checklist covers general issues for reviewing any set of artifacts for completeness, consistency, feasibility, ambiguity, conformance, and risk. Each category has three levels of issue criticality: high, medium, and low.

(41)

•Missing FRD evidence of mitigation strategies for low-probability, low-im pact risks •Missing FRD evidence of m itigation

strategies for low-probability high-im pact or high-probability, low-high-impact risks: unlikely disasters, off-line service delays, missing but easily-available inform ation

•Missing FRD evidence of critical capability feasibility: high-priority features, levels of service, budgets and schedules

•Critical risks in top-10 risk checklist: personnel, budgets and schedules, requirem ents, COTS, architecture, technology

Risk

•Non-misleading lack of conformance with docum ent form atting standards, m ethod and tool conventions, optional or low-im pact operational standards

•Lack of conformance with m edium -criticality operational standards, external interfaces

•Misleading lack of conform ance with docum ent form atting standards, method and tool conventions •Lack of conformance with critical operational

standards, external interfaces

Conformance

•Non-misleading, easily deferrable, low-im pact am biguities: GUI details, report details, error messages, help messages, gram matical errors

•Vaguely defined medium-criticality capabilities, test criteria

•Medium-criticality m isleading ambiguities

•Vaguely defined critical dependability capabilities: fault tolerance, graceful degradation,

interoperability, safety, security, survivability •Critical m isleading am biguities: stakeholder intent, acceptance criteria, critical user decision support, term inology

Am biguity

•Easily-deferrable, low-impact inconsistencies or inexplicit traceability: GUI details, report details, error messages, help messages, gram m atical errors •Medium-criticality shortfalls in

traceability, inter-artifact inconsistencies, evidence of consistency/feasibility in FRD

•Critical elem ents in OCD, SSRD, SSAD, LCP not traceable to each other

•Critical inter-artifact inconsistencies: priorities, assumptions, input/output, preconditions/post-conditions

•Missing evidence of critical consistency/feasibility assurance in FRD

Consistency/ Feasibility

•Easily-deferrable, low-impact missing elem ents: straightforward error m essages, help m essages, GUI details doable via GUI builder, project task sequence details

•Medium-criticality m issing elements, processes and tools: m aintenance and diagnostic support; user help •Medium-criticality exceptions and off-nom inal conditions; sm aller tasks (review, client demos), m issing desired growth capabilities, workload characterization

•Critical m issing elements: backup/ recovery, external interfaces, success-critical stakeholders; critical exception handling, missing priorities

•Critical m issing processes and tools; planning and preparation for major downstream tasks (development, integration, test, transition) •Critical m issing project assum ptions (client responsiveness, COTS adequacy, needed resources)

Completeness

Low -Criticality Issues M edium -Criticality Issues

High-Criticality Issues

•Missing FRD evidence of mitigation strategies for low-probability, low-im pact risks •Missing FRD evidence of m itigation

strategies for low-probability high-im pact or high-probability, low-high-impact risks: unlikely disasters, off-line service delays, missing but easily-available inform ation

•Missing FRD evidence of critical capability feasibility: high-priority features, levels of service, budgets and schedules

•Critical risks in top-10 risk checklist: personnel, budgets and schedules, requirem ents, COTS, architecture, technology

Risk

•Non-misleading lack of conformance with docum ent form atting standards, m ethod and tool conventions, optional or low-im pact operational standards

•Lack of conformance with m edium -criticality operational standards, external interfaces

•Misleading lack of conform ance with docum ent form atting standards, method and tool conventions •Lack of conformance with critical operational

standards, external interfaces

Conformance

•Non-misleading, easily deferrable, low-im pact am biguities: GUI details, report details, error messages, help messages, gram matical errors

•Vaguely defined medium-criticality capabilities, test criteria

•Medium-criticality m isleading ambiguities

•Vaguely defined critical dependability capabilities: fault tolerance, graceful degradation,

interoperability, safety, security, survivability •Critical m isleading am biguities: stakeholder intent, acceptance criteria, critical user decision support, term inology

Am biguity

•Easily-deferrable, low-impact inconsistencies or inexplicit traceability: GUI details, report details, error messages, help messages, gram m atical errors •Medium-criticality shortfalls in

traceability, inter-artifact inconsistencies, evidence of consistency/feasibility in FRD

•Critical elem ents in OCD, SSRD, SSAD, LCP not traceable to each other

•Critical inter-artifact inconsistencies: priorities, assumptions, input/output, preconditions/post-conditions

•Missing evidence of critical consistency/feasibility assurance in FRD

Consistency/ Feasibility

•Easily-deferrable, low-impact missing elem ents: straightforward error m essages, help m essages, GUI details doable via GUI builder, project task sequence details

•Medium-criticality m issing elements, processes and tools: m aintenance and diagnostic support; user help •Medium-criticality exceptions and off-nom inal conditions; sm aller tasks (review, client demos), m issing desired growth capabilities, workload characterization

•Critical m issing elements: backup/ recovery, external interfaces, success-critical stakeholders; critical exception handling, missing priorities

•Critical m issing processes and tools; planning and preparation for major downstream tasks (development, integration, test, transition) •Critical m issing project assum ptions (client responsiveness, COTS adequacy, needed resources)

Completeness

Low -Criticality Issues M edium -Criticality Issues

High-Criticality Issues

(42)

Artifact-oriented checklists

The set of value-based checklists also includes artifact-oriented checklists. Since the experiment had been performed using Model-based (system) Architecting and software Engineering (MBASE), the MBASE-oriented checklist is provided to the reviewers for reviewing the various sections of the following MBASE artifacts: Operational Concept Description (OCD), Requirements Description (SSRD), and Architecture Description (SSAD) specifications.

2

Are there no levels of service goals (OCD 4.4) included as system capabilities?

2

Are simple lower-priority capabilities (e.g., login) described in less detail?

3

Are capabilities traced back to corresponding project goals and constraints (OCD 4.2)?

3

Are capability priorities consistent with current system shortcoming priorities (OCD 3.3.5)?

3

Are capabilities prioritized as High, Medium, or Low?

3

Are there critical missing capabilities needed to perform the system services?

3

Are the system capabilities consistent with the system services provided as described in OCD 2.3?

Criticality Question

2

Are there no levels of service goals (OCD 4.4) included as system capabilities?

2

Are simple lower-priority capabilities (e.g., login) described in less detail?

3

Are capabilities traced back to corresponding project goals and constraints (OCD 4.2)?

3

Are capability priorities consistent with current system shortcoming priorities (OCD 3.3.5)?

3

Are capabilities prioritized as High, Medium, or Low?

3

Are there critical missing capabilities needed to perform the system services?

3

Are the system capabilities consistent with the system services provided as described in OCD 2.3?

Criticality Question

OCD 4.3 System Capability

(43)

The figure 3-3shows an example of artifact-oriented checklist. Since the artifacts are MBASE document, it is called MBASE-oriented checklist. Questions are provided in the checklist to check artifacts. Each question has criticality value which is determined by domain experts. First five questions are with criticality 3, while two questions in the bottom are with criticality 2. Based on the values, reviewers decide an order to check questions.

3.4

Example of Artifact Review Process

Below is part of an adapted OCD artifact from a 2003-04 CS 577 project. There are two main capabilities described in OCD section 4.3. Their priorities were determined through the project’s WinWin negotiation and MBASE guidelines. The example shows how to review an artifact using Value-based checklists. Two Value-based checklists in Figure 3-2, “The general Value-Based Checklist,” and Figure 3-3, “Artifact-oriented checklists”, are used in the example.

<Start of Example>

Figure 3-4 shows a part of artifact OCD section 4.3 “System Capabilities.” In the example, it describes two system capabilities, “Convert Word to XML” and “Error Check.” And some more detail information like descriptions, and priorities of the system capabilities are described.

(44)

Process-03

Used In

High

Priority

The application shell be able to convert SSRD word document to XML file.

Description Convert word 2000 SSRD to XML. Name CAP-01 Process-03 Used In High Priority

The application shell be able to convert SSRD word document to XML file.

Description Convert word 2000 SSRD to XML. Name CAP-01 Process-03 Used In Medium Priority

The application should be able to check errors during conversion. Description Error Check Name CAP-02 Process-03 Used In Medium Priority

The application should be able to check errors during conversion.

Description

Error Check

Name

CAP-02

OCD 4.3 System Capabilities

OCD 4.3.1 Convert Word to XML.

Table 3-3. Convert Word to XML capability

OCD 4.3.2 Error Check

Table 3-4. Error Check capability

Figure 3-4 OCD section 4.3 System Capabilities

When a reviewer reads section 4.3.1, (s)he sees that the priority of capability “Convert Word to XML” is high and reviews the section now. The next step the reviewer follows is checking the issue criticalities for the system capability.

(45)

Figure 3-3 shows the part of the Value-Based Checklist for section 4.3. There are seven potential issues, called Questions in the checklist. The criticalities of four of them are high (the value of criticality is 3); whereas the rest of them are medium (the value of criticality is 2).

When the reviewer reads section 4.3.1, (s)he needs to review high criticality questions first. The first five questions are about consistency with OCD 2.3 and OCD 3.3.5, missing capabilities and prioritizing all with a high criticality value. The reviewer checks the consistency with other sections, missing capabilities and prioritizing. In this case the reviewer finds that OCD 2.3 refers to conversion of SSRD Word files to HTML, and registers an inconsistency concern.

When the reviewer has reviewed all the high criticality questions, (s)he moves onto the next criticality questions. The criticalities of the rest of the questions are medium. So the reviewer reads the rest of the questions. The questions are about description of lower-priority capabilities and levels of service goals. The reviewer finds no concerns about them, and does not register any concerns.

The reviewer also notices several minor in consistencies and grammatical shortfalls (Word 2000 vs. word, shall vs. shell, awkward English). The reviewer consults the General Value-Based Checklist in Figure 3-2, determines that these

(46)

are Low-criticality, non-misleading Ambiguity and Conformance issues, and decides not to register them. Such concerns are often registered when using value-neutral guidelines; they are generally not cost-effective to process.

The reviewer finishes reviewing system capability ‘Convert Word to XML’ (section 4.3.1) and moves onto the next system capability, ‘Error Check’ in section 4.3.2. The reviewer sees that the priority of the capability is medium. The reviewer could defer section 4.3.2 and come back later, but decides that it is easy to review and that it makes sense to maintain review continuity and context. No issues are found with it.

<End of Example>

3.5

Weight of review issues

The main goal of the Value-based review is to increase the cost effectiveness of the review. Review effectiveness metrics involve weighted sums of distinct issues reported, using the impact metrics in Figure 3-5.

(47)

Effectiveness Metric Artifact Priority Issue Criticality H M L H M L 9 6 3 6 4 2 3 2 1 issues

(Artifact Priority) * (Issue Criticality)

=

Generally considered optional to review

Figure 3-5 Effectiveness metric; Issue metrics and optimality guidelines

Each issue reported has a priority value and criticality value. The impact of each issue is the product of its priority and criticality value. For example, if one issue has medium priority and high criticality, the impact of the issue is six from two (medium) times three (high). An issue with high priority and high criticality has value 9. The issue has nine times higher value than an issue with low priority and low criticality. The overall review effectiveness metric is the sum of all the issue impacts. As reviewer finds more number of higher valued issues, the review activity has higher impacts.

(48)

CHAPTER 4

EXPERIMENT DESCRIPTION

The experiment was performed during the fall 2004 semester and is described in this section Figure 4-1 shows the experiment processes that have been performed.

Experiment Preparation Experiment Planning Experiment Operation Experiment Data • Reviewers • Materials • Roles & Activity • Data • Variables • Hypotheses • Design • Extraction • Analysis

Figure 4-1 Experiment Process

Basically there are four steps for the experiment process. The preparation step covers general preparation for the experiment. I identified which artifacts to be reviewed, and what data to be collected. Also, materials for review techniques are prepared before the experiment. After this preparation step, I defined hypotheses to test and identified variables in the experiment. An experiment operation plan was built in the planning step.

(49)

There had been an experiment operation after planning. The operation had been performed in the course CSCI 577A Software Engineering. There were five times collections of data. The collected data were vetted before storing in Database. Final step is analysis of the data. The hypotheses were tested using the collected data.

4.1 CSCI 577A Course: Real-Client Projects with Independent

V&V

The CSCI 577A course is the first half of the USC Software Engineering project course, which focuses on software plans, processes, requirements, and

architectures. This course covers the application of software engineering process models and management approaches for developing plans, requirements, and architecture of large software systems. Students work in teams and apply the Win-Win spiral model [BOEH98] and Model Based System Architecting and Software Engineering (MBASE) guidelines for software engineering of real-client projects [BOEH99]. The three critical project milestones are the Life Cycle Objectives (LCO), the Life Cycle Architecture (LCA), and the Initial Operating Capability (IOC) in MBASE. In the LCO and the LCA, the focus is on requirement analysis and system design, however, coding is the main task in the IOC. The LOC and the

(50)

LCA are done in CSCI 577A and the IOC is performed in CSCI 577B. The main differences between the LCO and the LCA are shown in Table 4-1.

LCO LCA Less structured, with information moving

around

More formal, with solid tracing upward and downward.

Focus on the strategy or “vision” (e.g., for the Operational Concept Description and Life Cycle Plan), as opposed to the details.

No major unsolved issues or items, and closure mechanisms identified for any unsolved issues or items.

Many have sine mismatches (indicating unresolved issues or items).

No more TBDs expect possibly within Construction, Transition, and Support plans. No need for complete forward and

backward traceability

Basic elements from the Life Cycle Plan are indicated within the Construction, Transition, and Support plans.

Many still include “possible” or “potential” elements (e.g., Entities, Components, etc)

There should no longer be any “possible” or “potential” elements.

Some sections may be left as TBD, particularly Construction, Transition, and Support plans.

No more superfluous, unreferenced items: each element either should reference or be referenced by another element. Items that are not referenced should be eliminated, or documented as irrelevant.

Table 4-1 Differences between the LCO and the LCA

The review guidelines in CSCI 577 have been value-neutral. Every artifact and issue was equally important and was reviewed from the beginning to the end with the same weight.

(51)

4.2 Real-Clients Projects in the experiments

This section described 18 real-clients projects in the experiment.

Project: Quality Management Information System Enhancement

There are many techniques to detect the defects from the system. However, currently, QMIS manages only the defect information which are produced by Agile Review technique. The goal of this project is to design and implement an enhanced QMIS system which will provide full Quality Management capability. Firstly, the system shall collect the defects which are detected by Agile Review technique, testing and bug reporting. Secondly, the system shall manage and extract the metadata for future analysis. Moreover, the proposed system not only organizes the data for future analysis, but also provides action guidelines for quality assessment techniques and a data tracking system for current projects.

Project: Interactive USC Maternal, Child and Adolescent Center for

Infectious Disease and Virology

The MCA program which is part of the Keck School of Medicine's Department of Pediatrics provides comprehensive HIV services to approximately 900 patients per year in the Los Angeles area and is the largest center for services to pregnant women with HIV in area. The program has grown dramatically in the last few years and has been very successful in the treatment of vertical transmission of

(52)

HIV---meaning that women with HIV disease do not pass their disease to their children. The Clinic's successes culminated in a visit by UN Secretary General Kofi Annan, in December 2003 where he heralded the model of care as one that should be duplicated around the world.

The primary objective of the project is to develop a website with tools for program development. The website should be interactive, outcome based and include a component for prevention of the HIV disease. It will include successful program strategies, papers written by experts in the field of infectious disease, clinical care guidelines. The goal is to help programs around the world

successfully care for those suffering with HIV.

Project: Online Bibliographies on Chinese Religions in Western Languages

The main objective of the project is to develop the system for management of bibliography online so that the entire corpus is easily available and updating can be done electronically. Main features of the system are

z Chinese Character entry system

z Author submission of entries and abstract of content. z Graphics capability.

(53)

z Cross-referencing to the topic-based table of contents which is now the primary matrix of presentation should be much more thorough than in the print-based versions.

Project: Hunter-Gatherer Anthropological Database Search Engine

The objective of the project is developing a Google-type search engine for the specialized needs of scholars in evolutionary biology, psychology, and

anthropology who will use this database when it is eventually made public. The search engine will have to be able to handle several thousand coding categories, and it will need to do specialized searches involving combinations of categories. It also will need to be searchable either for full texts falling under a given code, or brief summaries of these texts, or both together, and it also will need to be

searchable with or without pictures.

Project: Unified Cost Model

The challenge is to integrate the COCOMO suite of models and provide a single Graphical User Interface that allows the user select which models should be included in the estimate. When these models are used together there needs to be a way to track the scope of the estimate (what WBS activities are being estimated) and the life cycle stages being covered by the estimate.

(54)

Project: Bar-coding for Maternal-Child Virology Research Laboratory,

LAC/USC

The project is developing a barcode reader system compatible with laboratory involved in research on infectious agents like HIV and Hepatitis. Bar-coding reader will enable laboratory personnel to avoid error-prone data entry. Data format generated by barcode reader has to be compatible with a proprietary laboratory data management system (LIS) and allowing for a wide variety of outputs (text files, excel files, scanned documents).

Project: Data Mining of Digital Library Usage Data

Developing automated methods for the collection and normalization of DL usage data, the evaluation of clickstream mining methods and automated construction and prototyping of recommender services. We are presently exploring the applications of log analysis to the formation of networks of complex objects (Buckets) within the framework of data preservation efforts.

Project: Development of Open Source Technologies for Creating a

Correspondence Document Data Center

A user-friendly Correspondence Document Data Center to provide access to pieces of correspondence from many departments within one organization

(55)

(intranet). The Client is extremely interested in creative, innovative approaches (to an otherwise boring database/data center project!) by the students.

The main objective of the project is developing a correspondence data system. The system should be 1) Innovative back-end data exchange architecture and related applications 2) A user-friendly

Project: Requirements Tracking

Currently while a project is underway there is no way to track requirements, their inter-dependencies, and any changes to them besides the documentation.

So the primary objective of the project is developing that is able to track any requirements that are part of a project. The program will need to be able to track dependencies between requirements in system of systems levels of projects as well as changes to those requirements and reasons behind the requirements. It must also be able to interface through the web so that it can be accessed remotely.

Project: NSF Database

Over time, the number of video clips has increased, requiring the need to create some type of data base structure that will allow users to name, locate, retrieve, use, and analyze movement data stored in digital video form.

(56)

1. Video archiving system needs to be designed and created with the

understanding that the end user needs simple ways of retrieving files for student use, maintaining the data base over time with minimal knowledge, and

mechanisms for adding additional video clips over time.

2. Video searching system that will allow end users (faculty and students) to search for video clips on the basis of a pre-defined search criteria.

3. Video viewing system needs to be designed and created that will allow students to view a subset of the video clips and email their analysis using an e-journaling system set up by the USC Center for Scholarly Technology. Students need to be able to synchronize video clips, view them side by side, advance images frame by frame, and draw on selected images as a form of communicating analysis results.

Project: Total Quality Management Application

Through the Intranet, a variety of action items are created for individuals within the Institute or at the University. The primary objective is to develop is a "Total Quality Management" module that tracks action items and people's performance against tasks, helping them and management understand their efficiency,

(57)

The system could potentially be developed within ASP, integrated tightly with our current code, or alternatively as a stand-alone application/COM object(s) with a clear API for interfacing with our business systems.

Project: Automated Reconciliation

Currently, all research administrators across campus are required to reconcile every transaction against the source paperwork supporting it. This is an extremely labor-intensive project, as it is common for research units to have over 100 accounts that must be reconciled every month. By creating a system that would compare the data downloaded from the University’s financial system to the known transactions (documentation for which would be retrieved from several different sources, including user input for paper supporting documentation) and provide us with an exception listing (known transactions that have not yet hit the accounts and transactions hitting the accounts for which we do not have

documentation), our available manpower would be optimized by focusing on researching the exceptions.

The system could potentially be developed within ASP, integrated tightly with our current code, or alternatively as a stand-alone application/COM object(s) with a clear API for interfacing with our business systems.

(58)

Project: STS Database Evaluation

Recently, clients of the project want to transfer their vendor who maintains database, which is based on the Society of Thoracic Surgeons (STS) criteria. The main objective of the project is obtaining an evaluation of our set up and

suggestions for improvement. Also, clients want to have an evaluation of the organization between STS and its vendors regarding maintaining integrity of the software.

Project: Data Mining from Report Files

Clients are looking for data mining from reports application, that allowing for a wide variety of input (text files, excel files, scanned documents). This application would be able to understand the relational nature of reports, versus a flat file. This application needs to be able to handle the common unique formatting techniques, such as column and row wrapping. The application needs to be easily customized for use with new reports, defining keywords, column and row wrapping, so that it can be adapted for a variety of existing and future reports.

The application create techniques for data mining from

DEVELOPMENT AND EVALUATION OF VALUE-BASED REVIEW (VBR) METHODS. Keun Lee