This section describes several metrics found in literature, which seem useful for use in ACS. These are described and their relevance to SoftCo is discussed. Possible problems are also outlined, which may occur if the metric would be followed in a narrow focus.
These problems are usually not described in the literature found.
Requirements Coverage by Test Cases
Requirements Coverage by Test Cases [9] solves problem with ever increasing banks of requirements and test cases. Bidirectional links need to be established between require-ments and test cases - for each requirement, there is at least one test case which checks that the requirement is implemented properly and thus the product works as expected.
21
2. RESEARCH& LITERATUREREVIEW
Also, each test case must be linked to at lest one requirement, so that it is clear, why it exists. If it does not test a requirement, it does not need to be in the test repository in the first place. If a test case exists because of a requirement but they are not linked, a prob-lem can occur when the requirement is amended or removed. In such a case, related test cases need to be amended to reflect new state of the requirements, or it needs to be deleted. If this does not happen, there is inconsistency in project specification.
This metric displays the amount (percentage) of requirements which have associ-ated test cases. It also provides the project manager with confidence on the product quality and acts as an important indicator of effectiveness of testing team.
Possible problem can occur when a requirement is not fully covered by linked test cases. E.g a requirement can be complicated and test cases cover only a part of cases which can possibly happen. Such a situation cannot be covered by this metric. Solution exists on a different layer - someone responsible for that particular requirement shall validate completeness of test cases and update status of that requirement, making it clear it is tested completely.
Test Coverage
Test Coverage [9] provides the project manager with progress and state of testing, dis-playing number of test cases executed. It can be displayed for the project as a whole (i.e. overall test coverage) or per component/project feature. This metric shows number of executed tests in relation to all test cases recorded in the system (resp. in a compo-nent/feature). Example is shown in Figure2.2.
Figure 2.2: Test Coverage metric (Overall)
Possible problem can occur when test cases are still being written and thus the final number of them is not known. In such case, this metric can provide high numbers only because the test bank is not complete. Solution is in combination with previous metric, requirements coverage by test cases, in combination with which it provides better overview of tests already executed.
22
2. RESEARCH& LITERATUREREVIEW
There may be more distinct states in which a test case can be (than just passed/failed), which would result into more detailed Test Status Breakdown. It may be interesting to show also this detailed breakdown, with following test cases states:
• Not Scheduled– test case had been written, but not yet scheduled for execution nor assigned to a tester.
• Scheduled– manager scheduled the test case for execution and assigned it to a tester.
• In Progress – a test is being executed at the moment. This state may be useful for long-running tests.
• Blocked– test execution had to be paused for some reason (e.g. part of the testing infrastructure is not working at the moment). Tester will continue on this test when the problems have been resolved.
• Passed
• Failed
Component-wise Defect Distribution
Component-wise Defect Distribution [9] tries to answer which components/features are most problematic, i.e. have most defects found. Example is shown in Figure2.3.
Figure 2.3: Component-wise Defect Distribution
Defect Status
In addition to Component-wise Defect Distribution, defects may be broken down by status. It provides additional information about which parts of the software need to be 23
2. RESEARCH& LITERATUREREVIEW
fixed by developers (when the defects are open) or which needs to be focused on by testers for re-testing.
Typically, a defect passes through several states:
• New – Defect has just been recorded. It needs to be validated, whether it con-tains all relevant information or whether it is not a duplicate. When not properly specified, the issuer shall provide any additional information required.
• Open– Defect has been recorded and validated and can be assigned to a devel-oper for resolution.
• In Progress
• Resolved – Developer resolved the problem. It needs to be tested again by a tester.
• Closed– A tester has confirmed, that the defect is no longer present.
Summary
This section described several metrics found in the literature, which seem potentially useful for being implemented in ACS projects.
24
PART III
STATE OF THE PRACTICE
Chapter 3
ACS Software Development Process
This chapter describes ACS Software Development Process (ASDP), a customized software development process used in Honeywell ACS. It is mandatory for individual teams, and thus any proposed metrics and indicators shall be based on its work products. This chapter also describes the tools recommended by SoftCo, namely Contour for require-ments and test management and Jira for issue tracking.