Foundations of Software Testing
Chapter 1: PreliminariesThese slides are copyrighted. They are for use with the Foundations of Software Testing
book by Aditya Mathur. Please use the slides but do not remove the copyright notice.
Aditya P. Mathur
Test Metrics
The term metric refers to a standard of measurement.
The test Metrics are divided into –
1) Organizational 2) Project
3) Process 4) Product
Test Metrics
A test metric measures some aspect of the test process. Test metrics
Test Metrics(Cont)
The four core areas that assist in the design of metrics are
schedule,quality,resources and size.
Schedule related metric measure actual completion times of various
activities and compare these with estimated time for completion.
Quality related metrics measure quality of product or process.
Resource related metrics measure items such as cost in
dollars,manpower and tests executed.
Size related metrics measure size of various objectssuch as the source
code and no.of tests in test suite.
Organizational metrics
Useful in overall project planning and management. Some of these metrics are obtained by aggregating
compatible metrics across multiple projects.
Thus, for example, the number of defects reported after
product release, averaged over a set of products developed and marketed by an organization, is a useful metric of
Organizational metrics
Computed at regular intervals to show the quality of
organization.
Allows the senior management to monitor the overall
strength of the organization and points the area of weakness.
Helps senior management in setting new goals and plan for
resources needed to realize these goals.
Organizational metrics
For example, one might say “The number of defects
reported in the field over all products and within three
months of their shipping, has dropped from 0.2 defects per KLOC (thousand lines of code) to 0.04 defects KLOC.” Other organizational level metrics include testing cost per KLOC, delivery schedule slippage, and time to complete system testing.
Defect density is the number of defects per line of code.
Organizational metrics
Example 1.17 The average defect density across all software
projects in a company is 1.73 defects per KLOC. Senior
management has found that for the next generation of software products, which they plan to bid, they need to show that product density can be reduced to 0.1 defects per KLOC. The management thus sets a new goal. Given the time frame from now until the time to bid, the management needs to do a feasibility analysis to
determine whether or not this goal can be met.
If a preliminary analysis shows that it can be met, then a detailed plan needs to be worked out and put into action. For example,
management might decide to train its employees in the use of new tools and techniques for defect prevention and detection using
Project Metrics
Project metrics relate to a specific project.
Useful in monitoring and control of specific project. The ratio of actual-to-planned system test effort in one
project.
Measured in terms of tester-man-months.
Tracking the ratio of estimated effort to actual for any
phase assists the project manager in allocating testing resources.
Ratio of no. of successful tests to total no. of tests is
Process Metric
Every project uses some test process. ex – single developer
project may use big bang approach.
The goal of the process metric is to assess the goodness of
the process.
The later the defect found, costlier it to fix.
Process Metric
In one software development project, it was found that
15% of the total defects were reported by customers, 55% of the defects prior to shipping were found during system test, 22% during integration test, and the remaining during unit test.
Defects are generated in each life cycle
production activity
Injected defects are removed in testing
activities after code is completed
Not all defects are removed at SHIP
Product Metrics(General)
Product metric relate to a specific product such as a
compiler for a programming language.
Useful in making decisions related to the product. “Should
this product be released for use by the customer ?”
Product metrics– Cyclomatic complexity.
Product Metrics(General) contd..
Given the CFG G of the program P containing N nodes, E
edges, and p connected procedures, the cyclomatic complexity V(G) is –
V(G) = E – N +2p.
Larger the value of V(G) the higher the program
complexity.
Product Metrics : OO Software
Metric Meaning
Reliability Probability of failure of software product w.r.to given operational profile in a given environment.
Defect density Number of defects per KLOC.
Defect severity Distribution of defects by their level of severity.
Test coverage Fractions of testable items.
Cyclomatic complexity Measures complexity of program based on its CFG.
Weighted methods per class Complexity of method in class.
Product Metrics : OO Software
Response set Set of all methods that can be invoked directly or indirectly when a message is sent to object O
Progress Monitoring and trends
Metrics are used for monitoring purpose. So
measurements need to be made at regular intervals
of time.
The saturation indicates that the product is
reaching stability.
Static and dynamic metrics
Static metrics are computed without executing the product.
ex: no. of testers working in a project.
Dynamic metrics require code execution.
Testability
Testabiltity is the degree to which a system or component
facilitates the establishment of test criteria and the
performance of tests to determine whether those criteria have been met.
There are static and dynamic testability metrics.
Software complexity is static testability metrics. The more
complex an application lower the testability.
Dynamic metrics for testability include various code based
coverage criteria.
High testability is the desired goal.
Testability
The features to be tested and how are they tested need to
