Rick Kuhn and Raghu N. Kacker, NIST
Yu Lei, University of Texas at Arlington
Dimitris Simos, SBA Research
Testing is the most commonly used approach for
software assurance, yet it remains as much judgment
and art as science. We suggest that structural coverage
measures must be supplemented with measures of
input space coverage, providing a means of verifying
that an adequate input model has been defined.
O
ne of the key objections to testing as a formof software verification is that it is never possible to show that the system under test (SUT) works for all possible inputs. It is also difficult to provide meaningful statements about the
adequacy of a test set for verifying that the SUT works correctly. Conventional structural coverage measures, typically statement or branch coverage, leave much to be desired. Even if all statements are executed and all branches are taken, there is no guarantee that the input space is adequately covered for fault detection. A latent error may show up later with the appearance of a very rare combination of conditions that was not included Digital Object Identifier 10.1109/MC.2019.2951980
Date of current version: 15 January 2020
Input Space
OUTLOOK
in testing. Methods of systematically partitioning the input space have been studied extensively, but most necessarily involve a good deal of sub-jective judgment and do not provide quantitative measures of complete-ness. Combinatorial methods offer ways to build on existing techniques for input space partitioning, to pro-vide more rigorous testing.
COVERAGE MEASURES
A complete input model is one part of the goal of achieving thorough testing. Depending on which system aspects are to be considered when defining completeness, a variety of approaches exist for determining when testing is considered sufficient. Typically, these include some notion of fully covering requirements and may also consider structural coverage of the code. In software engineering, structural cov-erage refers to measures of the degree to which programs have been exer-cised. Two of the most widely used measures are statement coverage, the proportion of program statements that have been executed and branch cover-age (also known as decision covercover-age), the proportion of branches that have been evaluated as both true and false. Many other measures (or test criteria) exist, including condition coverage and modified condition decision cov-erage (MCDC) and it can be shown that
these criteria form a hierarchy.4 For
example, decision coverage subsumes statement coverage. Structural cover-age measures are of value for gauging the thoroughness of a test set, although their utility is somewhat limited. Statement coverage is the weakest of these measures, but failure to achieve at least full statement coverage indi-cates that the code has not been tested well enough. Branch coverage provides
a stronger measurement, and more complex criteria related to branch cov-erage, in particular MCDC, are used for some life-critical applications such as aviation.
Of course, simply achieving some level of structural coverage is hardly adequate because it must be shown that the SUT accomplishes its intended function. If the code does only one of 10 required functions, the level of struc-tural coverage is obviously an inade-quate measurement for determining whether all 10 functions have been implemented. The SUT may work well for the subset of inputs related to the implemented function, but it would fail if presented with inputs that direct the execution of unimplemented func-tions. Similarly, if all of the functions are implemented but are not all cor-rect, the code may produce the right result for some input values but fail with others.
Among the key questions in soft-ware testing and assurance are which parameters matter and what values should be included in testing. Except for trivial problems, it is generally intractable to test all possible inputs, so some form of equivalence parti-tioning must be used, that is, the input space is divided into sets of inputs that are considered equivalent with respect to some relation meaningful for the software under test. In other words, the problem is to find param-eter values that each adequately rep-resent a much larger set of values. A simple example is partitioning into valid and invalid input values. Another example might be ranges of weights or sizes that are equivalent with respect to the shipping charge for each class.
Because the input space is far too large to test exhaustively, inputs must
be discretized for continuous-valued variables or a small subset selected from enumerated values. This problem is fundamental in software assurance, and an extensive body of research has been developed to solve it. In general, input parameters cannot be consid-ered in isolation because the value of one may limit the values that must be considered when testing for other parameters, that is, there may be con-straints among parameters.
How can an adequate input model be found? Part of the problem was solved decades ago through the use of systematic methods of analyzing and partitioning the input space. The accepted practices for this task include
boundary value analysis,4,5 which
identifies boundaries in range-de-fined variables and divides the input space into partitions of values for which the SUT can be expected to produce equivalent results based on the specification. An alternative is to use the source code to identify inputs that will provide required path
cover-age through symbolic execution19,20
or concolic testing21,22 approaches.
These methods traverse program paths to find input parameter values that will cover each path and may do error checking, for example, assertion violations, exceptions, overflows, and some types of security vulnerabilities in the process.
Among the methods for deriving test input values from specifications,
the classic category partition method1
provides a systematic way to inte-grate the definition of boundary val-ues and equivalence partitions into test frames, consolidating values into sets that exercise a particular func-tionality. A more recently developed approach that builds on these ideas is
input model.
After an input model has been con-structed, how can one ensure that it is adequate for testing, that is, do the equivalence partitions contain values that are truly equivalent in terms of sys-tem response? More generally, can we find measures of the IPM that are rele-vant to testing? To answer this question, it is useful to first consider empirical data on the distribution of faults.
BACKGROUND OF FAULT
DISTRIBUTION
Empirical data show that most failures are triggered by a single parameter value or interactions between a small number of parameters (generally two
to six7), a relationship known as the
interaction rule. An example of a sin-gle-value fault might be a buffer over-flow that occurs when the length of an input string exceeds a particular limit. Only a single condition must be true to trigger the fault input length > buffer size. A two-way fault is more complex because two particular input values are required to trigger the fault. One exam-ple of this is a search/replace function that only fails if both the search and replacement strings are single charac-ters. If one of the strings is longer than one character, the code does not fail; thus, we refer to this as a two-way fault. More generally, a t-way fault involves t such conditions.
with interaction strength, up to t = 4. With the medical device applications, for example, 66% of the failures were triggered by only a single parameter value, 97% by single values or two-way combinations, and 99% by single values or two- or three-way combina-tions. The detection-rate curves for the other applications studied were similar; they reached 100% detection with four to six-way interactions.
correspond to actual defects, the faults and defect will not be detected. Con-versely, with the right input model, we have evidence that the software works correctly for t-way combinations con-tained in passing tests.
COVERAGE IMPLICATIONS
OF FAULT DISTRIBUTION
The empirical distribution of faults suggests that it is beneficial to considerFIGURE 1. A cumulative fault distribution. DBMS: database management system; FDA: U.S. Food and Drug Administration; NW Sec: network security.
FDA MySQL NW Sec NeoKylin Browser MySQL2 DBMS DSCS Server Apache2 0 10 20 30 40 50 60 70 80 90
100 Cumulative Proportion of Faults for t = 1 … 6
OUTLOOK
the degree to which combinations of input values are covered in a test set. The data in Figure 1 suggest that we may miss 5–10% of possible errors if we do not test four-way, or more com-plex, combinations. However, these findings also mean that, in general, it will not be necessary to test all possible combinations of input values because relatively few factors are involved in fail-ures. Covering t-way combinations of inputs for small values of t is, to some extent, equivalent, or at least close to, exhaustive testing. Thus, it is import-ant to understand the coverage of input combinations for any test set.
As noted, a covering array may be constructed to cover all t-way com-binations of input parameters, but any test set (of course) contains many combinations. We can measure the combinatorial coverage, that is, the coverage of t-way combinations in a test set, for a better understanding of test set quality. These measurements provide quantitative levels of quality very different from those of conven-tional structural coverage. In partic-ular, combinatorial coverage has a direct relationship with fault detec-tion. As displayed in Figure 1, a sig-nificant portion of fault-triggering combinations involves more than two factors, so the level of combination
coverage measures the ability of the test set to detect, for example, faults induced by four- or five-way combi-nations. These measurements can also be computed independently of structural coverage prior to running any tests because they relate to the (static) content of the test set. There is, however, an interesting relation-ship between combinatorial coverage and structural coverage, as discussed later in this article.
MEASURING THE COVERAGE
OF FAULT-TRIGGERING
COMBINATIONS
Measuring combination coverage can help with understanding the degree of risk that remains after testing. If a high level of coverage of input state-space variable combinations has been achieved, then the risk is small that there is a latent combination that may induce a failure. Lower coverage reflects a greater risk that there is some untested failure-triggering combination. A vari-ety of combinatorial coverage measure-ments are helpful with estimating this risk. In this section, we describe some
of the basics (see Kuhn et al.15 and
Leithner et al.18 for more details).
For a set of t variables, a variable- value configuration is a set of t valid val-ues, one for each of the variables; in other words, the variable-value configu-ration is a particular setting of the vari-ables. For example, consider four binary variables, a, b, c, and d. For the selection of three variables, a, c, and d, the set {a = 0, c = 1, d = 0} is a variable-value con-figuration; the set {a = 1, c = 1, d = 0} is a different configuration.
It is also useful to measure the num-ber of t-way combinations covered out of all possible settings of t variables. For a given combination of t variables, total variable-value configuration coverage is
the proportion of all t-way variable-value configurations covered by at least one test case in a test set. This measurement may also be referred to as total t-way coverage. For the array listed in Table 1, there are four-choose-two, C(4, 2) = 6 possible variable combinations, and
22 × C(4, 2) = 24 possible variable-value
configurations. Of these, 19 variable- value configurations are covered, and the only ones missing are ab = 11, ac = 11, ad = 10, bc = 01, and bc = 10, so the total variable-value configuration coverage is 19/24 = 79%. Although the example in Table 1 uses variables with the same number of values, it is also possible to compute coverage for test sets in which parameters have differing numbers of values.
Figure 2 is a graph of the two- (red/ solid) and three-way (blue/dashed) coverage data for the tests in Table 1. The combination coverage of variable values is given as the y-axis, with the percentage of combinations reaching a particular coverage level given as the x-axis. Of particular interest is the
minimum t-way coverage, Mt, which
is the smallest proportion of the cov-erage of variable-value configurations for parameters taken t at a time. For example, Table 1 lists four binary vari-ables (a, b, c, and d) where each row rep-resents a test; parameters taken two at a time have four possible configu-rations: 00, 01, 10, and 11 (see Table 2). Of the six possible two-way variable combinations, ab, ac, ad, bc, bd, cd, only bd and cd are covered at different lev-els, as listed in the second column. The minimum coverage for two-way
com-binations M2 = 0.5 is shown as y = 0.5
at x = 0.833 because one out of the six variable combinations has two of out the four possible settings of two binary variables covered. The area under the curve for two-way combinations is
TABLE 1.
A test array with
four binary components.
a b c d
0 0 0 0
0 1 1 0
1 0 0 1
approximately 79% of the total area of the graph, which reflects the total coverage of 19 out of 24 two-way combinations.
PRACTICAL EXAMPLE
Combinatorial coverage measurements were originally developed to ana-lyze the input space coverage of tests
for spacecraft control software.9,10 A
test suite of 7,489 tests was developed using conventional techniques, such as use case analysis and knowledge of likely error sources. The control sys-tem included 82 variables with a test
configuration of 132754262 (three
one-value, 75 binary, two four-one-value, and two six-value variables). Figure 3 shows the combinatorial coverage achieved by the full set of 7,489 tests. The area under the curves is the proportion of the input space tested at the various t-way levels, while the area above the curve represents untested space, that is, where errors may still be found. For example, two-way coverage is 94%, so
there is relatively little risk of faults that may be triggered by two-way com-binations (see Table 3).
THE RELATIONSHIP
BETWEEN COMBINATORIAL
AND STRUCTURAL
COVERAGE
It is obvious that any thorough assess-ment of a system must show that all requirements have been met; but it is
also true that the system should not produce unexpected behavior. One of the most effective means of confirm-ing these objectives is to generate tests from requirements and ensure that full structural coverage has been achieved for some strong coverage criterion. Tests must be requirement based; structural coverage is used only to val-idate the quality of these tests. This approach has been required by the U.S.
FIGURE 2. The graph of coverage for the tests in Table 1. 0
0
0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5 0.55 0.6 0.65 0.7 0.75 0.8 0.85 0.9 0.95
1
Combinations
OUTLOOK
Federal Aviation Administration (FAA) for testing life-critical aviation soft-ware in Radio Technical Commission
for Aeronautics Standard DO-178B13
and was continued in the updated
stan-dard DO-178C.14 This requirement uses
a stronger relative of branch/decision coverage called MCDC which subsumes branch coverage. (Branch coverage necessitates that each branch of every control structure, such as “if” or “while” conditionals, has been taken in the test suite. MCDC requires that every condi-tion in a decision in the program has taken on all possible outcomes at least once and that each condition has been shown to independently affect the deci-sion outcome, and that each entry and exit point have been invoked at least once.) The intuition is clear: if full struc-tural coverage has not been achieved, then either tests are inadequate or there is some system functionality that is out-side of the specification.
Structural coverage is a suitable heu-ristic for evaluating whether we have
a correct input model. Chilenski11
notes that “MCDC should be used as a structural coverage criterion because it attempts to provide a cost-effective form of logic verification. In essence, MCDC can be thought of as a (weak) measurement of the coverage of equiv-alence classes and boundary values.”
According to DO-178B:12
If branches (decision outcomes) are left uncovered by the require-ments-based verification process, then that process failed to consider the special cases the development process thought were important. Decision coverage ensures that the verification process has considered sufficient operational scenarios to execute every special case the system was designed for in at least one operational context.
One interesting development from combinatorial testing research is the result that demonstrates a connection between (static) combinatorial cover-age of the input space and (dynamic)
structural coverage.16 This
connec-tion provides a formal justificaconnec-tion for the DO-178B requirement that MCDC (and, consequently, branch coverage) should be achieved as a way to verify the adequacy of equivalence classes and boundary values.
As noted previously, structural cov-erage is a dynamic measurement in the sense that it requires code execu-tion. Combinatorial coverage, as defined in this section, looks at the input space, which is a static measurement of value combinations. Clearly, the input to a program must affect the paths exe-cuted in the code and, thus, impact structural coverage measures. In fact,
we can show that there is a relation-ship between the coverage measure-ments introduced in this section and structural coverage, which is captured in simple theorem that defines what
we call the branch coverage condition:16
Branch coverage condition: A test set provides 100% branch coverage for t-way condition-als if Mt + Bt > 1 where Mt = the minimum combinatorial coverage at level t, and Bt = the mini-mum proportion of t-way com-binations that is guaranteed to trigger a branch within the code, where all the variables in deci-sion predicates have values from the variable set with minimum coverage characteristic Mt.16 The reason this condition is import-ant is that when test sets are based on covering arrays, which, by definition,
have Mt = 100% for a t-way covering array,
then branch coverage should be 100%.
The term Bt refers to the minimum
pro-portion of input variable-value combina-tions that reach a conditional expression and trigger a branch. For example, in the statements if (A&&B) line1; if (C) line2;, where A and B are Boolean
vari-ables, Bt = 0.25, because line1 is executed
for only one of the possible four settings of A&&B.
A corollary also shows that the branch coverage condition is obtained
with Mt > 1 − (k/vt), if k or more t-way
set-tings satisfy every predicate where all the variables in decision predicates have values v from the variable set with
min-imum coverage characteristic Mt. If one
uses a covering array in testing, how can
Mt + Bt > 1 ever be false? In fact, it
can-not if the input space has been properly modeled, that is, if the variable-value partitions have been properly defined.
TABLE 3.
The total t-way coverage for the configuration in Figure 3.
Interaction Combinations Settings Coverage
Two way 3,321 14,761 .940
Three way 88,560 828,135 .831
Four way 1,749,060 34,364,130 .688
model that has been produced. That is,
Mt is for an ideal IPM that has not
nec-essarily been produced, while Mt′ is
mea-sured on the current IPM, which may need to be revised.
To clarify, if Mt′ is from a
cover-ing array for an adequate IPM, then
Mt′ + Bt > 1 => branch coverage = 1, so,
the branch coverage is < 1 => Mt′ + Bt ≤ 1,
but Bt ≥ v−t for all parameters and M
t′ = 1
because it is from a t-way covering array.
Therefore, Mt′ is not from an adequate IPM
and must be revised. This result gives us a formal basis for the DO-178B approach of using MCDC to validate a require-ments-derived test set.
Accordingly, failure to achieve the branch coverage condition indicates that the input model must be revised. The corollary described in this section suggests that it is possible to detect a deficient input model even if 100% branch coverage is not achieved, a result that can be useful in developing the IPM. Note, however, that we do not want to imply that less than full structural cov-erage is adequate for testing; rather, it is only for developing an adequate IPM. During the full testing process, incom-plete structural coverage indicates a dif-ferent aspect of deficient testing.
A study by Bartholomew17
illus-trates this relationship between input models and coverage very well. The initial input model definition pro-duced branch coverage in the range of approximately 70–75%, but by rede-signing the IPM, 100% branch and MCDC coverage (subsumes branch erage) was obtained. The branch cov-erage condition provides a theoretical justification for the heuristic of requir-ing structural coverage as a means of validating requirements-based tests.
Note that achieving 100% branch coverage with a particular level of t does not indicate that higher strength test-ing is not needed. For example, if we find full branch coverage with two-way testing, it will still be important to cover higher-interaction strengths because some failures may involve multiple fac-tors that depend on nested conditions or variable states. That is, full branch coverage can be considered as a nec-essary, but not sufficient, condition for achieving an adequate IPM. Unavoid-ably, some engineering judgment will be needed for selecting representative values for the model, but measuring branch or MCDC coverage provides an assured method of validating IPM ade-quacy. Conventional methods of input space partitioning, such as the
cate-gory partition1 and classification tree
methods,2 are highly effective for this.
The combinatorial coverage measures described in this section can be used to improve test engineering disciplines and provide measures of risk related to the untested input space.
D
espite its shortcomings,test-ing is still the most widely used method of software assurance, but better measurements are needed to ensure the quality of the test design. The most common measurements of test completeness are various forms of structural coverage; however, more can be done to improve the test engi-neering process. Structural coverage is a dynamic measurement that should be supplemented with static measures of the input space. Combinatorial cov-erage measures the proportion of the input space relevant to the testing covered by tests. Because only a small
DISCLAIMER
Certain products may be identified in this document, but such identifi-cation does not imply recommenda-tion by NIST, nor does it imply that the products identified are neces-sarily the best available for the purpose.
evaluation of the uncertainty in outputs of physical and computational mea-surements. Kacker received a Ph.D. in statistics from Iowa State University. He is a fellow of the American Statistical Association and American Society for Quality. Contact him at [email protected].
YU LEIis a professor in the Department of Computer Science and Engineer-ing at the University of Texas at ArlEngineer-ington. His research interests include soft-ware analysis, testing, and verification. Lei received a Ph.D. in computer sci-ence from North Carolina State University, Raleigh. Contact him at ylei@uta .edu.
OUTLOOK
number of variables is involved in failures, measuring the proportion of t-way combinations covered in testing provides information on residual risk when the software is released.
This approach of using both static and dynamic measures also provides a means of ensuring that the input space partitioning has been performed cor-rectly. A simple theorem defining the branch coverage condition shows that if full branch coverage is not achieved when input space coverage is suffi-ciently high, then the input model has not been defined correctly and must be revised. Tools exist to compute the necessary measures and can be a valu-able addition to the test engineering
process.15,18
ACKNOWLEDGMENTS
This work was supported by the U.S. Department of Commerce and the National Institute of Standards and Technology under financial assistance award 70NANB18H207.
REFERENCES
1. T. J. Ostrand and M. J. Balcer, “The category-partition method for specifying and generating fuc-tional tests,” Commun. ACM, vol. 31, no. 6, pp. 676–686, 1988. doi: 10.1145/62959.62964.
2. M. Grochtmann and K. Grimm, “Classification trees for partition testing,” J. Softw., Testing Verif. Rel., vol. 3, no. 2, pp. 63–82, 1993. doi: 10.1002/stvr.4370030203 3. D. J. Richardson and L. A. Clarke,
“Partition analysis: A method com-bining testing and verification,” IEEE
Trans. Softw. Eng., vol. SE-11, no. 12,
pp. 1477–1490, 1985.
4. P. Ammann and J. Offutt,
Introduc-tion to Software Testing. Cambridge,
U.K.: Cambridge Univ. Press., 2016.
5. B. Beizer, Software Testing Techniques. New Delhi, India: Dreamtech Press, 2003.
6. E. J. Weyuker and T. J. Ostrand, “Theories of program testing and the application of revealing sub-domains,” IEEE Trans. Softw. Eng., vol. SE-6, no. 3, pp. 236–246, 1980. 7. D. R. Kuhn, D. R. Wallace, and
A. J. Gallo, Jr., “Software fault interac-tions and implicainterac-tions for software testing,” IEEE Trans. Software Eng., vol. 30, no. 6, pp. 418–421, June 2004. 8. D. R. Kuhn, R. N. Kacker, and Y. Lei,
“Practical combinatorial testing,” NIST, Gaithersburg, MD, NIST Spe-cial Publication 800-142, Oct. 2010. 9. J. R. Maximoff, D. R. Kuhn, M. D.
Trela, and R. Kacker, “A method for analyzing system state-space cover-age within a t-wise testing frame-work,” in Proc. IEEE Int. Systems Conf., Apr. 2010, pp. 598–603.
10. D. R. Kuhn, R. N. Kacker, and Y. Lei, “Combinatorial coverage as an aspect of test quality,” CrossTalk, vol. 28, no. 2, pp. 19–23, 2015. 11. J. J. Chilenski, “An investigation of
three forms of the modified con-dition decision coverage (MCDC) criterion,” U.S. Department of Trans-portation, Federal Aviation Admin-istration, Rep. DOT/FAA/AR-01/18, Apr. 2001.
12. L. A. Johnson, “DO-178B, Software considerations in airborne systems and equipment certification,”
Cross-Talk, vol. 199, Oct. 1998. [Online].
Available: http://www.dcs.gla .ac.uk/~johnson/teaching/safety/ reports/schad.html
13. Software Considerations in Airborne
Systems and Equipment Certification,
RTCA/DO-178B, Dec. 1, 1992. 14. T. K. Ferrell and U. D. Ferrell, “RTCA
DO-178C/EUROCAE ED-12C,” in Digital
Avionics Handbook, C. R. Spitzer, Ed.
New York: Taylor & Francis, 2017, pp. 16.1–16.11.
15. D. R. Kuhn, I. Dominguez, R. N. Kacker, and Y. Lei, “Combinatorial coverage measurement concepts and applications,” in Proc. 2nd IEEE Intl
Workshop on Combinatorial Testing (IWCT2013), Luxembourg, Mar. 2013,
pp. 352–361.
16. D. R. Kuhn, R. N. Kacker, and Y. Lei, “Measuring and specifying combi-natorial coverage of test input con-figurations,” Innov. Syst. Softw. Eng., vol. 12, no. 4, pp. 249–261, 2016. 17. R. Bartholomew, “An industry
proof-of-concept demonstration of automated combinatorial test,” in Proc. 8th Int. Workshop on
Auto-mation of Software Test, May 2013,
pp. 118–124.
18. M. Leithner, K. Kleine, and D. E. Simos, “CAmetrics: A tool for advanced combinatorial analysis and measurement of test sets,” in
Proc. IEEE Int. Conf. Software Testing, Verification and Validation Workshops (ICSTW), Apr. 2018,
pp. 318–327.
19. L. A. Clarke, “A program testing system,” in Proc. ACM Annu. Conf., Oct. 1976, pp. 488–491.
20. J. C. King, “Symbolic execution and program testing,” Commun. ACM, vol. 19, no. 7, pp. 385–394, 1976. doi: 10.1145/360248.360252
21. P. Godefroid, N. Klarlund, and K. Sen, “DART: Directed auto-mated random testing,” ACM
SIGPLAN Notices, vol. 40, no.
6, pp. 213–223, June 2005. doi: 10.1145/1064978.1065036 22. K. Sen, D. Marinov, and G. Agha,
“CUTE: A concolic unit test-ing engine for C,” ACM SIGSOFT
Softw. Eng. Notes, vol. 30, no.
