in general. To the best of our knowledge, we could not find work generating samples of models to overcome the lack of availability of real models. The generation of models is widely used for different purpose, but the meta-heuristic search within the space of models and the concern in having realistic samples of models is a unique feature of SBMTE.
2.3
Specification-Based Regression Testing
Regression testing is performed after modifying a software system. The goal is capturing regression defects, which, in turn, are defects inserted due to a modification [Binder 1999], i.e. the test cases are executed to verify if the modifications caused the software to stop functioning as expected.
In short, let P be a baseline version of the program, and P0be the next version (i.e. delta
version) of P . In turn, S and S0are, respectively, the baseline and delta specifications for P
and P0. The test suite used to test P is referred to as T , and T0 is the test suite used to test
P0. Throughout this work, T and T0will be referred as baseline test suite and delta test suite,
respectively. P (t)/P0(t0) stands for the execution of P/P0 with test case t ∈ T /t0 ∈ T0.
There are two main types of regression testing: Corrective and progressive regression testing. Corrective regression testing is applied when specification is not changed, for exam- ple when code refactoring is performed. Since the specification remains the same, test cases can be reused. On the other hand, in progressive regression testing the specification changes and new test cases must be designed (at least for new parts of the specification) [Tamimi and Zahoor 2011].
After each regression testing session is finished, the test cases used to test the delta ver- sion become a part of the regression test suite, and the cycle repeats each time a set of modifications are performed. Since modifications are performed frequently, the test suite size and the costs related to testing often increase significantly. Test case selection can be used to alleviate the costs incurred in regression testing. The goal is to select a (minimal)
subset of test cases Ts that tests P0. Whenever the costs to select and execute the subset are
smaller than the costs to execute all of the test cases (i.e. the retest all approach), the cost of regression testing has been reduced [Rothermel and Harrold 1996].
2.3 Specification-Based Regression Testing 19
defects in P0. However, identifying this subset is impractical because there is not enough
information to select only those test cases. Instead, a weaker criterion is considered and the goal becomes selecting all modification-traversing test cases, i.e. test cases exercising
new or modified parts of P0/S0, or test cases that formerly executed removed parts from
P/S [Rothermel and Harrold 1996]. The subset of modification-traversing test cases is the closest approximation to the fault-revealing test cases that can be achieved without executing all test cases [Yoo and Harman 2012].
Our selection strategy (SART) focuses on progressive regression testing, where modifi- cations are performed on a specification model and the goal is to select all test cases that
exercise the parts of the system that has been modified in S0 (compared to S). Two types
of modifications are considered here, the addition and removal of model elements in the specification models.
As an example, consider that a client requests removal of use cases or scenarios from a system. This removal is reflected by a removal of transitions from a specification model. In turn, addition of new transitions represent, for example, new functionalities or scenarios. More complex modifications can be expressed as a combination of these two [Korel et al. 2002; Chen et al. 2007]. According to Leung and White [Leung and White 1989], the test cases for regression testing can be classified as:
• Obsolete: This class contains test cases that cannot be executed anymore due to an invalid input/output relationship, or for traversing a removed part of S or P .
• Reusable: This class comprises test cases exercising unmodified parts of the specifi- cation and their corresponding unmodified program construct. Since no modification is exercised, the same result is expected, meaning that they do not need to be executed during progressive regression testing.
• Retestable: This class includes test cases that exercise unmodified parts of the speci- fication and may present a different result. An example of retestable test cases are sce-
narios exercising unchanged parts of S0 but with new program constructs (e.g. bound-
ary values).
2.3 Specification-Based Regression Testing 20
• New-specification: These test cases exercise the modified parts of the specification by
executing new code in P0.
Distinguishing the classification of retestable and reusable test cases at system’s specifi- cation level can be challenging. For models with lower levels of abstraction (e.g. control flow graphs built from a source code) the program construct can be easily accessed. However, that is very difficult to achieve with abstract test cases. Briand et al. adapted Leung and White’s classification to consider UML designs in order to handle a higher level of abstraction. Ac- cording to their definition, a retestable test case remains valid in terms of the sequence of messages to boundary objects but one or more of these messages may have changed (e.g. op- eration postcondition, signal class), whereas reusable test cases remains unchanged both in the sequence and the internal messages [Briand et al. 2009]. We consider a similar definition where retestable test cases are sequences of transitions that remain the same but at least one of the labels of those transitions has changed (i.e. no addition or removal of transitions hap- pened, just changes in the label). In turn, unchanged sequences and labels will be considered as reusable test cases.
Similarly, classification and selection of obsolete test cases is challenging and yet very important. If executed, an obsolete test cases will fail not due to a regression defect, but due to an attempt to execute removed parts of the software system. Thus, maintenance to identify and remove these test cases from the test suite is required. Nonetheless, removals can also cause regression defects. For example, an inappropriate removal may cause the SUT to reach a state that should not be reached according to the new specification. But how can we test a removed part of the SUT? One solution is to exercise transitions (of an obsolete test case)
that were not removed [Korel et al. 2002] based on the assumption that a region2 (named
firewall) around the removal can be defined where regression defects can be triggered. Selected test cases can belong to any of the mentioned classes according to the selec- tion goal. Specification-based regression test selection benefits from model-based tech- niques since most types of models support a high level representation of the system. Thus, specification-based strategies provide a more precise traceability between test cases and the
2For specification models, we consider regions to be a set of transitions and states within a small distance
from the state where modifications were performed. For example direct predecessors and successors of the