Execution-Based Software Testing

Figure 2.2 shows the process of execution-based software testing. In this figure, the rectangles denote the testing activities and the parallelograms denote the outcome of these activities. The testing process starts with a system under test, P , and a test adequacy criterion, C, as input. The testing process is iterative and stops iterating

Figure 2.2: A software testing process model.

when the test cases in test suite T satisfies the adequacy criterion C and assures some level of confidence in the quality of P [220]. However, the testing process can also stop in case of deadline misses or budget overrun. The testing process can also halt if the tester gets an intuition of achieving some acceptable level of quality. Even if the testing process stops or meets with C, it does not guarantee that all the defects have been revealed by the test cases. Therefore, testers set many adequacy criteria (different coverage criteria, software metrics, etc.) to build the confidence on quality. The test adequacy criteria depend on the chosen program representation and definition of some quality parameters that T should satisfy. In Section 2.1.4, we discuss the test adequacy criteria considered in this thesis and some test adequacy metrics that exist in the literature and practiced frequently. The test specification stage evaluates P in the context of chosen C in order to construct an adequate T . Once the test case descriptions for P have been generated, the test case generation phase begins. A lot of different techniques and tools have been

proposed to manually or automatically generate the test cases. The generation of test cases is beyond the purview of this thesis because of the availability of JUnit test cases available in Software-artifact Infrastructure Repository (SIR) [59] for the experimental programs. After the generation of the test cases, the execution of the test cases starts. Once again, the execution of the tests within T can be performed in a manual or automated fashion. The results from the execution of the test cases are analyzed using JaButi framework [196] to determine the quality of individual test cases in terms of coverage. Thus, iterative testing of P continues throughout its initial development. However, it is also important to continue testing after

P undergoes changes in maintenance phase. Regression testing is an important

software maintenance activity carried out to ensure that the changes made does not adversely affect the correctness of P . All of the previously mentioned stages iteratively continue for the regression testing process based on the existing test cases (new test cases may be added) and the adequacy measurements defined for these tests [2, 41, 107, 127, 173].

2.1.3 Regression Testing

Regression testing is considered as a part of the validation activity and seems to pose a big problem in testing the software. It becomes a big challenge to manage the retesting process with respect to the time and cost, especially when the test suite becomes too large. A software system undergoes changes in the form of bug fixes or addition/deletion of functionality. During the process of maintenance the software needs to be regression tested to validate that these changes introduced no defects. Figure 2.2 shows that the regression testing process has to go through all the testing stages for every change made to the program. Thus, regression testing ensures that the evolution of an application does not inadvertently lowers the software quality. Indeed, the importance of regression testing is well understood. However, many software development teams may not afford thorough regression testing because they often expense one-half the cost of software maintenance [127]. The execution of the test suite often makes regression testing an expensive activity. According to Rothermel et al. [176], complete regression testing of a software of 20,000 lines of code require around seven weeks of continuous execution. This necessitates development of many techniques to enhance the efficiency of regression testing (selection, minimization, and prioritization). Thus, the problem of regression testing is formally defined as follows:

Definition 2.1 (Regression testing). Given a program P , its modified version P0, and a test suite T that is used to test P, regression testing finds a way to exercise T to restore confidence on the correctness of P0.

Selective Regression Testing

Many researchers have devised methods as an attempt to reduce the cost and time of regression testing. Regression test selection approaches aim to reduce the cost of regression testing by selecting some relevant subset of the existing test suite. An obvious approach to select the test cases is to use the source code of a program to determine the tests that are appropriate with respect to the changes [172]. There- fore, selective retest techniques [27] attempt to identify those test cases that can exercise the modified parts of the program and the parts that are affected by the modification to reduce the cost of testing. The features of selective retest technique are as follows:

a. The resources required to retest a modified version of the program are mini- mized.

b. This is achieved by minimizing the number of test cases to be exercised. c. The test suite grows uncontrollably due to the continuous modifications done

to the programs for which selective retesting is required.

d. The relationship between the test cases and the program parts that are covered by the test cases can be analyzed better.

Thus, regression test case selection is formally defined as follows:

Definition 2.2. Given the program P , its modified version P0, and a test suite T , find a subset T0 of the test suite T = {t₁, t2, . . . , tn} comprising of n test cases, i.e.

T0 ⊂ T such that ∀ ti ∈ T, t ∈ T0 ⇔ P0(t) 6= P (t), 1 ≤ i ≤ n, i.e. ti executes

all the code affected with respect to the modifications.

Test Suite Minimization

Test suite minimization techniques aim to identify a reduced test suite that can still assure software quality. The size of the reduced test suite should therefore be much smaller than the original test suite. The minimization problem described in Definition 2.3 follows the definition given in [210]. This definition is considered as the minimal hitting set problem. This is so because it is assumed that a single test

case satisfies each test requirement ri. However, in reality, it could be different.

For example, a functional test requirement may require more than one test case to satisfy. Therefore, the functional granularity of the test cases need to be defined in order to apply the given formulation of the problem. Owing to the fact that the test suite minimization problem is NP-complete by nature, many researchers have encouraged the application of different heuristics [28, 105, 131, 164, 174, 208, 210] while formulating the minimization problem. The test suite minimization as formally defined by Harman et al. [210] is given below:

Definition 2.3. Given a test suite T , a set of test requirements {r₁, r2, . . . , rn} that

must be satisfied to provide the desired adequate testing of the program, and subsets

{T₁, T2, . . . , Tn} ⊂ T , each of them associated with each of the ri such that any one

of the test cases t_j ∈ T_i can be used to achieve requirement r_i. Find a representative set, T0 ⊂ T that satisfies all ri.

When every test requirement in {r₁, r2, . . . , rn} is satisfied by T0then the testing

criterion is said to be satisfied.

Test Case Prioritization

Regression test prioritization techniques [66, 69, 104, 108, 175, 184] attempt to find an order for executing the test cases so that the likelihood of detecting the defects is early and maximum in the regression testing process [66, 177]. There are two types of prioritization [41]:

i. General test case prioritization: For a given program P and a test suite T , the test cases are prioritized such that the prioritization is useful to a succession of program modifications done to P , without the knowledge of the modification.

ii. Version specific test case prioritization: In this approach, the test cases are prioritized whenever program P is modified to P0, with the knowledge of the changes made in P .

All the existing regression test case prioritization approaches [66, 69, 104, 108, 162, 163, 175, 184] target to find an optimal ordering of the test cases based on the rate of fault detection or rate of satisfiability of coverage criterion under consideration. More formally the prioritization problem as defined by Rothermel et al. [175] is as follows:

Definition 2.4. Given a test suite T , the set of permutations of T denoted as P T ,

a function f , from P T to the real numbers. Find T0∈ P T such that ∀T00 T00 ∈ P T

∩ T00 6= T0


f T0

≥ T00


,

where, P T is the set of all possible orderings of the test cases in T and f is a function that maps the ordering with an award value.

Rothermel et al. [175] proposed a metric to ensure the efficiency of any of the existing prioritizing techniques. This metric is called Average Percentage of Fault

Detected (APFD) and is used by many researchers to evaluate the effectiveness

of their proposed techniques. APFD measure is calculated by taking the weighted average of the number of faults detected during execution of a program with respect to the percentage of test cases executed.

Let T be a test suite and T0 be a permutation of T . The APFD for T0 is defined as follows: AP F D = 1 − Pn−1 i=1 Fi n ∗ l + 1 2n (2.1)

Here, n is the number of test cases in T , l is the total number of faults, and F_i is the position of the first test case that reveals the fault i.

The value of APFD can range from 0 to 100 (in percentage). Higher is the APFD value for any ordering of the test cases in the test suite, higher is the rate at which software faults are discovered [60, 175].

Throughout our discussion of regression testing in the rest of this thesis, we will continue to use the notations described in this chapter. Therefore, we will use P0 to denote a modified version of program P under test. It is important to note that any attempt to solve regression testing worth mentioning that any attempt to address the problem of regression testing in a more cost-effective way will essentially be build upon regression test selection, minimization, and prioritization, in conjunction with or in isolation from one another.