• No results found

An Evaluation Strategy of Spacecraft Test Case Adequacy Based on Mutation Testing

N/A
N/A
Protected

Academic year: 2020

Share "An Evaluation Strategy of Spacecraft Test Case Adequacy Based on Mutation Testing"

Copied!
7
0
0

Loading.... (view fulltext now)

Full text

(1)

2017 2nd International Conference on Computer, Mechatronics and Electronic Engineering (CMEE 2017) ISBN: 978-1-60595-532-2

An Evaluation Strategy of Spacecraft Test Case Adequacy

Based on Mutation Testing

Fan YANG

1

, Shi-wei GAO

2

, Jian-hai DU

1

, Jiang-hua LV

1

and Shi-long MA

1

1

State Key Laboratory of Software Development Environment, Beihang University, No. 37, Xueyuan Road, Haidian District, 100191 Beijing, China

2

Beijing Aerospace Control Instrument Research Institute, 100039, China

*Corresponding author

Keywords: Mutation testing, Test case evaluation, Test adequacy, Spacecraft testing.

Abstract. As typical safety-critical system, the trustworthiness of spacecraft has become a widespread concern. The evaluation of trustworthiness depends on the execution of spacecraft test cases, of which the adequacy is of great importance for the spacecraft to conduct effective assessment of trustworthiness. This paper focuses on the evaluation of the adequacy of spacecraft test cases based on mutation testing technique. Mutation testing theory is extended to spacecraft testing, and mapping between spacecraft faults and mutation operators is constructed. A virtual spacecraft testing platform is developed, on which the spacecraft fault injection strategy is given and test cases generated by IPOG algorithms are evaluated by adequacy, verifying the effectiveness of the proposed strategy.

Introduction

As typical safety-critical system, system failure of spacecraft will cause property, life and other significant losses. The trustworthiness of spacecraft has become a widespread concern [1]. Trustworthiness is mainly evaluated and verified through the execution of spacecraft test cases. Therefore, the adequacy of the test cases is of great importance for the spacecraft to conduct effective assessment of trustworthiness.

The purpose of the test cases is to recover and to locate system errors. Therefore, error detection capability is a matter of primary concern in test case designation. The ultimate goal of the various evaluation criteria of test cases is to evaluate the adequacy of test cases, so as to improve the error detection ability. However, current research on test cases focuses mainly on the efficiency and size of the test cases, while less attention is paid to the adequacy of the test cases. Through studying relevant literature, it is found that mutation testing [2] is a very effective method to evaluate adequacy of test cases. Mutation testing is usually applied to the adequacy assessment of software test cases, but it cannot be directly applied to test cases evaluation of complex systems, such as spacecraft.

To resolve the problems above, evaluation of the adequacy of spacecraft test cases is studied based on mutation testing. First of all, mutation testing theory is extended to spacecraft testing, and the mapping between malfunction of spacecraft and mutation operators is built. Secondly, a virtual spacecraft testing platform is designed and implemented, on which the mutation operators are performed and fault injection strategy is given. At last, spacecraft test cases are generated by IPOG algorithms [3]. Finally, test cases are performed on the fault-injected virtual platforms and evaluated by adequacy.

Extension of Mutation Testing to Spacecraft Testing

(2)

Mutation testing is a fault-based software technique. The general principle underlying mutation testing is to generate faults which represent the mistakes that programmers often make [2]. These mistakes, or faults, are introduced by automatically generating variants of the code. Rules that defines the syntactic variation of the code is called mutation operators, resulting in fault versions which are called mutants [4]. Mutation testing will eventually provide a testing criterion called the “mutation score”, which can be used to measure the adequacy of a test case. The mutation score is the ratio of the number of killed mutants over the total number of non-equivalent mutants [5]. The mutation score ranges from 0 to 1, and the higher the mutation score reaches, the stronger the test adequacy is. When the mutation score reaches 1, all faults can be recovered by the test case.

Extension of Mutation Testing’s Hypotheses

The number of faults that exist in one system can be enormous, and it is impossible for the mutation operators to guarantee to generate all kinds of faults. Therefore, mutation testing merely targets on a subset of faults, which are expected to cover all kinds of faults. This theory is supported by two fundamental hypotheses: the Competent Programmer Hypothesis (CPH) [6], [7] and Coupling Effect Hypothesis [7]. When mutation testing is applied to simulate faults of spacecraft, these hypotheses should be satisfied to ensure the effectiveness of simulation.

For the Competent Programmer Hypothesis, we need to assume that the overall design of the spacecraft is correct. Mutation testing focuses on the testing of implementation, while the evaluation of spacecraft design should be taken care of elsewhere in life cycle.

Coupling Effect Hypothesis is described as, complex faults are coupled to simple faults in such a way that a test data set that detects all simple faults in a program will detect a high percentage of the complex faults [8]. Similarly, spacecraft faults are classified into four kinds, including three unit fault and system fault, where the system fault is usually caused by the coupling of the first three [9]. Therefore, it is reasonable that the Coupling Effect Hypothesis is also worked in spacecraft testing.

To summarize, mutation testing is available for the testing of implementation of spacecraft, where the fault detection capability of spacecraft test cases are evaluated.

Mapping between Spacecraft Faults and Mutation Operators

When mutation operators are used to simulate spacecraft faults, a mapping should be built for further analysis. The experiment platform is developed based on C++, and 11 C++ operators are selected [10], as shown in Table 1.

Table 1. C++ Muation operators.

mutation operators description

1 MRO Modify relational operator (>,<, >=, <=, ==, !=) 2 ICS Invert the condition statement

3 MAO Modify arithmetic opretor (+, -, *, /, =, ++, --, +=, -=, *=, /=)

4 MLO Modify logical operator (&&, ||)

5 SRB Set/return different Boolean value (true, false) 6 ISM Invoke different (syntactically similar) method 7 SRA Set/return different attibutes

8 SRV Set/return different variable name 9 SRC Set/return different integar value 10 EPM Exchange two parameters in a method 11 SRS Set/return different string value

(3)
[image:3.612.97.513.109.213.2]

malfuntioning operating units, as incorrent operators in codes. And system fault is a coupling of sensor fault and actuator fault. Connecting the mutation operators and the spacecraft faults, the mapping is described as Figure 1.

Figure 1. Mapping between spacecraft faults and mutation operators.

Spacecraft Fault Injection Strategy Based on Mutation Testing

Design of Virtual Spacecraft Testing Platform

Traditional process of spacecraft testing requires massive execution of test cases, which results in low efficiency and high cost. Development of virtual spacecraft testing platform will effectively reduce test costs and improve test efficiency. According to the actual spacecraft testing scenario, the platform is consist of test client, master test processor, test equipment and the spacecraft under test. The function of each module is as follows:

Test client: send test control instruction, analyze test result.

Master test processor: resolve test control instruction and distribute test instructions to test equipment.

Test equipment: forward instructions to specified spacecraft components and return the result

Spacecraft under test:make the corresponding changes in state parameters according to the test instructions, and return the results of the execution. It should be noted that the virtual spacecraft consists of 19 components, each treated as a C++ object. These components have simulated the behaviors and parameters in the spacecraft with functions and attributes.

The overall structure of the virtual platform is as shown in Figure 2.

[image:3.612.133.479.485.713.2]
(4)

By setting up the virtual spacecraft testing platform, an experiment platform for mutation testing is provided, while the test efficiency has also been greatly improved.

Design of Spacecraft Fault Injection Strategy

Fault injection is performed through the mutation on the virtual platform. Meanwhile, in traditional mutation testing process, there is often a large number of mutants, leading to lower testing efficiency. Two mutant reduction techniques are used to reduce test execution in the experiment.

Selective Mutation: Mutation operators generate different numbers of mutants, and some mutation operators generate far more mutants than others, many of which may turn out to be redundant [11]. According to this theory, same amount of mutants generated by each mutation operator are extracted through random sampling.

Higher Order Mutation: Higher order mutant (HOM) are created by applying mutation operators more than once [12]. HOM contains several faults, so as to reduce several first order mutants into one. For each spacecraft mutant, each components within can be injected once. These co-existing faults will be detected through the parameter error of each component.

Combining the structure of the virtual spacecraft and two mutant reduction techniques, the spacecraft fault injection strategy is described as follows:

1) Each mutation operator corresponds to a mutant, which are 11 mutants in total.

2) For each spacecraft mutant, the corresponding mutation operator is applied to every components.

3) When there is several locations for the mutation operator to work with, choose one randomly. 4) Check and exclude equivalent mutants and syntax errors.

5) Eventually, there will be 11 spacecraft mutants, in which each component is injected ideally.

Experiment

To evaluate the performance of the proposed strategies, mutation is performed on the virtual spacecraft testing platform, on which three test cases generated from IPOG algorithms are executed and evaluated by test adequacy.

IPOG algorithm is an automatic test case generation algorithm based on combinatorial testing [13]. System behaviors are treated as combinations of parameters. Numbers of parameters in a combination is referred as t, a measure of test strength. A test of strength t needs to ensure that every possible combination of parameters occurs at least once in test cases. Therefore, with the increase of t, test case will cover more combinations of parameters, which results in higher capability of fault detection and stronger adequacy. In this experiment, spacecraft instructions from digital management system are treated as parameters to generate test cases using IPOG, which are later evaluated by adequacy.

The development environment is Microsoft Visual Studio 2013, and operating system is Windows 7. Experiment results reported here are for a core Core(TM) Inter(R) i5-4570 processor clocked at 3.20GHZ and 4G memory.

[image:4.612.172.442.630.747.2]

Firstly, fault injection strategy is performed on the virtual spacecraft testing platform. Results of fault injection is shown in Table 2.

Table 2. Fault injection results. Mutant Mutation

Operator

Faults Injected

Equivalent Faults

Non-equivalent Faults

1 MRO 8 3 5

2 ICS 8 0 8

3 MAO 7 0 7

4 MLO 9 4 5

(5)

Mutant Mutation Operator

Faults Injected

Equivalent Faults

Non-equivalent Faults

7 SRA 11 2 9

8 SRV 8 0 8

9 SRC 7 1 6

10 EPM 8 3 5

11 SRS 10 1 9

Total 99 15 84

99 faults are injected into 11 spacecraft mutants in total, where consists of 15 equivalent mutations and 84 non-equivalent mutations.

[image:5.612.171.444.66.181.2]

Secondly, test cases for digital management system are generated with IPOG algorithm, as shown in Table 3.

Table 3. Size of IPOG test cases. t Size of test case

(numbers of instructions)

2 95

3 476

4 1919

5 5889

6 15284

As shown in Table 3, test strength of test cases ranges from 2 to 6. It is found that all the known faults are caused by interactions among 6 or fewer parameters [14]. With the increase of test strength, test adequacy will increase theoretically, and the size of test cases is also growing.

[image:5.612.227.387.255.362.2]

Test cases above are finally executed on each of spacecraft mutants. Analyzing the faults recovered by each test case, the adequacy of the test cases is shown in Table 4, measured by mutation score.

Table 4. Results of test adequacy. t Test case adequacy

(Mutation score)

2 64.23%

3 72.62%

4 88.09%

5 97.62%

6 98.81%

[image:5.612.225.388.450.554.2]
(6)
[image:6.612.137.478.73.243.2]

Figure 3. Growth of mutation scores of different IPOG algorithms.

It can be seen from the experimental results that test adequacy of test cases under different test strength differs on the virtual platform applied with the proposed fault injection strategy. The test adequacy evaluation strategy proposed in this paper can effectively evaluate test cases generated by the IPOG algorithm, and the evaluation result is in line with the actual situation.

Conclusion

This paper proposes an evaluation strategy of test case adequacy in spacecraft testing. Mutation testing theory is extended to spacecraft testing, and mapping between mutation operators and spacecraft faults is constructed. A virtual spacecraft testing platform is designed and implemented to simulate the actual spacecraft testing process, on which a fault injection strategy is given and applied. Finally, test cases generated from IPOG algorithm is executed and evaluated on the platform. The test results has distinguished the test adequacy of each test case under different test strength, which verifies the effectiveness of the proposed strategy.

Reference

[1] Gao S, Lv J, Yang F, et al. Automatic Test Case Generation and Optimization Strategies based on Components with Sequential Constraints[J]. International Journal of Control and Automation, 2016, 9(12): 35-50.

[2] Jia Y, Harman M. An Analysis and Survey of the Development of Mutation Testing [J]. IEEE Transactions on Software Engineering, 2011, 37(5):649-678.

[3] Lei Y, Kacker R, Kuhn D R, et al. IPOG: A General Strategy for T-Way Software Testing[C]// Engineering of Computer-Based Systems, 2007. Ecbs '07. IEEE International Conference and Workshops on the. IEEE, 2007:549-556.

[4] Andrews J H, Briand L C, Labiche Y. Is mutation an appropriate tool for testing experiments?[M]. 2005.

[5] Offutt A J, Pan J. Automatically detecting equivalent mutants and infeasible paths [J]. Software Testing Verification & Reliability, 2015, 7(3):165-192.

(7)

[7] Demillo R A, Lipton R J, Sayward F G. Hints on Test Data Selection: Help for the Practicing Programmer [J]. Computer, 1978, 11(4):34-41.

[8] Offutt A J. Investigations of the software testing coupling effect [M]. ACM, 2008.

[9] Ouyang G, Ni M, Sun C, et al. Spacecraft fault modeling and application [J]. Aerospace Control, 2011, 29(5):22-29.

[10] Memon A M, Xie Q. Studying the Fault-Detection Effectiveness of GUI Test Cases for Rapidly Evolving Software[J]. IEEE Transactions on Software Engineering, 2005, 31(10):884-896.

[11] Offutt A J, Rothermel G, Zapf C. An experimental evaluation of selective mutation[C]// International Conference on Software Engineering, 1993. Proceedings. IEEE, 1993:100-107.

[12] Jia Y, Harman M. Constructing Subtle Faults Using Higher Order Mutation Testing[C]//Eighth IEEE International Working Conference on Source Code Analysis and Manipulation. IEEE, 2008:249-258.

[13] Lei Y, Kacker R, Kuhn D R, et al. IPOG: A General Strategy for T-Way Software Testing[C]// Engineering of Computer-Based Systems, 2007. Ecbs '07. IEEE International Conference and Workshops on the. IEEE, 2007:549-556.

Figure

Figure 1. Mapping between spacecraft faults and mutation operators.
Table 2. Fault injection results.
Table 4. Results of test adequacy.
Figure 3. Growth of mutation scores of different IPOG algorithms.

References

Related documents

When water level reaches 94 m elevation at Zodia Creek reservoir, water starts to flow from Zodia Creek Detention Basin to Potami Creek by means of the diversion channel..

This paper presents a new approach to precise assessment of the effect of the broken rotor bars on the stator currents in an induction machine during startup through direct

In addition to their role in communicating with athletes, using both parents and coaches as opinion leaders strengthens the dissemination of concussion prevention and

Conclusions: The current study demonstrates that postoperative pulmonary complications represent a significant source of morbidity and incremental cost after major small intestinal

Uncertainty is present in this analysis through failure rate inputs (some were based on field data from a particular model of wind turbines, whereas as a formal methodology was

This Article develops a construct of judges as gatekeepers in corporate and securities litigation, focusing on the last period—or settlement stage— of the cases. Many accounts of

Accelus Audit Manager’s AutoAudit is used by hundreds of companies, non-profits, and governmental organizations across the globe to implement audit plans that balance board audit

This world of convenience is based on the accessibility of information and commerce , where online streaming and torrented films and video disregard the cinema for the