Combinatorial Testing

Combinatorial testing [CG12] is a testing technique that requires covering all t-sized tuples of values out of n parameter attributes or properties modelled after the input parameters or the configuration domain of a system under test.

Two forms of combinatorial testing:1) use combinations of configuration pa- rameter values, or 2) combinations of input parameter values. There are four combination strategy criteria which are used in combinatorial software testing [KKL13]:

• All Combinations Coverage: Every possible combination of values of the parameters must be covered.

• Each Choice Coverage: Each parameter value must be covered in at least one test case.

• Pairwise Coverage (2-way): Given any two parameters, every possible combination of values of these two parameters must be covered in at least one test case.

• T-way Coverage: Given any t parameters, every possible combination of values of these t parameters must be covered in at least one test case. Kuhn et. al [KLK08] show the practical use of combinatorial testing and its feasibility for small to medium-sized modules. They show also the automated generation of tests that provide combinatorial coverage. A tool support for automated combinatorial testing called ACTS1 _{was developed by Kuhn at the}

American National Institute of Standards and Technology to generate combina- tion sets of t parameters with nvalues.

2.3.2.4. Software Safety Testing

Software safety testing [NAS04;Lut00] is a crucial process in developing safety- critical systems to verify whether a software system meets its safety requirements. Safety-critical software should be tested extensively to ensure that the potential software-related hazards have been eliminated or controlled to a low level of risk. The term software safety testing [NAS04] was introduced and implies that software testing should not only address functional requirements, but also the software safety requirements. Therefore, the process for testing safety-critical software combines conventional testing and safety analysis approaches to focus the testing efforts in a specific way to address the safety of the software and test the critical risky situations. In the literature, the software safety testing is also known as “risk-based testing”.

The integration between safety analysis and software testing approaches is not trivial. Erdogan et al. (2014) [Erd+14] presented a systematic literature review on the combined use of risk analysis approaches to support testing and testing approaches to support risk analysis. They identified only 32 scientific papers which focus on the combined use of risk analysis and testing with dif- ferent purposes. The results highlight the value of the combination between risk analysis and testing approaches. Furthermore, the results show that there is a demand for developing more structured and rigorous approaches to use risk-based analysis to support software testing and developing tool support for the combined risk-based testing. Felderer and Schieferdecker (2014) [FS14] presented a taxonomy of risk-based testing and provided a framework to under- stand and categorize risk-based testing approaches to aid their selection and adoption for specific purposes.

Many existing testing approaches and tools do not incorporate information from safety analysis. Some of them rely on traditional safety analysis such as Fault Tree Analysis (FTA) [Ves+81] and Failure Mode and Effects and Criticality Analysis (FMECA) [FME67] which are grounded in reliability theory and com- monly used for the purpose of safety-based testing. However, these approaches

focus only on single component failures and they have limitations to cope with complex systems including software. Leveson [Lev11] noted that the primary safety problem in software-intensive systems is not software “failure” but the lack of appropriate constraints on software behavior. The solution is to identify the required constraints and enforce them in the software and overall system design. Thus, STPA [Lev11] was developed to overcome the limitations of the traditional techniques in addressing the unsafe scenarios of complex systems. One of the objectives of this dissertation is to investigate the application of STPA on deriving software safety requirements and develop a novel approach to integrate STPA in the software development process activities, especially the formal verification and testing activities.

C

��

����

3

S���� �� ��� A��

“

Software is hazardous if it can cause other compo- nents to become hazardous or it is used to control a hazard.

”

— NASA S������� S����� G��������

This chapter provides an overview of the closely related work to this disserta- tion which has been done in STPA safety analysis, the combined application of safety analysis, formal verification and software testing.

3.1. Generating the Unsafe Control Actions in STPA

Thomas [Tho13] introduced an extended approach to STPA with the purpose of identifying unsafe control actions in STPA Step 1 based on the combination of the process model variables of each controller in the control structure diagram. A combination of process model variables is called a context. Two contexts of control actions are proposed: Provided control action and not provided control action. The control action will be hazardous only in a certain context. The main problem of context tables is the difficulty in defining the combination for a large number of values of the process model variables which have an effect on the safety of control actions. To solve this problem, we [AWL15] developed an algorithm based on the concept of combinatorial testing [KKL13] to automatically generate the context tables and to allow safety analysts to identify a minimal combination of the process model variables and automatically generate the unsafe control actions. The safety analysts can add and apply constraints and Boolean relations to the generated context tables to ignore some unnecessary combinations from these tables. Furthermore, we explain how to automatically

refine the unsafe control actions based on the results of the context tables and generate unsafe scenarios for each unsafe control action. The unsafe scenarios will be automatically translated into the refined safety constraints and expressed them into formal specification in LTL. Both algorithms are implemented as an Eclipse plug-in and integrated with XSTAMPP platform.

3.2. Combination of Safety Analysis Techniques and Model Checking

The combination of the safety analysis techniques and model checking for safety verification of complex systems purposes is not something new. There is a number of considerable work has been done in the field of integrating model checkers with traditional safety analysis approaches such as FTA and FMEA. In the following, we discuss the most related work which incorporates the safety analysis and model checking approaches:

Sere and Troubitsyna (1999) [ST99] showed how to formalise the hazard analysis results in a formal system specification, which are semantically different from the specification terms of the controlling software. They used formal methods like the action system formalism [Bac90] (action systems model) which describes parallel programs and reactive programs as a modeling technique of the system behavior and they showed how the results of fault tree analysis (FTA) as a hazard analysis approach can be encoded into the system formalism.

Tribble et al.(2002) [TLM02] proposed a technique for combing traditional safety analysis techniques such as Functional Hazard Analysis (FHA), FTA and FMEA with the formal method approach to conduct a software safety analysis of the flight guidance system requirements model. They used the NuSMV [Cim+00] model checker to verify the safety properties derived from the traditional safety analysis techniques. They also verified some of safety properties with PVS (Prototype Verification System) theorem prover.

Bieber et al. (2002) [BCS02] combined the fault tree analysis and the model checking for safety assessment of complex systems. They designed a language which is called Altarica to model the behavior of systems when faults occur. They used the fault tree analysis to derive the requirements that constraint the design of the system controllers. Then, they used the model checking to assess the designed controller.

Ortmeier et al. (2004) [Ort+04] proposed an integrated approach called For M oSAwhich combines safety analysis and formal methods and provides all engineering practices of traditional safety analysis, temporal logic and ver- ification. ForMoSA is built based on fault tree analysis as a safety analysis approach.

Bozzano and Villafiorita (2006) [BV06] presented a safety assessment plat- form called FSAP/NuSM V SAbased on the NuSMV-SA model checker. The platform integrates the activities of the model design and safety analysis and supports the activities of formal verification. It also supports generating the fault trees and link their top events directly to failure causes.

Recently, Shariva and Papadopoulos (2015) [SP15] proposed an approach that combines the new Symbolic Model Verifier (NuSMV) model checker [Cim+00] with the Hierarchically Performed Hazard Origin and Propagation Studies (HiP- HOPS) safety analysis technique, which automatically constructs fault trees and FMEA from a system model. They showed how such a combination between these approaches can help to verify the design of a system at an early stage in the design phase of a safety critical system. They translated the model of Hip-HOPS into an abstract state machine model. Next, they manually converted the abstract state machine model of a brake-wire system into an SMV Model to be verified by NuSMV. However, its verification is focused on verifying the system Hip-HOPS at the system level.

Most of the above integrated approaches were based on the reliability engi- neering techniques (e.g. FTA and FMEA) to identify the failure conditions to be verified by model checkers. These approaches aimed at increasing confidence in a software by showing that some classes of errors which are identified by traditional hazard analysis techniques are not present. Even though the model checker shows that the system fulfilled all requirements, that does not necessarily mean the system is safe. In other words, proofing the correctness of system from these errors cannot make it safe and ensure the safety operations of the system.

According to the STAMP accident model assumptions [Lev11], the accidents with systems are caused by software flaws, software specification errors and uncontrolled interaction between different components which form the system rather than failures of a single component [Lev11]. We differentiate our work here from the aforementioned approaches by identifying the STPA-generated

unsafe scenarios for each unsafe control action of software controller in the control structure diagram, deriving the detailed software safety requirements at the system level and verifying the software design and implementation against the STPA-generated software safety requirements which constrain the software from these unsafe behaviors. To the best of our knowledge there is no work that integrates STPA safety analysis with software verification. Moreover, we also are not aware of other attempts to automatically transform the software safety requirements derived during safety analysis into formal specifications in LTL.

3.3. Translating Simulink Models into Verification Models

A considerable amount of work has been done on translating Simulink models into models supported by formal verification approaches. In the following, we will discuss the most related work:

Banphawatthanarak et al. [BKB99] developed a tool called sf2smv that gener- ates input for the symbolic model checker SMV [McM93] from Stateflow models. In our work, we use the same concept for translating Simulink’s Stateflow into the SMV model. As sf2smv is not available yet, however, it is difficult to compare it with our approach in detail.

Meenakshi et al. [MBR06] discussed the principles of translating Simulink models into an input language of a suitable model checker and providing re- verse translation of traces violating requirements into Simulink notation for playback. They developed a translator from Simulink to the model checker NuSMV [Cim+00]. The translator takes a Simulink model as input and gen- erates an equivalent NuSMV model. However, this translator is restricted only to discrete Simulink models and support only the basic blocks of Simulink (e.g. logical block or Selector block) that forms the finite state model of a system. Moreover, the translator does not support the translation of Simulink Stateflow into the input language of NuSMV.

Chen and Dong [CD06] proposed a systematic approach to translate Simulink diagrams to Timed Interval Calculus (TIC) [Fid+98], a notation extending Z to support real-time system specification and verification. The translated TIC specification covers the functional and timing aspects of the Simulink blocks. This work aims to guarantee the correctness of control systems. However, this

work does not cover Stateflow diagrams.

Chen et al. [Che+12] proposed an approach to systematically translate State- flow diagrams to into CSP# [Sun+09a]. They developed a translator which is integrated inside the PAT model checker [Sun+09b] to automate the process with support of different Stateflow features. This work aims to validate the functional correctness of Stateflow diagrams by detecting the bugs in the State- flow model. The properties to be verified are declared in the CSP# model with preprocessor such as #define. The translation process preservers the execution semantics of Stateflow and considers advanced Stateflow modelling features such as implicit events and history junctions.

Ferrante et al. [Fer+12] developed a modified tool called Parallel NuSMV (PNuSMV) based on NuSMV model checker [Cim+00] that integrates the ManySAT parallel STA solver [HJS08]. This tool is part of the formal speci- fication verification framework for the formal verification of Simulink/Stateflow models. The tool translates a subset of Simulink blocks (e.g. logical operators and arithmetic blocks) into the NuSMV meta model. The interesting properties are expressed as temporal logic to be verified with the PNuSMV tool. However, this work does not consider translating the Stateflow model into the NuSMV specifications.

We use a similar principle of transforming the Simulink’s Stateflow model into an intermediate model with consideration of the state decomposition (AND_STATE and OR_STATE) and the attached actions (Entry, During and Exit). The main contribution of their work, however, is an approach to model check Simulink models which is not our main focus of our approach. Our contri- bution is to visualise the STPA process model, which is created during the safety analysis, with the Stateflow notation and check its correctness against the STPA software safety requirements. In this way, we ensure that both models contain the same specifications (e.g. names of states, variables and control actions) before using it for generating test cases.

In conclusion, the existing work provide a great basis for our approach but are different in their focus. They concentrate on model checking Simulink models, not the integration with safety analysis or testing. To the best of our knowledge, there is no existing work on constructing the Stateflow diagram based on the information derived during a safety analysis for test case generation.

3.4. Risk-based Software Testing

There are several software safety test techniques in the literature that combine safety analysis principles with model-based testing. Most of them use the term “Risk-based Testing", which combines risk analysis approaches such as FTA,

FMECA or Markov chains with software testing approaches (e.g. model-based testing) to create a prioritization criterion for generating test cases.

Redmill [Red04] explored the benefits of risk-based testing as the basis of test planning in the software testing process and how to understand the risks of the system to focus test efforts. He does not show how to generate the test cases from the risk analysis approach.

Zimmermann et al. [Zim+09] proposed a refinement-based approach to the reliability analysis of safety-critical systems. They used statistical testing as a model-based testing technique and a Markov chain model to model the system under test. They also used FTA and FMECA as risk-based analysis techniques to identify the critical situations that represent high risk.

Kloos et al. [KHE11] proposed a model-based testing approach which uses the information derived from FTA in combination with a system model to generate the risk-based test cases. They used FTA to select, generate and prioritize the test cases. They derived the test cases from the combination of fault trees and a basic system behavior model called the “base model".

Our approach uses a similar idea of combining a risk analysis approach with model-based testing. The main difference is that we employ STPA for safety analysis which is based on system and control theory rather than reliability theory like FTA and FMEA. STPA copes with the analysis of complex, modern systems and tackles the dynamic behavior of the system by treating safety as a control problem. Furthermore, STPA provides an abstract model of the system under analysis called the safety control structure diagram which views all main interacting components including the software components of the system. This allows us to directly construct the test model from the control structure diagram and constrain its transitions with the STPA-generated safety requirements.

3.5. Generating Test Cases Using Statechart Diagrams

Over the years, many approaches have been developed to generate test cases from statechart diagrams. The idea behind these approaches is to transform the statechart diagram into an Extended Finite State Machine (EFSM) and generate the test cases from this model. In the following, we will discuss the most related work:

Ural [Ura87] proposed a method to transform the extended finite state ma- chines into a flow graph and generate test sequences. This method is based on the principles of using data flow analysis techniques in software reliability [FO76] to trace the flow graph and generate test cases.

Bourhfir et al. [Bou+97] proposed a unified method for automatic executable test case and test sequences generation which combines both control and data flow testing techniques with control flow criteria (Unique Input Output) and the all-du paths coverage criterion. Their approach generates only executable test cases for EFSM-specified systems by using symbolic evaluation techniques.

Kim et al. [Kim+99] proposed an approach to generate test cases from UML state diagrams based on the conventional control and data flow analysis. The authors have first transformed the state diagrams into EFSM with consideration of the hierarchical and concurrent structure of states (flattened and broadcast) of the UML state diagrams. Then, the EFSM are transformed into the flow graphs. They applied the conventional data flow analysis techniques to the resulting flow graph to generate the test cases. However, they focused only on identifying possible control and data flow and not the values of input variables.

Hong et al. [Hon+00] developed a method for the selection of test sequences from statecharts. The method is based on the STATEMATE semantics of stat- echarts by Harel [HN96]. The basic idea is to transform the statechart into an EFSM which contains all the possible runs of the statechart. The authors have considered the input variables in the EFSM which was generated from the state machine diagram. The resulting EFSM model will then be transformed into a flow graph to generate test sequences that cover all associations between definitions and uses of each variable that appear in the original state machine. The authors used the existing method of Ural [Ura87] to transform the EFSM into a flow graph that models the flow of both the control and the data in the

statechart.

In conclusion, our approach uses a similar principle of generating test cases from the test model by using graph search algorithms (depth-first search, breadth- first search and both combined) which are presented in the aforementioned mentioned approaches. However, we choose different test coverage criterion to generate test cases such as all states coverage, all transition conditions coverage

In document A system-theoretic safety engineering approach for software-intensive systems (Page 54-71)