Verification & Validation
2.1. Software Quality
2.2.4. Formal Methods
The term “formal methods” is used to refer to any activities that rely on mathematical representations of software including formal specification and verification. In the 1980s, many software engineering researchers proposed that using formal development methods was the best way to improve software quality. They predicted that by the 21st century, a large proportion of software would be developed using formal method. This is not true due to the fact that [54]:
‐ Successful SE techniques. The use of other SE methods such as structured methods or configuration management has resulted in improvements in software quality.
‐ Limited scope of formal methods. Formal methods are not well suited to specifying user interfaces and user interaction, and nowadays the user interface component has become a greater and greater part of most systems.
‐ Limited scalability of formal methods. Projects that have used these techniques have mostly been concerned with relatively small, such as critical kernel systems. As systems grow, the effort required to develop a formal specification grows excessively. Formal methods comprise a set of mathematically‐based techniques for the specification and verification of software systems. Both formal specification techniques and formal verification techniques are widely referred to as formal methods collectively in literature. The existence of formal specifications is a prerequisite for formal verifications. 2.2.4.1. Formal Specifications Formal specifications produce an unambiguous set of product specification such as customer requirements, environmental constraints and design intentions. They can be produced in several different forms [132].
Firstly, descriptive specifications are focused on the properties or conditions associated with software products and their components. There are several kinds of descriptive specifications, such as Entity‐Relationship (ER) diagrams are commonly used to describe product components and connections. These diagrams show data entities, their associated attributes and the relations between them [132]. Logical specifications focus on the formal properties associated with different product components or the product as a whole. They are logical statements or conditions associated with the states, or program states, of programs or program segments. The basic elements of these logical specifications are the pre‐conditions, post‐conditions, and invariants, which are generally associated with program code. Some examples are the Z specification [133] and VDM (Vienna Definition Method) [81] languages. Contracts it is also a kind of logical specification. In addition, the Object Constraint Language (OCL) was initially designed as a logical specification for UML [144]. Nowadays it is part of the Meta‐Object Facility (MOF) standard by the Object Management Group (OMG). Algebraic specifications focus on functional computation carried out by a program or program‐segment and related properties, for example Larch [57] or the OBJ family [46]. Syntactic specifications are used to describe languages used in computing, such as programming languages. The BNF (Backus–Naur Form) notation is universally used for syntactic specifications.
Secondly, operational specifications are focuses on the required behaviour of the software systems, for example: Data Flow Diagrams (DFDs) specify information flow among the major functional units. They are used to show how data flows through a sequence of processing steps. Unified Modelling Language (UML) provides a general‐purpose visual modelling language used to specify, visualize, construct and document software artefacts. Finite‐State Machines (FSMs) is a behavioural model composed of a finite number of states, transitions between those states, and actions. Labelled Transition Systems (LTS) is also a state machine representation. The main difference between FSM and LTS is that while an FSM has transitions labelled with pairs of (input, output), an LTS specifies interactions, which can, but need not be interpreted as input or as output. Similarly, Petri Nets are considered as a special kind of FSMs with two distinct types of nodes called places and transitions [119]. Graphs‐based models are the abstract representation of a set of points (vertices or nodes) connected by lines (edges or links). Graphs or digraphs (i.e. directed graphs) are sometimes used in the literature to model the behaviour of a software system. For instance, an Event‐Flow Graph (EFG) is used in GUI testing since it represents all the possible interactions among the events in a GUI. Similarly, an
Event Interaction Graph (EIG) contains nodes, one for each system‐interaction event in the GUI. An Event Semantic Interaction Graph (ESIG) contains nodes that represent events, directed edges from nodes shows that there is a semantic relationship from the event represented by nodes. Specification and Description Language (SDL) specifies the description and the behaviour of reactive and distributed systems. It provides both a graphical Graphic Representation (SDL/GR) and a textual Phrase Representation (SDL/PR) [123]. A system is specified as a set of interconnected abstract machines which are extensions of FSM.
2.2.4.2. Formal Verification
Formal verification checks the conformance of software design or code to the formal specifications, ensuring that the software is fault‐free with respect to its formal specifications. Axiomatic correctness (Hoare logic) [65][147] works with the logical specifications of programs or formal designs by associating with each type of program or design elements with an axiom to prescribe the logical transformation of program state before and after the execution of this element type. Weakest pre‐conditions [36][56] works focusing on the goal or the computational result that is captured by the final state of the execution sequence. A series of backward chaining operations through the use of the so‐called weakest pre‐conditions transform this final state and its properties into an initial state and its properties.
Functional correctness (program calculus) [107] is similar to the axiomatic approach in the sense that some basic axioms or meanings of program elements are prescribed. Symbolic execution is used to connect these elements in a program. It involves executing (interpreting) a program symbolically. During the execution, the values of variables are held in algebraic form and the outcome of the program is represented as one or more expressions. Decisions are handled by following both outcomes while remembering the condition value corresponding to each of the paths now being followed separately. At the end of the evaluation, there will be two facts about each path through the program: a list of the decision outcomes made along the path and the final expression of all the variables expressed algebraically. Together these define the function of the program and can be compared with the required function.
Semi‐formal techniques check certain properties instead of proving the full correctness of software. For example, model checking, which is an approach to automatically or algorithmically check certain properties for some software systems [138]. In model checking, a software system is modelled as a FSM, with some property of interest expressed as a suitable formula, or a proposition, defined with respect to the FSM. After that, the model checker runs an algorithm to check the validity of the proposition.
2.3. Software Testing
Software testing consists of the dynamic evaluation of the behaviour of a program on a finite set of test cases, suitably selected from the usually infinite executions domain, against the expected behaviour [1]. The key concepts of this definition are depicted as follows:
‐ Dynamic: The SUT is executed with specific input values to find failures in its behaviour. Thus, the actual SUT should ensure that the design and code are correct, and also the environment, such as the libraries, the operating system and network support, and so on. ‐ Finite: Exhaustive testing is not possible or practical for most real programs. They usually
have a large number of allowable inputs to each operation, plus even more invalid or unexpected inputs, and the possible sequences of operations are usually infinite as well. Testers must choose a number of tests so that we can run the tests in the available time. ‐ Selected: Since there is a huge or infinite set of possible tests but can afford to run only a small fraction of them, the key challenge of testing is how to select the tests that are most likely to expose failures in the system. ‐ Expected: After each test execution, it must be decided whether the observed behaviour of the system was a failure or not.
2.3.1. Testing Levels
Typically, a commercial software system has to go through three stages of testing [132]: 1. Development testing, where the SUT is tested during development to discover defects.This stage is performed by software engineers (i.e. programmers, testers, system designers, and so on).
2. Release testing, where a separate testing team tests a complete version of the system before it is released to users. The aim of this stage is to check that the SUT meets its requirements.
3. User testing, in which potential or real users of the system test it in their own environment. 2.3.1.1. Development Testing Development testing includes all testing activities that are performed by the team developing the system. In this stage, testing may be carried out at three levels of granularity [132]: 1. Unit testing, where individual program units are tested. Unit testing should focus on the functionality of objects or methods.
2. Integration testing, where units are combined to create composite components.