Verification & Validation
3. System testing, where all of the components are integrated and the system is tested as a whole System testing should focus on testing components interactions
2.3.2. Testing Methods
Testing methods (or strategies) define the approaches for designing test cases. They can be responsibility based (black‐box), implementation based (white box), or hybrid (grey‐box) [120]. Black‐box techniques design test cases on the basis of the specified functionality of the item to be tested. White‐box ones rely on source code analysis to develop test cases. Grey‐box testing designs test cases using both responsibilities based and implementation based approaches. 2.3.2.1. Black‐Box Testing Black‐box testing (also known as functional or behavioural testing) is based on requirements with no knowledge of the internal program structure or data. Black‐box testing relies on the specification of the system or the component that is being tested to derive test cases. The system is a black‐box whose behaviour can only be determined by studying its inputs and the related outputs [82]. There are a lot of specific black‐box testing techniques; some of the most well‐known ones are described as below.
Systematic testing refers to a complete testing approach in which SUT is shown to conform exhaustively to a specification, up to the testing assumptions. It generates test cases only in the limiting sense that each domain point is a singleton sub‐domain [82]. Inside this category, it can be found for example pairwise (all‐pairs) testing, which is a combinatorial testing method that, for each pair of input parameters to a SUT, tests all possible discrete combinations of those parameters. Other systematic black‐box testing techniques are equivalence partitioning and boundary value analysis (described in section 2.5.2).
Random testing is literally the antithesis of systematic testing: the sampling is over the entire input domain. Duran and Ntafos [37] have demonstrated, with both theoretical and empirical evidence, that random test case selection criteria can be as effective at defect detection as partitioning methods. This means of testing seems a better choice than systematic testing in two general situations [58]: i) Sparse sampling: for a large, unstructured input domain. ii) Persistent state: the usual theoretical assumption is that software is reset between tests, so that results are repeatable. Fuzz testing is a form of black‐box random testing which randomly mutates well‐formed inputs and tests the program on the resulting data [54]. It delivers randomly sequenced and/or structurally bad data to a system to see if failures occur.
Graphic User Interface (GUI) testing is the process of ensuring the specification of software with a graphic interface interacting with the user. GUI testing is event driven (e.g. mouse movements or menu selections) and provides a front end to the underlying application code through messages or method calls [98]. GUI testing at unit level is used typically at the button level. GUI testing at system level exercises the event‐drive nature of the SUT. GUI applications offer a small benefit for testers: there is little need for integration testing. GUI testing is mainly used for ensuring the correctness the entire system’s functionality, safety, robustness, and usability [82].
Smoke testing is the process of ensuring the main functionality of the SUT. A smoke test case if the first to be run by testers before accepting a build for further testing. Failure of a smoke test case will mean that the build is refused by testers. The name of “smoke testing” derives electrical system testing, whereby the first test was to switch on and see if it smoked [42]. Sanity testing determines whether or not it is reasonable to proceed with further testing. The difference with smoke testing it is that if a smoke test fails, it is impossible to conduct a sanity test. In contrast, if the sanity test fails, it is not reasonable to attempt more rigorous testing. Both sanity tests and smoke testing are ways to avoid wasting time and effort in more rigorous testing. The typical example of sanity testing for development environment is the “Hello world” program.
2.3.2.2. White‐Box Testing
White‐box testing (also known as structural testing) is based on knowledge of the internal logic of an application's code. It determines if the program‐code structure and logic is faulty. White‐ box test cases are accurate only if the tester knows what the program is supposed to do. White‐box testing does not account for errors caused by omission [60].
Black‐box testing uses only the specification to identify use cases, while white‐box testing uses the program source code (implementation) as the basis to of test cases identification. Both approaches used in conjunction should be necessary in order to select a good set of test cases for the SUT [60]. Hence, the following table summarizes the main differences between black and white‐box testing approaches:
Table 1. Black‐Box Vs. White‐Box Testing
Feature Black‐box White‐box
Tester visibility Specification/Requirements (input & output)
Code
Defect type Failures Faults
Defect identification No, debugging in needed to find the fault which causes the failure Yes, test cases identify the specific LOC involved Usually done by Independent tester team Developers
Some of the most significant white‐box techniques are described as follows. Code coverage defines the degree of source code which has been tested, for example in terms of percentage of Lines of Code (LOC). There are several criteria for the code coverage:
‐ Decision (branch) Coverage. Control structure (e.g. if‐else) coverage granularity. ‐ Condition coverage. Boolean expression (true‐false) coverage granularity. ‐ Paths coverage. Every possible route coverage granularity. ‐ Function coverage. Program functions coverage granularity. ‐ Entry/exit coverage. Call and return of the coverage granularity. Fault injection is the process of injecting faults into software to determine how well (or badly) some SUT behaves [42]. Defects can be said to propagate in that their effects are visible in program states beyond the state in which the error existed (a fault became a failure).
Mutation analysis validates tests and their data by running them against many copies of the SUT containing different, single, and deliberately inserted changes. Mutation analysis helps to identify omissions in the code [42]. 2.3.2.3. Grey‐Box Testing Grey‐box testing is the technique that uses a combination of black‐box and white‐box testing. Grey‐box testing is not black box testing, because the tester does know some of the internal workings of the SUT. In grey‐box testing, the tester applies a limited number of test cases to the internal workings of the software under test. In the remaining part of the grey‐box testing, one takes a black‐box approach in applying inputs to the SUT and observing the outputs. 2.3.2.4. Non‐Functional Testing
The non‐functional aspects of a system can require considerable effort to test and perfect. Within this group it can be found different means of testing, for example performance testing conducted to evaluate the compliance of a SUT with specified performance requirements [42]. These requirements usually includes constraints about the time behaviour (capability of the software product to provide appropriate response and processing times and throughput rates when performing its function, under stated conditions) and resource utilization (capability of the software product to use appropriate amounts and types of resources when the software performs its function under stated conditions). Performance testing may measure response time with a single user exercising the system or with multiple users exercising the system. Load testing is focused on increasing the load on the system to some stated or implied maximum load, to verify the system can handle the defined system boundaries. Volume testing is often considered synonymous with load testing, yet volume testing focuses on data. Stress testing exercises beyond normal operational capacity to the extent that the system fails, identifying actual boundaries at which the system breaks. The aim of stress testing is to observe how the system fails and where the bottlenecks are [132].
Security testing tries to ensure the following concepts: confidentiality (protection against the disclosure of information), integrity (ensuring the correctness of the information), authentication (ensuring the identity of the user), authorisation (determining that a user is allowed to receive a service or perform an operation), availability (ensuring that the system perform its functionality when required) and non‐repudiation (ensuring the denial that an action happened).
Usability testing focuses on finding user interface problems, which may make the software difficult to use or may cause users to misinterpret the output.
Accessibility testing is the technique of making sure that your product is accessibility (ability to access to the system functionality) compliant.
2.4. Testing of Web Applications
Web‐based applications (or simple web applications) shares the same objectives of traditional application testing, i.e. to ensure quality and finding defects in the required functionality and services. A web application can be viewed as a client‐server distributed system, with the following main characteristics [34]:‐ A wide number of users distributed all over the world accessing concurrently.
‐ Heterogeneous execution environments (different hardware, network connections, operating systems, web servers and browsers).
‐ A heterogeneous nature, because of different technologies (programming languages and models), and different involved components (generated from scratch, legacy ones, hypermedia components, Commercial Off‐The‐Shelf ‐COTS‐ and so on).
‐ Dynamic nature. Web pages can be generated at run time according to user inputs and server status.
The aim of web testing consists of executing the application using combinations of input and state to reveal failures. These failures are mainly caused by faults in the running environment or in the web application itself. The running environment mainly affects the non‐functional requirements of a web application (e.g. performance, stability, or compatibility), while the web application is responsible for the functional requirements. Therefore, web testing has to be considered from this two distinct perspectives (functional and non‐functional), since they are complementary and not mutually exclusive. All in all, different types of testing have to be executed to reveal these diverse types of failures [34].