Verification & Validation
2.1. Software Quality
2.1.2. Quality Assurance
Quality Assurance (QA) is a “systematic, planned set of actions necessary to provide adequate
confidence that the software development and maintenance process of a software system product conforms to established specification as well as with the managerial requirements of keeping the schedule and operating within the budgetary confines” [47]. QA is primarily
concerned with defining or selecting standards that should be applied to the software development process or software product. Moreover, QA process selects the V&V activities, tools and methods to support these standards [132].
V&V is a set of activities carried out with the main objective of withholding products from shipment if they do not qualify. In contrast, QA is meant to minimize the costs of quality by introducing a variety of activities throughout the development and maintenance process in order to prevent the causes of errors, detect them, and correct them in the early stages of development. As a result, QA substantially reduces the rates of non‐qualifying products. All in all, V&V activities are only a part of the total range of QA activities [47]. 2.1.2.1. Quality Standards Various quality standards have been proposed to accommodate these different quality views and expectations. This section describes the ISO/IEC‐9126 (maybe the mostly influential in the SE community to date) and its successor, the ISO/IEC‐25000. 2.1.2.1.1. ISO/IEC‐9126 ISO/IEC‐9000 is a family of standards for quality management systems. In 1991, ISO published its first international consensus on the terminology for the quality characteristics for software product evaluation (ISO 9126 on Software Product Quality Characteristics and Guidelines for their Use) [77]. Afterwards, from 2001 to 2004, ISO published an expanded four‐part version, containing both hierarchical framework for quality models and metrics for these models. The current version of the ISO/IEC‐9126 series now consists of one International Standard (IS) [73] and three Technical Reports (TR) [74][75][76]. The ISO/IEC‐9126 quality model distinguishes three different views on software product quality:
‐ Internal quality: concerns the properties of the system that can be measured without executing it.
‐ External quality: concerns the properties of the system that can be observed during its execution.
‐ Quality in use: concerns the properties experienced by its users/customers during operation and maintenance of the system.
Ideally, the internal quality determines the external quality and external quality determines quality in use, as depicted in the following picture:
Figure 5. ISO/IEC‐9126 Quality Lifecycle
The first document of the ISO/IEC 9126 series (quality model) contains two‐part quality model for software product quality [5]: i) Internal and external quality model; ii) Quality in‐use model. The first part of the two‐part quality model determines six characteristics in which they are subdivided into twenty‐seven sub‐characteristics for internal and external quality [73]. Measures for estimating external, internal, and quality‐in‐use characteristics are listed in three technical reports accompanying the standard quality model. ISO/IEC 9126‐2 [74], ISO/IEC 9126‐ 3 [75], and ISO/IEC 9126‐4 [76] define respectively: external, internal, and quality in use quality metrics. Quality model of ISO/IEC‐9126 divides the internal and external software product quality into six top‐level quality features:
Figure 6. ISO/IEC‐9126 Quality Model (External and Internal Quality)
The following definitions have been extracted directly from the norm ISO/IEC‐9126‐1 [73]: ‐ Functionality: “The capability of the software product to provide functions which meet
stated and implied needs when the software is used under specified conditions”. The sub‐
characteristics include:
‐ Reliability: “The capability of the software product to maintain a specified level of
performance when used under specified conditions”. The sub‐characteristics include:
‐ Usability: “The capability of the software product to be understood, learned, used and
attractive to the user, when used under specified conditions”. The sub‐characteristics
include:
‐ Efficiency: “The capability of the software product to provide appropriate performance,
relative to the amount of resources used, under stated conditions”. The sub‐characteristics
include:
‐ Maintainability: “The capability of the software product to be modified. Modifications may
include corrections, improvements or adaptation of the software to changes in
External and Internal Quality Functionality Suitability Accuracy Interoperability Security Functionality Compliance Reliability Maturity Fault Tolerance Recoverability Reliability Compliance Usability Understandability Learnability Operability Attractiveness Usability Compliance Efficiency Time Behaviour Resource Utilisation Efficiency Compliance Maintainability Analysability Changeability Stability Testability Maintainability Compliance Portability Adaptability Installability Co‐Existence Replaceability Portability Compliance
Internal
Quality
External
Quality
influences depends onQuality
in Use
influences depends on
environment, and in requirements and functional specifications”. The sub‐characteristics
include: ‐ Portability: “The capability of the software product to be transferred from one environment to another”. The sub‐characteristics include: The attributes of quality in use are categorised into the following four characteristics: Figure 7. ISO/IEC‐9126 Quality Model (Quality in Use) ‐ Effectiveness: “The capability of the software product to enable users to achieve specified goals with accuracy and completeness in a specified context of use”. ‐ Productivity: “The capability of the software product to enable users to expend appropriate amounts of resources in relation to the effectiveness achieved in a specified context of use”. ‐ Safety: “The capability of the software product to achieve acceptable levels of risk of harm to people, business, software, property or the environment in a specified context of use”. ‐ Satisfaction: “The capability of the software product to satisfy users in a specified context of use”. 2.1.2.1.2. ISO/IEC‐25000
ISO/IEC‐9126 presents some weaknesses found by researchers and practitioners [4]. Since 2005 and up‐to‐date, the ISO is updating the current ISO/IEC‐9126 international standard on software product quality measurement. However, this current standard will be superseded by the upcoming ISO/IEC‐25000 series of international standards on Software product Quality Requirements and Evaluation (SQuaRE). One of the objectives of this new standard series is the harmonization of its contents with the software measurement terminology of ISO/IEC‐15939 (software measurement process). ISO/IEC‐25000 series will replace the series of standards ISO/IEC‐9126 (software product quality) and also the ISO/IEC‐14598 (software product evaluation).
The work on ISO/IEC‐25000 series is unfinished at this time. It is being carried out by Working Group 6 (WG6) of the software and system engineering subcommittee (SC7) of the ISO/IEC Joint Technical Committee (JTC1) on Information Technology (ISO/IEC JTC1/SC71). SQuaRE consists of the following five divisions:
‐ ISO/IEC‐2500n: Quality Management Division. The standards form this division define all common models, terms and definition referred further by all other standards from SQuaRE series. ‐ ISO/IEC‐2501n: Quality Model Division. SQuaRE employs the same quality model proposed by ISO/IEC‐9126, dividing quality in characteristics for internal, external, and quality in use.
Quality in
use
This division details this quality model decomposing, the internal and external software quality characteristics into sub‐characteristics
‐ ISO/IEC‐2502n: Quality Measurement Division. Software product quality measurement reference model, mathematical definitions of quality measures, and practical guidance for their application.
‐ ISO/IEC‐2503n: Quality Requirements Division. This division helps to specify quality requirements. These quality requirements can be used in the process of quality requirements elicitation for a software product to be developed or as input for an evaluation process.
‐ ISO/IEC‐2504n: Quality Evaluation Division. Requirements, recommendations and guidelines for software product evaluation, whether performed by evaluators, acquirers or developers.
‐ ISO/IEC 25050 to 25099 are reserved to be used for SQuaRE extension International Standards, Technical Specifications, Publicly Available Specifications (PAS) and/or Technical Reports: ISO/IEC 25051 and ISO/IEC 25062 are already published.
2.1.3. Verification and Validation
Verification and Validation (V&V) ‐also known as Software Quality Control‐ is concerned with evaluating that software being developed meets its specification and delivers the functionality expected by the consumers. These checking processes start as soon as requirements become available and continue through all stages of the development process [54]. Verification is different to validation, although they are often confused. Barry Boehm expressed the difference between them [19]:‐ Verification: are we building the product right? The aim of verification is to check that the software meets its stated functional and non‐functional requirements (i.e. the specification).
‐ Validation: are we building the right product? The aim of validation is to ensure that the software meets consumer’s expectations. It is a more general process than verification, due to the fact that specifications not always reflect the real wishes or needs of consumers (i.e., users and customers).
V&V activities include a wide array of QA activities. Although software testing plays an extremely important role in V&V, other activities are also necessary. Within the V&V process, two big groups of techniques of system checking and analysis may be used [111]:
‐ Software testing. It is the most commonly performed activity within QA. Given a piece of code, software testing (or simply testing) consists of observing a sample of executions (test cases), and giving a verdict over them [16]. Hence testing is an execution‐based QA activity so a prerequisite is the existence of the implemented software units, components, or system to be tested. For that reason, it is sometimes called dynamic analysis. Software testing is a broad term encompassing a wide spectrum of different concepts, such as testing level (unit, integration, system, user testing, and so on), testing strategies (black‐ box, white‐box, grey‐box, and non‐functional testing), and testing processes (manual, model‐based, automated testing, and so on). On one hand testing establishes the existence of defects. On the other hand, debugging is concerned with locating and
correcting these defects [132]. As major parts on this dissertation, testing is covered in section 2.3 and automated testing in section 2.4.
‐ Static analysis. It is a form of V&V that does not require execution of the software. Static analysis work on a source representation of the software: either a model of the specification of design, or the source or the program [3]. Perhaps the most commonly used are inspection and review, where a specification, design or program is checked by a group of people. Additional static analysis techniques may be used, such as automated program analysis (the source code of a program is checked for patterns that are known to be potentially erroneous) and formal methods (mathematical arguments that a program conform its specification) [132].
Nowadays, the executable code per excellence is code (although there are some executable specification and design languages, there are not widespread). Thus, any product during development can be evaluated using static analysis, including of course code. However, testing (dynamic analysis) almost exclusively executes code.
It should be noted that there is a strong divergence of opinion about what types of testing constitute validation or verification. Some authors believe that all testing is verification and that validation is conduced when requirements are reviewed and approved. Other authors view unit and integration testing as verification and higher‐order testing (e.g. system or user testing) as validation [122].
To solve this divergence, V&V can be treated as a single topic rather than as two separate topics [1]. Therefore, V&V can be seen as a disciplined approach to assessing software products throughout the product life cycle. All in all, V&V activities can be summarized in the following picture:
Figure 8. Verification & Validation Schema
2.1.3.1. Defects
Key to the correctness aspect of V&V is the concept of software defect. The term defect generally refers to some problem with the software, either with its external or internal
V&V
Testing Levels Strategies Processes Static Analisys Inspection Review Automatic Analysis Formal Methods
behaviour. Software problems or defects are also commonly referred to as “bugs”. The IEEE Standard 610.12 defines the following terms related to defects [82]:
‐ Error: A human action that produces an incorrect result. Errors can be classified into two categories: i) Syntax error (program statement that violates one or more rules of the language in which it is written); ii) Logic error (incorrect data fields, out‐of‐range terms, or invalid combinations).
‐ Fault: An incorrect step, process, or data definition in a computer program. It is a condition that causes a system to fail in performing its required function.
‐ Failure: The inability of a system or component to perform its required functions within specified performance requirements.
In addition to this level of granularity for defects, it is interesting to contemplate also incidents as symptom associated with a failure that alerts the user to the occurrence of a failure. All in all, error, faults, failures, and incidents are different aspects of software defects. A causal relation exists among these four aspects of defects [138]. Errors may cause faults to be injected into the software, and faults may cause failures when the software is executed.
2.2. Static Analysis
Static analysis of a software piece is performed without executing the code. There are three advantages of software analysis over testing [132]:
1. During testing, errors can hide other errors. This situation does not happen with static analysis, because it is not concerned with interactions between errors.
2. Incomplete versions of a system can be statically analysed without additional cost. In testing, if a program is incomplete, test harnesses have to be developed.
3. Static analysis can consider broader quality attributes of a System Under Test (SUT) than searching defects, such as compliance with standards, portability, and maintainability.
2.2.1. Inspections
Inspections are critical examinations of software artefacts by human inspectors aimed at discovering and fixing faults in the software systems. All kinds of software artefacts for are subject to be inspected. This is primary reason for the existence of inspection: not waiting for the availability of executable programs (such as in testing) before starting performing inspection [138].
The original Fagan inspection process included five steps [41]: i) Planning: Deciding what to inspect, who should be involved, and what role. ii) Overview meeting: The author assigns the individual indications of inspection to the inspectors. iii) Preparation: Each inspector performs individual inspection. iv) Inspection meeting to collect and consolidate individual inspection results. v) Rework. The author fixes the identified problems or provides other responses. vi) Follow‐up: Closing the inspection process by final validation.
The Gilb inspection supposes a variation of Fagan inspection since an additional step (called “process brainstorming”) is added right after the inspection meeting [52]. This step is aimed at preventive actions and process improvement in the form of reduced defect injections for future development activities.
2.2.2. Review
Review is the process in which a group of people examine the software and its associated documentation, looking for potential problems and non‐conformance with standards, and other potential problems or omissions. The review team makes informed judgment about the level of quality of the system under review. This review process is based on documents produced during the software development process, such as specification, design, code, models, test plan, configuration management procedures, or user manuals [54].
A special form of review is called walkthrough, a more organized review typically applied to software design and code. It is considered to be an informal type of review. According to IEEE Standard for Software Reviews, a walkthrough is a form of software peer review “in which a
designer or programmer leads members of the development team and other interested parties through a software product, and the participants ask questions and make comments about possible errors, violation of development standards, and other problems” [69].
2.2.3. Automated Software Analysis
Automated Software Analysis (ASA) assesses the source code using patterns that are known to be potentially dangerous [54] ASA technologies are usually delivered as commercial or open source tools and services. These tools can locate many common programming faults, analysing the source code before it is tested and identifying potential problems in order to re‐code them before they manifest themselves as failures [83]. The intention of this analysis is to draw a code reader’s attention to faults in the program, such as:‐ Data faults. For example, variable used before initialization, variables declared but never used, variables assigned twice but never used between assignments, and so on.
‐ Control faults. For example, unreachable code or unconditional branches into loops. ‐ Input/output faults. For example, variables output twice with no intervening assignment. ‐ Interface faults. For example, parameter‐type mismatches, parameter under mismatches,
non‐usage of the results of functions, uncalled functions and procedures, etc.
‐ Storage management faults. For example, unassigned pointers, pointers arithmetic, or memory leaks.