In this section we briefly discuss why software safety is important, what makes it different from more general system safety and what the UK MOD requires in terms of software safety. This discussion summarises the Technical Guidance of Section 3, but emphasises managerial concerns over technical details. Where more detail is required, the reader is referred to the relevant sections of the Technical Guidance.
The applicable UK MOD standard for system safety is Defence Standard 00-56 Issue 4. Herein, a system which is safe is defined to be one for which, in a given application and a given operating environment:
1. The associated risks have been demonstrated to be reduced to a level which is ALARP and broadly acceptable or tolerable
2. The relevant prescriptive safety requirements have been met
The safety properties of interest for software will be broadly dependent upon the potential for the software to contribute to system hazards. Clearly more confidence will be required in software which is safety-critical than in software which is neither safety-critical nor safety-related. Historically this has been achieved through the definition of “levels” of safety (for example, the Development Assurance Levels of DO-178B or the Safety Integrity Levels of IEC 61508). To achieve a given safety level, certain processes throughout the software development had to be completed – for example, particular types of testing, provision of a software development plan, demonstration of sufficiency of software requirements. These safety levels are then in some cases mapped to failure rates, leading to the conclusion that following the given development processes will result in producing software which satisfies a certain target failure rate.
These are two major problems with this approach. Firstly, there is no evidence that a good process will result in a good product (although there is a correlation between bad processes and bad products!). That is, there is no indication that completion of a given set of lifecycle activities really will produce a system with a target failure rate as claimed. Secondly, while the notion of “failure rate” has some credence in terms of hardware, software failures are systematic rather than random in nature. That is, given particular inputs, the software will react in the same way every time. The failure rate of the software is then determined solely by the rate at which the software will be presented with input which causes it to fail. This rate is hard to predict, requiring significant knowledge of residual faults.
Instead, Def Stan 00-56 introduces an evidential approach to software. Briefly, this requires that any claim about the safety of the software must be supported with rigorous evidence, and with a compelling safety argument demonstrating how this evidence supports the claim. The extent and rigour of the evidence supporting the claim must be commensurate with the risk. That is, the safety
requirements for software which can contribute to a catastrophic hazard must be supported with more rigorous evidence than the safety requirements for software which can contribute only to minor hazards.
A.1.1 Problems Associated with Software Safety
There are some factors involved in software development projects which affect safety, making the delivery and management of an acceptably safe item of software a complex proposition. Firstly, unlike hardware, software is “pure design” and there is no manufacture. When developing a tangible product, delays and budget overruns in the design can in some cases be compensated for during manufacture. When developing software, by contrast, there is no phase during which design delays can be absorbed. That is, any change to the four major properties (scope, quality, time, resources) in the software design is likely to impact on the project.
Additionally, safety can be hard to manage in software development. There is often little management visibility into the technical issues which develop – errors in code are often inaccessible to management personnel and the quality of the software is hard to judge. Consequently, it can be difficult to determine whether safety is being managed correctly. One solution to this problem is to make use of domain experts or ISAs where possible, in order to obtain an independent assessment of the progression of the development.
Safety management throughout the development cycle can also be an important and somewhat problematic issue. At the time of establishing contracts for a software development project, detailed information about the safety requirements for that software may not be known. In many cases, these will only emerge later, as further work on the project provides more information about the properties which must be present in the software for it to be deemed acceptably safe. Consequently, safety management is an important issue throughout the entire project, as emerging requirements can affect any of the four considerations of scope, quality, time and resources.
A.1.2 Aspects of Software Safety
The overall aims of safety management are to ensure that safety requirements are met with sufficient confidence (assurance). There are three major factors in obtaining the required assurance, as described below. These can broadly be considered as providing the following benefits:
1. Capability Assessments – these aid in selecting a supplier who is likely to satisfy the acceptance criteria and wider managerial concerns.
2. Visibility into Safety Processes – this provides process evidence, as well as increased confidence in the ensurance aspects of software development. 3. Safety Argument – a safety argument should be supported by rigorous
evidence and should be sufficiently compelling to justify confidence in the safety of a system.
Capability Assessment
Capability assessments (such as Capability Maturity Model Integration appraisals) can provide a certain level of confidence in the capabilities and processes of the supplier. In particular, they can be used to distinguish between suppliers in terms of the maturity of their processes and their conformance with “best practice” (note that there is no guarantee that CMMI level 5 is equivalent to safety-critical software best practice!).
Capability assessments, however, cannot be used as a significant source of support for safety arguments. The reason for this is that these assessments do not provide explicit evidence that any safety properties of the system in question (e.g. “Function X satisfies a target failure rate of 10^-3”) are or will be satisfied. In addition, capability assessments rarely provide much reliable information about the safety of software being developed (that is, they do not contribute significantly in terms of knowledge of ensurance). Although some aspects of development processes are repeatable, they cannot be relied upon to guarantee specific properties of the end product.
However, capability assessments still have value. Like SILs, they can be used as tools to address managerial concerns such as selecting a supplier, estimating a budget, setting acceptance criteria and so on. In particular, capability assessments can act as a “forecasting” tool when selecting a supplier. In this capacity they can predict whether a compelling safety case is likely to be produced, whether sufficient visibility into the development process is likely to be provided, whether the supplier shows an understanding of UK safety management and policy, whether functional requirements are likely to be met and whether budget and quality concerns will be satisfied. It is important to understand that a satisfactory capability assessment does not guarantee that a system which is demonstrably acceptably safe will be developed, but in some cases it may indicate that this is more likely than in others.
In addition to this, capability assessments may be used to judge whether problems are “containable” as shown in the swim-lane diagram of Section 2. It is not unknown for there to be little or no visibility into the safety and development processes of software (for example, this situation may arise when purchasing COTS software from the US). In this case the Visibility of Safety Processes – described below – is lacking, even though a safety argument and associated evidence may be provided. In this case, a capability assessment can go some way towards judging whether the vendor is sufficiently trustworthy to accept their claims. Additionally, it may provide confidence that there are no major faults, such as a lack of traceability or satisfaction of requirements. Other existing policies and procedures may also be used for this, where necessary being supplemented by the capability assessments.
Visibility of Safety Processes
Sufficient visibility into the safety processes can increase confidence in the development and safety management on a project. Specifically, visibility into the safety processes can increase confidence in the competence of the personnel, as well as provide process evidence to help support a safety argument. However, visibility into safety processes does not itself constitute product evidence. That is, while visibility of safety processes can provide confidence that safety is being managed acceptably, it does not necessarily constitute sufficient evidence to support a safety argument. However, the process evidence obtained from this visibility can increase confidence in the integrity of the evidence chain, thereby supporting the trustworthiness of any product evidence which is generated.
Safety Argument
Def Stan 00-56 requires a compelling safety argument supported by rigorous evidence to be provided in order to show software is acceptably safe (56-1 9). In general, a software safety argument should be supported by both product and process evidence (respectively, evidence relating directly to safety properties of the software, and evidence relating to the sufficiency of the development processes). Sufficient visibility of safety management processes can provide both the required process evidence, as well as confidence in the trustworthiness of the product evidence. Where this visibility is not provided, capability assessments may be used instead to increase confidence in the trustworthiness of product evidence, and in the suitability of development processes.
A.1.3 Principles of Safety Arguments
The evidential approach requires that specific safety requirements be identified and supported with rigorous evidence. The purpose of the safety argument is then to convince the reader that the requirement is satisfied, by relating the evidence to the requirement. For example, a safety requirement stating that “Safety-related function X is correctly implemented” may be supported by evidence consisting of test results demonstrating the execution of Function X. In this case the safety argument would then be designed to convince the reader that these test results really do demonstrate that Function X is correctly implemented.
The software safety requirements are – in the first case – obtained from system safety requirements. That is, in order for the entire system to function safely, this software component must meet certain safety requirements. Safety requirements obtained in this way are often very high-level and abstract, requiring that the software should not contribute unacceptably to system hazards. In order to obtain these high-level software safety requirements, it is therefore necessary to examine the system and software boundaries, and the system consequences of software failure. High-level software safety requirements are refined to obtain derived safety requirements, for example that the software should not fail in a given way under given conditions, where this failure can contribute to system
hazards. The task of obtaining these derived safety requirements can only be undertaken once a certain level of knowledge of the software and its operating conditions has been obtained.
The Technical Guidance of Section 3 contains more detailed analyses of safety arguments. Nevertheless, as management personnel may be responsible for assessing the suitability of a completed safety argument, a briefer summary of important considerations is provided here.
A.1.3.1 Safety Argument Terminology
Safety arguments will be defined in more detail in the Technical Guidance, but in this section we will briefly summarise their main characteristics.
Within a software safety case, a claim is a statement made about the software which may or may not be true. For example ‘All omission failures are detected and handled acceptably’ is a claim about a system.
A software safety argument is a connected sequence of statements intended to establish a claim. For example, ‘All omission failures are detected and handled acceptably, because Components A and B are present in the system, and tests show that they detect and handle all possible omission failures’ is (part of) an argument.
Typically the top-level claims in a software safety argument are high-level software safety requirements obtained from system safety requirements. The safety argument is intended to demonstrate the truth of these by breaking them down into derived software safety requirements.
There are four high-level objectives (the last of which contains three facets which are more fully explained in the Technical Guidance) which are required in a software safety argument:
Requirements validity – the argument must demonstrate that all software safety
requirements are complete and accurate for the purposes of mitigating the software contribution to system-level hazards.
Requirements satisfaction – the argument must comprehensively demonstrate
satisfaction of all the identified software safety requirements.
Requirements traceability – the argument must demonstrate that the high-level
software safety requirements are traceable to system hazards, and also down through all levels of development (derived software safety requirements, design, code, evidence etc.)
Software quality – the argument must demonstrate that the software (and the
development processes) exhibits the basic qualities necessary to place trust in the evidence presented.
A.1.3.2 Assurance of Safety Arguments
Safety arguments are typically inductive, meaning that the supporting claims do not imply the higher-level claim with absolute certainty. That is, the premises of the argument support the conclusion, but do not entail it. The intent of a safety argument is to provide sufficient confidence in the truth of the top level safety claim. To do this, the argument may make deductive, inductive and judgemental conclusions (56-2 9.5.5). Because software safety arguments are rarely completely deductive, it is generally impossible to achieve 100% confidence in the truth of a safety claim.
The assurance of a claim is the justifiable confidence we have in the truth of that claim. The assurance of a claim is determined by whether the safety argument is sufficiently compelling, and whether it is supported with sufficiently rigorous evidence. However, determining what constitutes “sufficient confidence” in a claim can be both subtle and complex. The confidence which is required in a claim should reflect the contribution of that claim to the safety of the system. For example, more confidence is required in a piece of software which could cause a catastrophic hazard on failure than a piece of software which could cause only a minor hazard.
Determining the confidence needed in a particular safety requirement – and how to obtain that confidence – can require significant domain and technical knowledge. Further details are available in the Technical Guidance, but from a managerial perspective, it is sufficient to know that the assurance required in the safety of software should be commensurate with the potential risk posed by that software.
An assurance deficit associated with a claim is a lack of information which affects the assurance of that claim. When constructing a safety argument, the intent is to justify all assurance deficits. This requires assessing the overall effect of this assurance deficit in terms of its contribution to system risk. A form of gap analysis can be performed to determine whether the assurance deficit is intolerable, tolerable, or broadly acceptable. If an assurance deficit is categorised as tolerable, its presence must be justified by demonstrating that the cost of addressing this deficit is grossly disproportionate to the benefit gained. Section 3 and Annex B contain further details on this.
A.1.3.3 Determining Assurance of Safety Claims
Broadly speaking, the assurance of a higher-level software safety requirement is dependent upon two major factors. The first of these is the assurance of the derived software safety requirements (DSSRs) which support this. That is, if we
have little confidence in the premise of an argument, we will have little confidence in its conclusion. The second factor is the extent to which these supporting DSSRs entail the entirety of the higher-level requirement (sometimes referred to as the “inductive gap”). That is, the extent to which the premises of the argument entail the conclusion. Consequently, if there is an assurance deficit it must be due to one of two factors:
1. The supporting claims are themselves not adequately assured
2. The supporting claims do not provide sufficient information about the truth of the higher-level claims
This means that to reduce an assurance deficit at the top level we must either increase the assurance of one or more supporting claims, or change the argument structure so that the inductive gap is narrowed.
When considering the first approach, however, it is important to note that not all supporting claims are equally important to the assurance of the higher-level claim. Consequently, increasing the assurance of one supporting claim may have a larger effect than increasing the assurance of a different supporting claim. From a project management perspective, it is important to ascertain the most cost- effective way of reducing the assurance deficit, whether this be choosing the best supporting claim to assure, or altering supporting claims to narrow the inductive gap. In terms of the four “project parameters” (scope, quality, resources and timing), this amounts to addressing the deficit and minimising the impact on time and resources.
In general terms, the contribution of a supporting claim to the higher-level claim can be assessed by considering the following questions:
- How much confidence do we have in the truth of this supporting claim? - How much information relevant to the truth of the higher-level claim is
provided by this supporting claim? - How important is this information?
- How “justified” is our confidence in the assurance of this supporting claim? o Is this information also supplied by a different supporting claim? o Is the information vulnerable to any common-cause failures?
We briefly discuss some of these questions below; further detail can be found in the Technical Guidance.
Scope
The scope of a supporting claim is defined as the extent of the information about the higher-level claim which is provided by that supporting claim.
The scope of a supporting claim addresses the question “what information does this supporting claim provide about the higher-level claim?” The scope is important because it tells us what information will be missing if we do not provide a particular supporting claim. Scope can also be useful for what it tells us about the inductive gap. By considering the scope of each supporting claim in turn, we
can deduce if there is some information about the higher-level claim which is not provided by any of the supporting claims. The more information which is “missing” in this way, the greater the inductive gap will be.
However, scope by itself only tells us what information is provided by the supporting claim. In order to estimate the importance of this supporting claim to the higher-level claims, we must also consider some other questions.
User-defined Preference
The user-defined preference of a supporting claim is defined as the “importance” placed by the reader on the information about the higher-level claim which is provided by that supporting claim.