5.8 Software verification process
5.8.3 Verification activities
5.8.3 Verification activities
5.8.3.1 Verification of requirements baseline
Unclear responsibilities assignment between system and software teams may result into software failures. In particular the interpretation of the “system needs” by the software teams is the cornerstone of the system and software processes.
A key principle of the ECSS-E-ST-40C to achieve as far as possible the common understanding of the software requirements, is to build the requirements in two steps (RB for needs then TS for solution)
and to perform a cross-check through two separate requirements documents (i.e. RB and TS) and reviews (i.e. SRR to check the needs completeness, and the PDR (potentially anticipated by the SWRR) to allow the Customer checking that the proposed software solution answers the needs and the software supplier checking that his understanding of the customer needs are correct, complete and results into a feasible software solution.
This handbook also recommends an early integration of the software engineering process in the system engineering phase (co-engineering) to facilitate the common understanding of the requirements: the supplier can early understand the software needs and checks their feasibility, before the SRR.
In the situation where the SW supplier is not yet selected when the SRR is conducted, the SVerP may be only available at the PDR, consequently after the SRR. This would prevent co-engineering process to be set up and would therefore delay the verification activities to be performed at System-Software engineering level.
In order to verify the completeness of the Requirements Baseline (RB), the following list provides a set of topics which could strongly affect the software during its development process or which are shared by different elements (e.g. interface, failure mechanism, and timing performance). These topics need to be checked carefully at system level (for completeness) and at software level (for feasibility):
a. the RB describes clearly the environment in which the software will operate. The description focuses on the operational context and the interfaces with external systems,
b. the RB specifies the characteristics of all external systems (e.g. bus, computer, ground interface) interacting with the software product. The above-mentioned characteristics include, but are not limited to, the communication protocol, the concurrency and real-time model,
c. the RB specifies the control and observation points for each function,
d. the RB specifies the fault detection, isolation, and recovery , in relation with the dependability and safety analysis outputs,
e. the RB specifies the modes/sub modes and transition between modes (modes automaton). For each mode/sub mode and transition, the associated activities (functional description) are specified,
f. the RB specifies how telemetries / packets TM are dated and the precision required,
g. the RB identifies the requested configurable data of the software, that are usually defined through a software database,
h. the RB identifies and justifies the margins policy in terms of memory and CPU allocation, i. the RB defines the operational scenarios (e.g. traffic scenarios),
j. the RB specifies the list of external events that are produced or used for execution by the software product, as well as their occurrence model (e.g. periodic, spontaneous),
k. the RB specifies, for each application function or hardware component, the type of failure (no failure, accidental failure on value or time, Byzantine or not, intentional or not [security]) and their occurrence model (e.g. periodic, a periodic),
l. the RB specifies the timeliness properties (i.e. constraints on time to start or terminate application functions), periodicity, and jitter when they are mandatory with their associated justification (e.g. AOCS commanding loop between sensors acquisitions and actuations),
m. the upper level project development plan (system, subsystem, equipment, instrument, platform development plan) specifies the schedule allocation for the software development, including margins consistent with respect to the development effort,
n. the upper level project development plan defines all inputs necessary for the development activities, such as the test facilities, input documentation, CFI (Customer Furnished Items ), environmental models if necessary and their availability,
o. the upper level project development plan identifies all needs in term of software product deliveries and their dates.
5.8.3.2 Verification of the technical specification
The “design feasibility” verification, required by the ECSS-E-ST-40C requirement 5.8.3.2, may be anticipated by activities such as:
a. when the requirements engineering is based on a model checking approach (Annex B), some properties of the model may be verified at the earliest stages of the life cycle, and from which the design model may be derived afterwards,
b. a strong system software co-engineering primarily established to consolidate the needs, also allows an early consolidation of the design feasibility when supported by fine SW experts analysis or prototyping,
c. the need for an evolutionary robust design approved by a peer review in particular if an iterative life cycle is selected ( See the section 5.3.1on life cycles that recalls the risks of each life cycle models),
d. the static and dynamic design are well defined and supported by a monitoring of the resources budget,
e. the reused generic architecture or building blocks are able to support the specific requirements.
5.8.3.3 Verification of the software architectural design
The ECSS-E-ST-40C requirement 5.8.3.3 requires developing "feasible operations and maintenance ".
In fact these two notions can be anticipated individually.
The operation feasibility can be covered through prior verification activities as:
a. the environment (the external interfaces definition completeness) in which the software operates is clearly described in the RB and correctly refines in the TS,
b. the controllability and observability definition of each function describes respectively the telecommand services and telemetry services used,
c. the fault detection, isolation, and recovery strategy is coherent, d. the operational scenarios are defined in the RB as use cases.
The future maintenance can be assured through prior verification activities on the following points:
a. up to date complete QR/AR data package,
b. a static and dynamic design sufficiently documented,
c. the Software Development Environment and Software Validation Facility are, as much as possible, decoupled from potential complex infrastructures existing in the companies,
d. availability under adequate configuration management control of all non-deliverables contributing to the software product e.g. models, source code, makefile, build procedures, test scripts,
e. the non-obsolescence of development, validation facility and production chains (potential obsolescence to be mitigated by potential migration),
f. training of skilled people e.g. involvement of maintenance team before the end of the software development activities,
g. the completeness of the maintenance plan
5.8.3.4 Verification of software detailed design
-
5.8.3.5 Verification of code
The purpose of code coverage measurement is to measure the confidence one may have in a software product and to assess the risk that potential faults remain hidden in part of the software which has never been run.
ECSS-Q-ST-80C 6.3.5.2 and ECSS-E-ST-40C 5.8.3.5.c requires code coverage to be measured.
For Category A and B there is an requirement [ECSS-E-ST-40C 5.8.3.5.b] for 100 % code coverage (100% MCDC only for Cat A). For Category C and D, the standard requires that the value is agreed with the customer.
ECSS-Q-ST-80C requires a quality model. One of the metrics of the software quality model can be the targeted number for code coverage for category C and D which can be any number according to the project needs and risk analysis.
The effort of performing code coverage and reaching the targeted number has to be considered in a context of risk analysis and reuse process: The risk is evaluated according to (i) the probability of a fault in the software and (ii) consequences in the system (i.e. criticality). The probability of a fault in code that has not been tested is the same for flight software and for ground software. The consequences are the same if they have the same criticality level. The risk is therefore the same for flight and for ground software. It is sometimes considered more acceptable to take higher risk on ground software than on flight software, because recovery actions are perceived easier on ground systems.
The complexity of a module may directly affect the risk of remaining faults. Therefore, complex units may deserve a higher code coverage target supported by proper unit testing.
In addition, traditional ground software usually relies on significant reuse of (reaching sometimes more than 90 %):
1. COTS and operating system for which the source code is usually not available,
2. reuse of open source or proprietary software for which the confidence has been proven, 3. firmware of hardware components such as firewall, routers.
For this part of the software, code coverage either cannot be conducted or requires a substantial effort.
In addition, the amount of unused code which is not relevant, but simply deployed as part of the reused components, can be significantly high.
More generally, the suitability of reusing the software in the project has to be evaluated in accordance with ECSS-Q-ST-80C 6.2.7.
5.8.3.6 Verification of software unit testing (plan and results)
-
5.8.3.7 Verification of software integration
-
5.8.3.8 Verification of software validation with respect to the technical specification and the requirements baseline
-
5.8.3.9 Evaluation of validation: complementary system level validation
-
5.8.3.10 Verification of software documentation
-
5.8.3.11 Schedulability analysis for real-time software
Schedulability analysis is discussed in 7.5.
5.8.3.12 Technical budget management
See also 7.3.
5.8.3.13 Behaviour modelling verification
NOTE Model based engineering is discussed in 6.3.
The ECSS-E-ST-40C requirement 5.8.3.13 requires the verification of the behavioural model. The following topics should be verified to anticipate the correct behaviour of the final SW product, during different steps of the life cycle:
a. the software dynamic aspects are well defined at design level: each function is properly described (e.g. with finite state machines or Statecharts), the way the different services are used (e.g. frequencies, task context, call order / dependencies, external activations by interrupts…).
Sequence diagrams / time lines is an efficiency way to show the dynamic behaviour of the services and the interactions between them;
b. the proposed design provides SW behavioural observability and debug features (e.g. time stamping of tasks starting and ending, semaphores changing);
c. the software design is safe and robust, in particular with regard to the external error propagation (e.g. defensive design, implementation analysis, test coverage).