5.3 Software management process
5.3.5 Software technical reviews description
5.3.5.1 Description
Technical reviews have been introduced in the ECSS-E-ST-40C standard, in order to account for the need of projects to handle reviews which may have a lower level of formalism, than the one foreseen by ECSS-M-ST-10-01C they are sometimes named “key points”. Their formalism is agreed between customer and supplier, as the technical reviews are joint reviews, the Customer is involved and has visibility.
A Technical Review includes the same tasks as a Project Review (ECSS-M-ST-10-01C 4.2):
a. initiation of the review (members and procedure)
b. distribution of the data package (availability time of the data pack can be reduced w.r.t the time requested for a project review)
c. review of the documentation d. review findings and conclusion
Flexibility with respect to ECSS-M-ST-10-01C (chapter 5) can be introduced for:
a. the independence b. the objectives c. the review bodies, d. the review meetings e. RID formalism
f. documentation DRDs (review procedure, RID, review team report, review authority report) For example, a technical review may not have a chairman. The minutes of the collocation meeting with the attached disposed comments may be used as a report.
SWRR/DDR (Software Requirement Review and Detailed Design Review)
The SWRR and the DDR are NOT Technical Reviews. They are anticipation of the PDR and the CDR in case the Technical Specification or the Detailed Design is baselined before the project review.
Therefore the formalism of the SWRR and the DDR is the one defined in ECSS-M-ST-10-01C.
SQSR (Software Qualification Status Review)
In case of reuse of existing software, the software reuse file can be reviewed in advance of the PDR, during a dedicated technical review. The reuse components/software and policy could have some impact of the software product design. This technical review enables to share with the customer the reuse approach sufficiently early in the development, and especially before the design activities.
The purpose of the review is to present the analysis that has been carried out on some already existing software components/software, in order to re-use them for the development of the current project/product. The analysis is based on assessment:
a. of all the information used to decide about the reuse (or not) of existing component/software, b. of the qualify level of existing component/software,
c. that the reused software meets the project requirements and the domain use, d. of the specific proposed actions, if they are required.
This review may be merged with SRR or SWRR.
TRR/TRB (Test Readiness Review and Test Review Board)
Two Technical Reviews are specifically mentioned in the software standard, the Test Readiness Review [TRR] and the Test Review Board [TRB]. They go together, one before a test execution (to ensure that the test specification and the test means are ready) and the other after the test activities.
Requirements 5.3.5.1a and 5.3.5.2.a are clear on the fact that these reviews need to be organized in relation to test activities, i.e. unit tests, integration tests and the various validations and acceptance testing. The tests activities subject to TRR/TRB are mentioned in the Software Development Plan.
The phrasing “as defined in the Software Development Plan” (actually duplicated in ECSS-Q-ST-80C 6.3.5.4) suggests that some testing activities might be considered as not subject to TRR/TRB. The common practice for unit testing or integration testing is indeed not to have specific TRR/TRB.
However, the TRR for Validation can be anticipated before the integration tests. It becomes a TRR for integration and validation where both Integration Test Plan and SVS are reviewed. Integration Test Report and Validation report are checked at the TRB for Validation (or at the CDR).
ECSS-Q-ST-80C requires (see requirement 6.1.5) that TRR/TRB are mandatory for any validation campaign, therefore before CDR, QR and AR. Common practice is to reflect it in a consistent way with ECSS-E-E40C (where requirement 5.3.3.3.a-b allows to define TRR/TRB in the SDP), according to the project needs. Typical examples are the following:
a. the TRB is very often handled together with/in the scope of formal reviews: TRB for validation together with the CDR, TRB for qualification together with the QR and TRB for acceptance together with the AR.
b. the TRR for qualification is sometimes handled together with/in the scope of the CDR c. the TRR for acceptance is sometimes handled together with/in the scope of QR.
d. for larger software development, several TRR/TRB may be handled (e.g. per function) for validation w.r.t. TS.
e. for smaller software developments, TRR/TRB may be suppressed.
Typical objectives of the TRR are introduced in annex P.2.1<6>a. Note 5.
It should be highlighted that, as the TRR is supposed to confirm the readiness of test activities, this implies that the relevant SVS (against TS or RB) is available at the TRR, as well as sufficient documentation to achieve the objectives of the review stated in the Review Plan, in particular documentation of the test environment.
TRR Deliverables are summarized in annex Q.4
Typical objectives of the TRB are introduced in ECSS-E-ST-40C annex P.2.1<6>a. Note 6.
TRB Deliverables are summarized in annex Q.5.
DRB (Delivery Review Board)
The DRB is an additional Technical Review not mentioned in the software standard, but actually necessary to authorize the delivery of a software version to an external team aside to the project
reviews (usually software versions are released at CDR, QR and possibly AR). This review aims at clarifying the state of the version (according to the previous agreement between the provider and the user of the version) intended to be delivered in terms of perimeter, validation, verification and configuration (whereas project reviews aim at acknowledging completeness of the activities demonstrating the fulfilment of the requirements).
When validation has been run on the version, the DRB can be merged with the Test Review Board [TRB].
DRB Deliverables depends on the agreement between provider and user, but includes at least the software release document (see 5.7.2).
5.3.5.2 Use of Technical Reviews to accommodate various life cycles 5.3.5.2.1 Introduction
Technical Reviews are a useful tool for the implementation of iterative lifecycles (and not only incremental as mentioned in the ECSS-E-ST-40C sub clause 5.3.3.3.c), such as incremental, evolutionary, spiral or agile (see clause 5.3.1). These life cycles spread the development in smaller waterfall-like steps. Running all the needed reviews as Project Reviews is potentially overloading the schedule and cost. Running some of them as Technical Reviews may allow achieving the objective while making it feasible within programmatic constraints.
There are many practices in projects. The whole review cycle may be applied for each version, or one review cycle may be spread over the versions, or the review cycle is applied only on the last version.
In addition, the objectives of the reviews are not seen the same way by all reviewers in all the projects.
However, each practice intends to save cost and schedule by tailoring the full formal process, while maintaining efficiency with respect to the review objectives and ensuring that the final quality is met.
5.3.5.2.2 A specific approach for flight software
In this section, the variety of practices cannot be presented exhaustively. Instead, some generic guidelines are given, followed by an example among others of a possible review plan.
The flight software development schedule is constrained by the overall spacecraft schedule. System requirements are not all available at the start of the software project development, and the AIT environment is in need of a software version before the full qualification of the software. Therefore, the flight software is often developed following an iterative approach (as defined in 5.3.1). Each iteration is usually called a "version". Indeed, ECSS-E-ST-40C clause 5.2.4.1 a. calls for the identification of software versions for software integration into the system.
As explained in 5.3.1, several constraints and factors influence the "versioning", i.e. the definition of versions and the associated life cycle. For example:
a. the maturity of system design, which indicates the priority in the development
b. the strategy and requirements in terms of integration and validation at higher level (AIT) In addition:
c. if a review is skipped, then the objectives of the skipped review are included in addition to the objectives of the next milestone.
d. configuration and quality activities outcome and assessments are considered at each review and deliveries.
Here is an example of versions definition and review setup in an evolutionary life cycle:
a. V1 is intended to allow the integration of all electrical units on the Electrical Functional Model (EFM). Therefore, V1 contains the basic Data Handling functions (typically a limited number of the functions). The architecture is defined, as well as some part of the detailed design. The technical budgets are still estimated. V1 must be tested enough such that it is usable on the hardware.
b. V2 is intended to allow the run of most close loop tests on the EFM using most of the AOCS involved equipment units. Therefore V2 contains the AOCS and Safe Mode (V2 is now typically a large number of the functions). The detailed design is more complete. Technical budgets are confirmed with high level of confidence. V2 is unit tested and validated on the stable functions.
c. V3 is intended to allow the satellite qualification. Therefore V3 contains the complete software.
The technical budgets are measured and proven. V3 is fully verified and validated with 100%
coverage of requirements.
d. V4 is the final flight software. V4 contains corrections of anomalies from system qualification.
V4 undergoes regression tests and representative mission simulation.
NOTE The software located in non-erasable memory (PROM) should be ready at equipment CDR. Therefore its lifecycle should be shorter. The boot software should have at least passed a CDR for V1 delivery and should have its QR before the SW V2 TRR, or before the starting of the qualification of the first flight model.
A possible review setup is the following (this example assumes a single TRR before CDR) Table 5-1: Possible review setup
V1 V2 V3 V4
Project Technical Project Technical Project Technical Project Technical
SRR X X X
PDR X X X
DDR X X X
TRR X X
TRB/DRB X X
CDR X
QR X
AR X
In an iterative/evolutionary life cycle, a way to reduce the overall review effort is to merge some reviews or to run some Project reviews as Technical reviews. The above scheme allows running half of the reviews as technical reviews. The rationale of their selection is:
a. Requirements Baselines are updated for the first three versions, they are important in the customer supplier relationship as they define the baseline for the validation and the system interaction. Therefore they deserve three formal SRRs (project review).
b. Technical Specifications are relatively complete for V2 (if V2 includes 80% of the functions). The Architecture is baselined for V2; no changes are expected for V3. Therefore formal PDRs apply to V1 and V2, V3 PDR is a technical review.
c. Detailed design is not mature in V1, and quite complete in V2, therefore a single formal DDR is done for V2, technical reviews only for V1 and V3.
d. TRR importance increases from V1 (no review) to V2 and V3 (technical review)
e. There is no CDR, but simply a TRB for V1 and V2, to review the results of the tests before delivery, but there is a single formal CDR (including the TRB objectives) and QR for V3.
f. The flight version V4 affords the formal acceptance.
NOTE All the documents are made consistent for the QR and AR. All the documents are available for QR (in the final DP) and updated for the AR.
Another possibility to inject flexibility in the review scheme, is to balance the review objectives according to the version. When the development is incremental, the objective of the reviews can also be incremental. For example, some objectives are reached in the scope of a particular version (denoted V in Table 5-2). Some objectives are fully reached for the complete software development, even if they are achieved at a preliminary version delivery (denoted F in Table 5-2). Some objectives, once achieved for the whole project at a preliminary version, do not need to be modified in terms of deliverable in the next versions (denoted NM in Table 5-2).
Table 5-2 is an example of review objectives and their applicability to each version. This table is an example only that is subject to review and adaptation for each project.
Table 5-2: Example of review objectives and their applicability to each version
SW-SRR Objectives V1 V2 V3
Agree with the customer or their representatives that all requirements captured in the requirements baseline are commonly understood and
agreed w.r.t. completeness and coherency. V V F
The software versions to be delivered are identified and each requirement of the requirements baseline is allocated either to HW or
SW, and for SW is associated to a version. V V F
Suitability of the software development plans including the software
planning elements with respect to the upper level planning F NM NM Suitability of the software product assurance activities to ensure the
development and verification will be performed taking into account the criticality of the software
F NM NM
Suitability of the configuration management environment to ensure
coherency and reproducibility of developed products F NM NM Adequacy and proof of verification performed by the developer V V F
SW-PDR Objectives V1 V2 V3
Agree with the customer or their representatives that all requirements of the requirements baseline (including interfaces requirements) are
captured in the technical specification V V F
Agree on the verification methods proposed to confirm conformance of
the product with the requirements and technical baseline V V F Verify the capacity of the design (including potential re-use) to
implement the technical requirements and their associated performance needs and constraints
V V F
Verify the assumptions taken for the budget and margins estimations in
order to ensure schedulability and performances achievements V V F
Decide on known unsolved issues way forward V V F
Confirm suitability of plans and development standards F NM NM Adequate verification performed by the developer including on potential
re-use V V F
SW-DDR Objectives V1 V2 V3
Confirm the stability of the technical baseline for the development V V F Confirm the detailed design is clear and correctly captures the technical
requirements and will allow an eased maintenance of the product thanks to the application of the design standards or if none at least to a homogeneous way of designing
V V F
Verify that the software units and integration plans implement an adequate strategy and coverage to fulfil the expected test level according
to the criticality V V F
Confirm the development environment is in place and ad hoc to start the
coding and testing F NM NM
SW-TRR Objectives for Validation vs. TS Testing V1 V2 V3
Confirm the Software behaviour through the Code analysis, the Unit Test and Integration Results and other verification performed for quality assurance with regard to the software criticality - justifications that can be agreed are provided when the results are not as expected*
V F
Confirm the foreseen validation/regression test cases (and possibly procedures), analysis, inspection or review of design will ensure the full coverage of the technical Specification applicable to the version
V F
Verify that software documentation, software code, procured software and support software and facilities are under proper configuration
control V F
Check the TS validation facilities readiness with regard to the tests
objectives V F
SW-TRB/SW-DRB Objectives V1 V2 V3*
Assess the validation/regression tests results with regards to the
objectives presented during the SW-TRR V V F*
Any observed deficiency is duly explained with proposed temporary
workaround and accepted by the receiver with regard to its need V V F*
Any late fix implemented to correct a blocking deficiency is duly explained, cross checked, validated and configured. Appropriate
regression has been performed F*
The complete definition of the product is clarified and limitations of use
are identified V V F*
Confirm the way to operate the software is correctly described and does
not have any unexpected side effect on the operations F*
* : covered by the CDR
SW-CDR Objectives V1 V2 V3
Confirm that the implemented product is fully coherent with its
definition F
Confirm that all the agreed verification and tests have been successfully performed and duly reported in order to ensure the required quality level and the technical requirements coverage
F
Assess and agree on the deficiencies (RFW) or known unresolved issues (SPR-NCR) which can have an impact on following phases and identify a resolution plan
F
Confirm the way to operate the software is correctly described and does
not have any unexpected side effect on the operations F
Confirm the foreseen validation/regression test cases (and possibly procedures), analysis, inspection or review of design will ensure the full coverage of the User Specification
F
Review all the configuration item to baseline the technical configuration
w.r.t. TS-TRR configuration and delta justification F
SW-QR Objectives V1 V2 V3
Confirm the applicable baseline is correct including RFDs F
Verify through the RB-validation tests, analysis, inspection or review of design results that the software meets all its user requirements, and that verification and validation process outputs enable transition to “qualified state” for the software products
F
Verify that the complete set of test is run on the same software version otherwise justified;
F
Confirm the correct closure of major SPRs/NCRs and the acceptability of RFWs
F
Review and baseline the item configuration w.r.t. RB-TRR configuration and delta justification;
F
Assess and agree on known unresolved issues (SPR-NCR) which can have an impact on following phases and identify a resolution plan
F
Review of the maintenance plan F
SW-AR Objectives V4 Identical to the SW-QR objectives after application of the resolution plan and correction/validation of any other NCR raised on the software during satellite test campaign and successful run of the mission representative test.
F
There is a link between version control and life cycle:
a. the Change Control can start at the first iteration in order to trace the changes of requirements from version to version. The reason is that the content of the reviews is impacted. For example, if a feature at V2 DDR is moved to V3, its coverage in the reviews needs to be clear. As another example, if requirements of V1 are changed in the middle of V1 development while preparing V2, it should be traced. It is highlighted that if a requirement which was fulfilled by V1 is changed after the delivery of V1, there is generally no need – in the frame of an iterative life cycle - to go back and change V1 - it is sufficient to fulfil the new/modified requirement in the final flight software.
b. the Customer should follow the Change Control Boards that will be run to baseline the various items of the versions, in particular the requirements. Once a baseline is approved for a version, the supplier is not allowed to change it outside of a Change Control Board to which the customer is invited, because any change impacts testing, documentation and schedule.
However, in the early phase, the process of Change Control Board might be lightened, provided that the Customer is made aware of the changes.