Verification and validation

The ASA management standard, AEO Authorisation Requirements, states the following mandatory requirement for verification and validation:

"An Authorised Engineering Organisation shall have verification and validation arrangements in place, relevant to the engineering services or products provided"

Verification and validation processes are designed to ensure that process outputs are correct.

The development of any system is not complete without rigorous verification that the implementation is consistent with the specifications.

The verification and validation processes and outputs will form part of the evidence supplied in product and project safety cases that lead to system acceptance and certifications.

The purpose of the verification process is to confirm that the specified design requirements have been fulfilled by the realised system. This verification process provides the information required to effect remedial actions that correct non-conformances in the realised system or the processes that act on it.

The purpose of the validation process is to provide objective evidence that the services provided by a system, when in use, comply with the stakeholders' requirements and achieve their intended use in their intended operational environment. The validation process performs a comparative assessment and confirms that the stakeholders' requirements are correctly defined. Where variances are identified, they are recorded and used to guide corrective actions.

System validation is ratified by the stakeholders.

8.7.1 Verification guidelines

System verification determines whether the system, its elements, and its interfaces satisfy their respective requirements. Verification ensures conformance to those requirements. Verification provides assurance that "you designed it correctly, and you built it correctly". Verification encompasses tasks, actions, and activities performed to evaluate the progress and effectiveness of the evolving systems solutions, and to measure compliance against requirements.

The primary objective of verification is to determine whether the systems specifications, designs, processes, and products are compliant with the requirements. The continuous feedback of verification data helps to reduce risk and forces problems to be identified at the earliest stages. The goal is to completely verify a system's capability to meet all of the requirements prior to the production build and operation stages.

The basic forms of verification activities include:

 design reviews

 inspection and test completion reviews

 inspection

 analysis

 demonstration

 test

 certification

8.7.2 AEO design review guidelines

TfNSW requires that an AEO declares and follows approved design review processes and procedures to provide specification-compliant design solutions. The designs should address all relevant in-scope requirements of the SRS.

The ultimate responsibility for the safety, technical accuracy and correctness of the design resides with the AEO that produced the design.

The outcome of any TfNSW involvement in an AEO design review activity only indicates agreement for the AEO to proceed with the design submission. It does not represent approval of the design by TfNSW.

AEOs are required to submit their design review processes and procedures for assessment by the ASA, as part of the authorisation process.

AEO design review processes and procedures are expected to align with the TfNSW systems life cycle model, the simplified model of TfNSW engineering design management process, and also the 'technical processes' requirements specified in ISO/IEC 15288-2008.

8.7.3 AEO inspection and test review guidelines

TfNSW will contract certain AEOs to build, integrate and test system designs.

TfNSW requires that AEOs declare and follow approved inspection and test completion review processes and procedures. This requirement is to satisfy TfNSW that the work an AEO performs under those inspection and test processes, addresses all relevant verification requirements of the SRS, and provides specification-compliant solutions.

Ultimate responsibility for the accuracy of technical aspects and integrity of the work performed resides with the AEO that constructs and integrates the design solution.

The outcome of any TfNSW involvement in an AEO inspection and test completion review activity only indicates agreement for the AEO to proceed with the submission. It does not represent TfNSW's approval of the technical aspects of any works.

AEOs are required to submit their integration and test completion review processes and procedures to the ASA for assessment and approval. AEO integration and test completion review processes and procedures are expected to align with the systems life cycle model, the simplified model of TfNSW engineering design management process, and also the 'Technical Processes' requirements specified in ISO/IEC 15288:2008 Systems and software engineering - System life cycle processes.

8.7.4 Validation

Validation is the assurance that the final installed system meets the stakeholder requirements that were captured in the principal requirements and user requirements documents - "are we building the correct thing?"

Validation determines whether a system does everything it should, and that it does not do what it should not, for all defined modes of system operations. End-users and other stakeholders are usually involved in validation activities. Validation occurs as part of the system testing processes.

The project design and construction can also be validated at earlier stages of the TfNSW asset life cycle. However, this requires the use of simulation, review and site checks to obtain stakeholder acceptance at the appropriate stage, and is only performed on risk and high-novelty projects.

The validation of each principal requirement is required to be mapped and to be traceable back to the user, principal, or system requirements.

Validation may take place in the operational environment under possession control, or, in a simulated operational environment if real-world conditions are too hazardous for validation observers.

Independent third parties may be called in to perform validation.

The allowable forms of validation activities include the following:

 demonstration

 inspection

 test

 analysis

 measurement

 simulation

 customer survey

Further guidance is provided in TfNSW AEO Guide to Verification and Validation.