OOI Verification and Testing Process 77

The OOI shall perform thorough and appropriate testing throughout the Construction and Operations and Maintenance phases of the program, to verify that each end system, subsystem, and component defined and implemented by each IO functions properly and conforms to the requirements of the IO and OOI logical and physical design solution. Error! Reference source

not found. shows an overview of the types of verification and testing that will occur on the

program, and indicates when each type of verification or testing occurs in relationship to the system development life cycle.

Throughout all verification and testing activities, the IOs and OL shall use OOI SDE Configuration Management tools and OOI SDE Collaboration tools to monitor and record verification and test results, including: variances identified (untraceable requirements and constraints, anomalies, variations, voids, and conflicts), corrective actions taken; lessons learned; outcomes achieved; tradeoff, effectiveness, and risk analyses completed with resulting key decisions; test activities completed; and the verified design solution.

If the cause of an identified variance was failure to properly complete the fully characterized design, the design and development is re-worked and the verification and testing procedures are then re-run against the modified product. If the cause of an identified variance was an

inadequate test plan or test environment, the test plan or test environment is corrected and the verification and testing procedures are re-run.

Design Integration Deployment Operations

Developmental Testing Acceptanc e By OL Integration Testing Requirements Verification Environmental Testing

Unit Testing / First Article Testing

Acceptance Testing Regression Testing Commissioning Evaluation Commissionin g Stability Testing Development

Figure 16, Verification and Testing Activities Throughout the System Development Life Cycle

6.3.1.1 Developmental Testing

Developmental Testing is conducted during the Design and Development stages of the system development life cycle. The purpose of this type of testing is to investigate and prove feasibility of design options, to develop and test prototype solutions, to demonstrate proof of concept, and to manage and reduce design and implementation risks. Developmental testing is conducted at the IO level. In the Cyberinfrastructure IO spiral development methodology, Developmental testing occurs primarily during the Inception Phase of the spiral development model.

Results of Developmental testing are documented and used to inform the ongoing design and development of the associated product.

6.3.1.2 Unit Testing / First Article Testing

Unit Testing / First Article Testing is conducted during the Development phase of the system development life cycle. It is the first level of functional testing performed on the lowest level production units which are produced or procured by the IOs. Unit Testing / First Article Testing confirms that there are no remaining design changes needed prior to production and deployment of the unit or second and subsequent articles. Successful unit testing of all component units must be completed before integrating the units into the next level of assembly for integration testing.

Unit Testing / First Article Testing is conducted at the IO level. In the Cyberinfrastructure IO spiral development methodology, unit testing occurs during the Elaboration and Construction Phases of the spiral development model.

During the formal test itself, all items except the Unit Under Test (UUT) are considered test equipment and must be carefully managed to avoid errors in interpretation of test results.

6.3.1.3 Environmental Testing

Environmental Testing is an optional type of testing conducted during the Design, Development and Integration phases of the system development life cycle. It is important to obtain in situ deployment and operating experience as early in the project as possible, especially for the critical elements of the system; thus Environmental Testing is sometimes necessary. In Environmental Testing, hardware components tested previously in Unit or Integration testing are tested again in a “wet environment,” with pressure, temperature, and other environmental factors as similar to those at the deployment site as practicable. The achieved performance of the product being tested is measured against the required performance.

In most cases water tanks and near shore testing are used for environmental testing of hardware components, and while simulating the deployment site environment, environmental testing is still considered testing in a non-deployed test. Additional refinements indicated by the environmental testing effort are incorporated into the product design during this cycle.

Environmental testing is conducted at the IO level, and each IO is responsible for the documentation, maintenance, performance, and monitoring of its environmental tests. The Initial Operating Capability (IOC) CI spiral development milestone is a gate to the

Environmental Testing described above. The IOC occurs at the end of the Construction phase of the spiral development model (see 4.1.1.3), and transitions the construction activities into the deployment activities. Success criteria include the requirement that a functioning system be delivered that implements the functional specification and external integration criteria from the LCA, documentation of unit tests for all system components and system tests for functionality,

and a plan for deployment metrics and integration points. The IOC review/test is performed internally to the CI IO with participation from the Program Office and other IOs as needed.

6.3.1.4 Integration Testing

Integration Testing is conducted during the Integration and Deployment phases of the system development life cycle. Integration is the incremental and iterative process of combining lower- level parts or components into higher-level assemblies. Testing of assembly functionality, performance and interfaces is conducted at each level of integration. Some levels of integration can only be tested fully after actual deployment of the integrated assembly.

Integration testing is conducted by individual IOs, and at higher levels of OOI integration, is conducted jointly by two or more IOs who are contributing components to the integrated assembly being tested. Integration test plans and procedures are documented and integration results are recorded. Interface requirements, interface diagrams, and interface control documents are used to inform the Integration test plans and to evaluate Integration testing results.

In the Cyberinfrastructure IO spiral development methodology, integration testing occurs during the Construction Phase of the spiral development model.

6.3.1.5 Stability Testing

The purpose of stability testing is to show that the product being tested can perform as intended over an extended period of time with no significant degradation in performance, functionality, or usability. Hardware burn-in testing and software memory growth testing are examples of stability tests.

Stability testing is conducted by the IOs during the Integration and Deployment phases of the system development life cycle. In the Cyberinfrastructure IO spiral development methodology, stability testing occurs during the Construction and Transition Phases of the spiral development model.

6.3.1.6 Requirements Verification

Requirements Verification is conducted during the Development, Integration, and Deployment phases of the system development life cycle. It is conducted by the IOs, and may be witnessed by OL personnel. In the Cyberinfrastructure IO spiral development methodology, requirements verification occurs during the, Construction, Phase of the spiral development model.

Requirements Verification is the formal process of assessing a product to determine whether it complies with its documented requirements. Individual requirements are verified according to the verification method assigned to them (see section 7.3.3). Requirements Verification results are tracked on the Requirements Verification Matrix. Requirements verification is a superset of other types of testing; i.e., requirements may be verified during unit testing, environmental testing, integration testing, or stability testing, or in several or all of these instances. All product

6.3.1.7 Acceptance Testing

As the first Ocean Leadership (OL) and OOI led verification and testing activity, Acceptance

Testing seeks tovalidate and verify that the requirements were met and that the subsystem,

system or item to be accepted is a quality product. Successful completion of Acceptance Testing results in formal Acceptance of the delivered product, and transfer of "ownership" of the product to OL. Acceptance criteria require that the system, subsystem or item be operating in a near- steady state or non-logistics-supported mode.

Acceptance Testing occurs during the Deployment phase of the system development life cycle. In the Cyberinfrastructure IO spiral development methodology, Acceptance Testing occurs during the Transition Phase of the spiral development model.

The act of Acceptance constitutes acknowledgment that the supplies or services offered to Ocean Leadership (OL) conform to applicable contract or sub-award quality and quantity requirements and are subject to and conform to other terms and conditions of the contract or sub-award. Verification of all requirements related to the delivered product is a prerequisite to formal Acceptance. Other Acceptance testing includes stability testing, usability testing, and the execution of use case scenarios which demonstrate that the system fulfills end user needs. Acceptance Testing and its process is described and maintained in the OOI Quality Assurance

and Quality Control Plan (QA/QC), the OOI Operations and Maintenance Plan (O&M), and as

part of required documentation of any supplier/contractor issued contract.

6.3.1.8 Commissioning Evaluation

The final stage of the OOI verification and testing process, the Commissioning Evaluation is a full deployment test at the Array level, against a fully integrated Array (including software). The Commissioning Evaluation includes the validation and verification that all required logistical and support systems are in place and operational, and that the Array is operating in a controlled "steady state." Commissioning criteria are formally documented, and include that the Array is ready to use in routine operations, operations staff has been trained, and operational

troubleshooting procedures have been developed and documented. Demonstration that the Array meets all of the formal commissioning criteria marks the transfer of responsibility from the Engineering to the Operations domain.

With the OOI Program Office managing and maintaining processes of Commissioning, with input and collaboration with each IO, the Commissioning Evaluation tests are performed by Array, lowest level item through component, assembly, element, sub-system and system as appropriate for each category of test. Commissioning and its process is described and maintained in the OOI

Commissioning Plan and Quality Control Plan and OOI Quality Assurance and Quality Control Plan (QA/QC), OOI Operational and Maintenance Plan (O&M), and as part of required

documentation of any supplier/contractor issued contract. Full Operational Capability (FOC) is achieved as of the completion of the deployment and commissioning cycle.

6.3.1.9 Regression Testing

Regression testing is conducted throughout the Development, Integration, Deployment, and Operations phases of the system development life cycle. In the Cyberinfrastructure IO spiral development methodology, regression testing occurs during the Elaboration, Construction, Transition, and Operations Phases of the spiral development model. Regression tests repeat previously-conducted unit, integration, environmental, stability, or requirements verification tests, and are conducted when hardware or software design or implementation changes are made after the original test is conducted. Regression tests verify that the design or implementation change not only fixed the problem or provided the improvement it was intended to address, but also that the change did not cause any unintended adverse side effects in other capabilities.

Regression tests are conducted by the IOs. Each IO documents and maintains a suite of regression tests, and chooses the correct regression tests to run when system design or implementation changes are made.