F ORMULATING THE TEMP V ERIFICATION R EQUIREMENTS T RACEABILITY M ATRIX The TEMP Verification Requirements Traceability Matrix (VRTM) provides a structured

approach that ensures planned testing is comprehensive and complete. It maps all

COIs/MOEs/MOSs/MOPs as well as the program requirements into T&E phases (DT or OT) and associated test activities (see Sections 6.1.1 and 7.1 for descriptions of the DT and OT test

activities). Additionally, each MOP and program requirement contained in the TEMP VRTM is assigned one or more of the following test verification method(s):

a) Demonstration (D): Verification that is accomplished by operation, adjustment, or reconfiguration of items performing their designed functions under specific scenarios. The items may be instrumented and quantitative limits of performance monitored, but only observational data rather than actual performance data is required to be recorded for verification. Demonstration is often used to verify compliance with requirements in servicing, reliability, maintainability, transportability, and human factors

engineering.

Note: Demonstration does not require any actions beyond those identified in the Test Steps section of the associated test procedures.

b) Test (T): Verification that is accomplished, with or without instrumentation, through systematic exercising of the application item under appropriate conditions with the collection, analysis, and evaluation of quantitative data.

Note: Acceptability of the item is determined by comparison of the data with pre- established quantitative criteria, requirements, and occurrences.

c) Analysis (A): Verification that is accomplished through use of one or more of the following analysis techniques to prove that an item meets specified requirements: 1) Mathematical representation such as math models, algorithms, and equations 2) Charts

3) Graphs

4) Circuit diagrams

5) Data reduction/recording

6) Representative data (may include data collected from previous or other equipment and system verifications)

d) Inspection (I): Verification that is accomplished by a visual examination of the item, reviewing descriptive documentation, and comparing the appropriate characteristics with predetermined standards to determine conformance to requirements without the

use of laboratory equipment or procedures. Examples of verification by inspection are:

1) Visual analysis of the item under test, such as displays, cables, and processors. 2) Reviewing descriptive documentation such as Contract Data Requirements List

(CDRL) items, vendor data, and engineering drawings.

3) Comparing the appropriate characteristics with a predetermined or reference standard such as FAA and industry standards.

Development of the VRTM is an iterative process. During early TEMP development, the TEMP VRTM is generated with a mapping of COIs/MOEs/MOSs/MOPs and program requirements down to the test phase level for DT and down to the test activity level for OT. As the program progresses, additional information such as that included in the ISPD, the CMTP, and the FAA specification will provide further details that must be incorporated into the VRTM.

For DT, the developing contractor will use the FAA System Specification to develop their contractor specification(s) defining the DT requirements. The CMTP will then be developed, containing a DT VRTM which maps the DT requirements to the DT test activities.

Subsequently, the developing contractor will refine the DT VRTM with test case information during DT Test Plan development (see Sections 6.2.2, 6.2.3, and 6.2.5). The TEMP VRTM should be updated with the DT test activity and test case information from the DT VRTM. For OT, the TEMP VRTM will provide the basis for developing the OT VRTM. During OT Test Plan development, the VRTM will be further refined to include COI/MOE/MOS/MOP and program requirement mapping to the individual OT test cases (see Sections 7.4.1 and 7.4.2).

Note: Refer to the V&V Repository for the VRTM template. 5.3.3.6 IDENTIFYING TEST CAPABILITY REQUIREMENTS

Test methodologies and scenarios are reviewed to identify the test tools and environment. Test tools and environment requirements should be identified in a consolidated list for traceability, including associated “need by” dates. Particular focus is placed on items with long lead times or high costs. Typical examples include the preparation, procurement, or development of: NAS interfaces that require significant coordination; telecommunications services; test tools; models and simulations; and laboratories and testbeds.

The set of all test methodologies, scenarios, and conditions are primary drivers for assessing required test tools and environments. When assessing the need for test tools and environments, the following factors should be considered:

a) Measures of Performance: Reviewing MOPs provides insight into the resources that may be required to collect and analyze data. Instrumentation, databases, data

extraction tools, questionnaires, and analysis tools are examples of resources that may be required.

b) Test conditions: Reviewing the required test conditions (e.g., loads, configurations, adaptations, and environments) provides insight into the required resources.

c) Test methodology: How a set of test activities is executed also contributes to the rationale or basis for resource requirements. Reviewing the test methodologies

provides insight into what might be needed to execute the test activities (e.g., field testing may add costs, modeling and simulation may incur both increased costs and potentially long lead times for development, etc.).

d) Test scenarios: Test scenarios are developed to exercise combinations of operational factors which influence system performance. Reviewing all factors that have been identified may provide insight into additional assets that may be required to

approximate operationally realistic Air Traffic Control (ATC) situations and capture “end-to-end” situational flow.