Determine Types of Tests

Testing of the ISS features can encompass a broad range of tests from a series of formal tests, as a part of the AMS system development and testing process, to a penetration test.

This is dependent on whether the system is a prototype, in development, currently operating or undergoing modifications. Therefore, the types of tests required must be tailored to each system test program. Table D-1 contains the types of tests that may need to be planned and a definition for each test type. The list is not all-inclusive.

Table D-1. Types of Tests

Test Description

Developmental Tests (DTs)

A series of tests designed to verify that system technical and performance requirements specified in the contract and system specification have been met. Performed by the developer and witnessed by William J. Hughes Technical Center (WJHTC) personnel.

Field

Familiarization

The purpose is to verify that the site is ready to transition to the new system. Conducted by AT and AF field personnel at each new site after the system has successfully completed installation and checkout.

Operational Tests (OTs)

A series of tests designed to demonstrate the system is operationally effective and operationally suitable for use in the NAS, and that the NAS infrastructure is ready to accept the system. These tests focus on

demonstrating operational requirements have been met and that all critical operational issues (COIs) have been resolved, including changes to the security environment. These tests must include both integrity validation and resource consumption testing. For example, tests should included verification of resource management and archiving of audit trail data and system log data. Major components of OT&E are integration tests, suitability tests, and effectiveness tests. The FAA at the WJHTC, an internal or third party, or a field site using field personnel conducts operational testing.

The evaluator attempts to circumvent the security features of a system to gain access. NOTE: Penetration testing on FAA information systems must have advanced coordination and formal authorization with the DAA for the line of business or staff office that owns the system, the information owner (if not the same as the system owner), and the Office of Chief Counsel. If the penetration test could impact one or more systems for which other DAA’s are responsible, then coordination must include all affected DAA’s.

In addition, all personnel participating in the testing should meet background investigation personnel requirements. See section D.9.3.

Production Acceptance Tests (PATs)

A subset of the design qualification tests conducted on the first article plus quality control testing. The vendor for each system conducts this test before it leaves the factory.

System Tests A series of tests designed to verify that a FAA System meets its specified requirements. Subsets of system test are development tests, operational tests, production tests and site acceptance tests. Each must verify satisfaction of all requirements associated with a system.

Test Description Vulnerability

Tests

The evaluator uses commercial and public domain testing tools to attempt to identify security vulnerabilities and modes of compromise that existing security safeguards do not address.

Regression Tests

A series of tests designed to verify the security safeguards introduced in the remediation phase have not altered the required functionality or performance of a system.

Positive Tests A series of tests designed to verify that a system meets its specified security requirements. Testing of boundary or limit values are included.

Negative Tests A series of tests designed to verify that a system does not do anything that is contrary to its security specifications. Tests include violation of

assumptions and specifications. Testing should also ensure that what it does will not have an adverse effect on any other FAA System.

ST&E, as part of the SCAP, should be performed after remediation/mitigation process and should involve ST&E against the mitigation items. The Penetration Test Plan and result may be required to satisfy the SCAP requirement.

For each security requirement to be tested, a test method should be clearly defined.

Table D-2 identifies the categories of test methods to be considered for this effort. Test methods should support testing in a way to provide repeatable and reproducible results.

Tests should be designed so that outcomes are self-evident, requiring a minimum of

subjective interpretation and administrative resolution. For the developer tests provided, the FAA determines whether the tests are repeatable, and the extent to which the developer’s tests can be used for the FAA’s independent testing effort. Any security function for which the developer’s test results indicate that it may not perform as specified should be tested independently by the evaluator so that an acceptance determination can be made. The FAA determines that functional requirements are stated in such a way that they are testable. The FAA also determines that assurance requirements avoid the need for subjective judgment.

Test tools that perform repeatable testing and minimal human interaction/interpretation are necessary. For demonstration, inspection, and test, both automated test tools as well as manual scripted tests and/or checklists may be conducted.

Table D-2. Definition of Test Methods

Methods Definition Implementation

A – Analysis The evaluation using recognized analytical techniques, such as

comparing design with requirements.

Accomplished by review of architectural documents.

D – Demonstration The evaluation by operation, movement, or adjustment under a specific condition to determine the capability to satisfy a stated

requirement.

Consist of test scripts that exercise system capabilities, which include Analysis.

I – Inspection The physical examination or review of the security feature, such as review of a configuration file, software version

T – Test The collection, analysis, and evaluation through systematic hands-on

measurement under appropriate conditions.

Consists of test scripts that exercise system capabilities.