Independent Test and Evaluation

(1)

Independent Test and Evaluation

ProPath

(2)

Independent Test and Evaluation Process Maps ... 1

Process: Independent Test and Evaluation ... 3

Independent Test and Evaluation Description and Goals ... 4

Description... 4

Goals... 4

Independent Test and Evaluation RACI Information ... 5

Independent Test and Evaluation Process ... 8

Process Activity Name: ITE-1 Review Testing Intake Assessment Results ... 8

Process Activity Name: ITE-1-DEC01 Independent Testing Required? ... 9

Process Activity Name: ITE-2 Conduct Enterprise Systems Engineering Testing Meeting ... 9

Process Activity Name: ITE-3 Perform Enterprise Systems Engineering Testing ... 10

Process Activity Name: ITE-4 Perform SQAS Systems Integration Testing .. 11

Process Activity Name: ITE-5 Perform Operational Readiness Review ... 13

Process Activity Name: ITE-5.1 Review ESE ERF Results ... 14

Process Activity Name: ITE-5.1-DEC01 ORR Required? ... 15

Process Activity Name: ITE-5.2 Create ESE Testing Checklist ... 15

Process Activity Name: ITE-5.3 Submit/Resubmit ORR ... 16

Process Activity Name: ITE-5.4 Execute ORR Acceptance Review ... 17

Process Activity Name: ITE-5.4-DEC01 Execute ORR? ... 18

Process Activity Name: ITE-5.5 Address ORR Acceptance Issues/Questions ... 19

Process Activity Name: ITE-5.6 Execute ORR ... 20

Process Activity Name: ITE-5.6-DEC01 Open Source Related? ... 21

(3)

Independent Test and Evaluation Process Maps

Pr oj ec t M an ag er En ter pr is e S ys tem s Eng ine er ing C ap ac ity P lan ni ng and E ng ine er ing O pe rat io nal Rea di nes s R ev iew A na lys t En ter pr is e S ys tem s Eng ine er ing Te st A na ly st Sy st em s Q ua lit y A ss ura nc e S erv ic e Ind ep end ent V er ific at io n a nd V al id at io n M an ag er

Independent Test and Evaluation

ITE-3 Perform Enterprise Systems Engineering Testing ITE-4 Perform SQAS Systems Integration Testing ITE-2 Conduct Enterprise Systems Engineering Testing Meeting Independent Testing Required? No ITE-6 Provide to Open Source ITE-1 Review Testing Intake Assessment Results Yes

home overview raci help

ITE-5 Perform Operational Readiness Review Open Source Related? No Yes

(4)

Pr oj ec t M an ag er Ent er pr is e S ys te m s E ng ine er ing C ap ac ity P la nni ng a nd E ng ine er ing O pe ra tio na l R ea din es s R ev ie w A na ly st

Independent Test and Evaluation: ITE-5 Perform Operational Readiness Review

ORR Required?

ITE-5.1 Review ESE ERF Results

home overview raci help

ITE-5.2 Create ESE Testing Checklist ITE-5.3 Submit/Resubmit for ORR No Yes ITE-5.4 Execute ORR Acceptance Review ITE-5.5 Address ORR Acceptance Issues/Questions ITE-5.6 Execute ORR Execute ORR? Rejected Accepted Issues/Questions process

(5)

Process: Independent Test and Evaluation

Overview: The process map for Independent Test and Evaluation cycles through the following process and review activities:

ITE-1 Review Testing Intake Assessment Results ITE-1-DEC01 Independent Testing Required?

ITE-2 Conduct Enterprise Systems Engineering Testing Meeting ITE-3 Perform Enterprise Systems Engineering Testing

ITE-4 Perform SQAS Systems Integration Testing ITE-5 Perform Operational Readiness Review ITE-5.1 Review ESE ERF Results

ITE-5.1-DEC01 ORR Required? ITE-5.2 Create ESE Testing Checklist ITE-5.3 Submit/Resubmit ORR

ITE-5.4 Execute ORR Acceptance Review ITE-5.4-DEC01 Execute ORR?

ITE-5.5 Address ORR Acceptance Issues/Questions ITE-5.6 Execute ORR

ITE-5.6-DEC01 Open Source Related? ITE-6 Provide to Open Source

(6)

Independent Test and Evaluation Description and Goals

Description

Enterprise Testing Service (ETS), an organization within Enterprise Systems Engineering (ESE), works hand-in-hand with Product Development (PD) to provide an independent evaluation of development artifacts and product software. Both Systems Quality Assurance Service (SQAS) and ETS are two forms of evaluations that are not mutually exclusive. The type of evaluation required is dependent upon the results of risk-based analysis. The evaluations help the Office of Information and Technology (OI&T) management minimize risk of schedule delays, cost overrun, poor quality, and software failure.

ETS provides:

- Test environments on which software products can be tested - Independent evaluations of project artifacts and project software Services falling under the category of Test Environments include: - New test database in the ETS Test Center (ETSTC)

- Miscellaneous issue for help with an ETSTC testing environment problem - Database refresh

- Database backup/restore

- User access request to add/reset/modify/deactivate user access to an existing ETSTC environment

Test environment services are requested with an ETSTC Service Request. Services falling under the category of Independent Evaluations include: - Work product reviews of project artifacts

- Test observation and validation of the operation and use of the software product - Various types of testing including legacy Patient Safety Issue (PSI) testing, risk-based independent testing, performance testing, and system integration testing

Goals

- Mitigate risk and provide feedback throughout the lifecycle

- Ensure that products meet acceptable quality levels before promotion - Provide feedback for continuous improvement for process improvement

- Continuously improve the likelihood that a quality product which meets user requirements is deployed on time and on budget

- Ensure consistency in the testing methodologies and practices during Agile development - Validate that products are technically ready for incorporation into the VA infrastructure

(7)

Independent Test and Evaluation RACI Information

The following describes the RACI information for this process:

ITE-1 Review Testing Intake Assessment Results

Responsible Role: Project Manager

Accountable Role: Director, Project Management Division

ITE-1-DEC01 Independent Testing Required?

ITE-2 Conduct Enterprise Systems Engineering Testing Meeting

ITE-3 Perform Enterprise Systems Engineering Testing

Responsible Role: Enterprise Systems Engineering Test Analyst Accountable Role: Director, Project Management Division

ITE-4 Perform SQAS Systems Integration Testing

Responsible Role: Systems Quality Assurance Service Independent Verification and Validation Manager

Accountable Role: Associate Director, Software Quality Assurance Service

ITE-5.1 Review ESE ERF Results

Responsible Role: Enterprise Systems Engineering Capacity Planning and Engineering Operational Readiness Review Analyst

Accountable Role: Director, Enterprise Systems Engineering Capacity Planning and Engineering Informed Role: Project Manager

(8)

ITE-5.1-DEC01 ORR Required?

Accountable Role: Director, Enterprise Systems Engineering Capacity Planning and Engineering

ITE-5.2 Create ESE Testing Checklist

Accountable Role: Director, Enterprise Systems Engineering Capacity Planning and Engineering Informed Role: Integrated Project Team, Project Manager

ITE-5.3 Submit/Resubmit ORR

ITE-5.4 Execute ORR Acceptance Review

ITE-5.4-DEC01 Execute ORR?

ITE-5.5 Address ORR Acceptance Issues/Questions

(9)

Informed Role: Enterprise Systems Engineering Capacity Planning and Engineering Operational Readiness Review Analyst

ITE-5.6 Execute ORR

Accountable Role: Director, Enterprise Systems Engineering Capacity Planning and Engineering Informed Role: Project Manager

ITE-5.6-DEC01 Open Source Related?

ITE-6 Provide to Open Source

(10)

Independent Test and Evaluation Process

Process Activity Name: ITE-1 Review Testing Intake Assessment

Results

Previous Process Activity

None

Next Process Activity

ITE-1-DEC01 Independent Testing Required?

Description

The Project Manager reviews all testing intake assessment results, including Risk Analysis and Testing Scope Report (RATSR) or Testing Intake Analysis (TIA) closure email. The Project Manager then incorporates ESE Enterprise Testing Service (ETS) Independent Testing and/or Systems Quality Assurance Service (SQAS) independent testing services into project plans and schedules.

Artifacts Used

ESE Risk Analysis and Testing Scope Report (RATSR) Initial ESE Test Plan

Requirements Specification Document (RSD) System Design Document (SDD)

Artifacts Created

Independent Testing Requirements (None, SQAS Testing, ESE Testing, Both)

Responsible Role

Project Manager

Tools and Websites

Capacity and Performance Engineering SharePoint site Enterprise Testing Service Portal

Standards

None Listed

More Info

ESE ETS and SQAS evaluations are not mutually exclusive and either one or both types of testing may be needed.

(11)

Process Activity Name: ITE-1-DEC01 Independent Testing Required?

ITE-1 Review Testing Intake Assessment Results

If Yes, the following activities are accomplished concurrently: ITE-2 Conduct Enterprise Systems Engineering Testing Meeting ITE-3 Perform Enterprise Systems Engineering Testing

ITE-4 Perform SQAS Systems Integration Testing

If No, the next activity is ITE-4-DEC01 ORR Review Required?

Description

Note: There is a decision dependency that determines the next activity to be performed. If the Project Manager determines independent testing is required, the following activities are accomplished concurrently:

ITE-2 Conduct Enterprise Systems Engineering Testing Meeting ITE-3 Perform Enterprise Systems Engineering Testing

If the Project Manager determines no independent testing is required, the next activity is ITE-4-DEC01 ORR Review Required?

Project Manager

Process Activity Name: ITE-2 Conduct Enterprise Systems

Engineering Testing Meeting

Note: This activity is performed concurrently with: ITE-3 Perform Enterprise Systems Engineering Testing ITE-4 Perform SQAS Integration Testing

ITE-1-DEC01 Independent Test Required

(12)

Description

The Project Manager conducts a kick off meeting with Enterprise Systems Engineering (ESE) Enterprise Testing Service (ETS) in order to plan and coordinate the delivery of testing services identified for the product in the ESE Risk Analysis and Testing Scope Report (RATSR)

(completed in Project Planning Process). The ESE Testing Checklist for the project is completed prior to the meeting.

Artifacts Used

ESE Risk Analysis and Testing Scope Report (RATSR) Initial ESE Test Plan

Artifact Review Agenda and Minutes

Project Manager

Enterprise Testing Service Portal

Standards

Enterprise Testing Service Testing Guide

More Info

None Listed

Process Activity Name: ITE-3 Perform Enterprise Systems

Engineering Testing

Note: This activity is performed concurrently with:

ITE-2 Conduct Enterprise Systems Engineering Testing Meeting ITE-4 Perform SQAS Systems Integration Testing

ITE-1-DEC01 Independent Testing Required?

(13)

Description

The Enterprise Systems Engineering (ESE) Test Analysts execute Independent Testing. The scope and type of Independent Testing is defined in the Project Risk Assessment and Testing Scope Report (RATSR) completed in the Project Planning Process.

Services include but are not limited to:

- Interoperability/Integration Testing

- Patch Release Testing for Patient Safety Issues - Requirements Validation

- Test Observation and Validation - Performance Testing

Artifacts Used

ESE Risk Analysis and Testing Scope Report (RATSR) ESE Testing Checklist

Project Documents Determined by Type of Test

Enterprise Testing Service Analysis Report

Enterprise Testing Service Performance Testing Information Request Final ESE Testing Test Plan

Enterprise Systems Engineering Test Analyst

Enterprise Testing Service Portal

Standards

Enterprise Testing Service Testing Guide

More Info

The Service Delivery and Engineering (SDE) Program Administration Office (PAO) Process Site provides additional information on the SDE PAO including organization charts, Governance Board Charter, and an overview presentation on PAO Services.

Process Activity Name: ITE-4 Perform SQAS Systems Integration

Testing

Note: This activity is performed concurrently with:

(14)

ITE-3 Perform Enterprise Systems Engineering Testing

ITE-1-DEC01 Independent Test Required

Description

In accordance with VA Directive 4900, the Systems Quality Assurance Service (SQAS) Independent Verification and Validation (IV&V) Manager ensures the required Systems Integration Testing (SIT) and Quality Assurance Activities are performed on software if it impacts VA financial systems of record or software that interfaces with such a financial system. SIT verifies the accuracy of the delivered software product and determines whether the product built satisfies the customers’ requirements based on functional, system and design specifications. SQAS develops the IV&V Software Quality Assurance Plan (SQAP) and IV&V SIT Plan, requirements-based test plans, test cases and test scripts with a focus on internal controls including general and application controls. Internal controls form the core of the Federal Government’s financial management standards. Test scripts are executed against project software deliverables to verify functionality, including interfaces, and to perform regression testing, or verify correction of defects (“bug fixes”). SQAS issues certification for production implementation and verifies the installation of software configurable items. The delivered software module or release is verified using configurable item attributes.

Artifacts Used

Business Requirements Document Defect Report

Interface Control Document Master Test Plan

Product Build

Product Documentation

Requirements Specification Document Requirements Traceability Matrix Software Design Document Software Quality Assurance Plan System Build

System Design Document System Integration Plan Test Cases/Test Scripts

Test Readiness Review Report Test Summary Report

IV&V SIT Defect Report IV&V SQAS Test Report

(15)

IV&V Test Scripts SQAS Certification

Software Quality Assurance Plan Updated System Integration Plan Updated

Systems Quality Assurance Service Independent Verification and Validation Manager

IBM Rational Quality Manager IBM Rational RequisitePro IBM Rational ClearCase

Standards

Federal Manager's Financial Integrity Act of 1982

IEEE Standard 829 for Software and System Test Documentation

IEEE Standard 1012 for System and Software Verification and Validation OMB Circular A-123, Management’s Responsibility for Internal Control OMB Circular A-127, Financial Management Systems (revised)

Software Configuration Management Plan Standard

VA Directive 4900, Internal Controls for Financial and Financial Interfacing Automated Information Systems

More Info

The Federal Managers Financial Integrity Act of 1982 requires that VA systems and their

interfaces comply with quality assurance processes that support functional, technical and internal control requirements.

Process Activity Name: ITE-5 Perform Operational Readiness Review

ITE

None Listed

Description

The process map for ITE-5 Perform Operational Readiness Review cycles through the following dependent activities:

- ITE-5.1 Review ESE ERF Results - ITE-5.1 ORR Required?

(16)

- ITE-5.2 Create ORR Checklist - ITE-5.3 Submit/Resubmit for ORR

- ITE-5.4 Execute ORR Acceptance Review - ITE-5.4-DEC01 ORR Rejected?

- ITE-5.4-DEC02 ORR Accepted?

- ITE-5.5 Address ORR Acceptance Issues/Questions - ITE-5.6 Execute ORR

Process Activity Name: ITE-5.1 Review ESE ERF Results

Note: The following activities are performed concurrently: ITE-2 Conduct Enterprise Systems Engineering Testing Meeting ITE-3 Perform Enterprise Systems Engineering Testing

ITE-5.1-DEC01 ORR Required?

Description

The Enterprise System Engineering Capacity and Performance Engineering (ESE CPE)

Operational Readiness Review (ORR) Analyst analyzes the disposition of the Enterprise Systems Engineering Registration Form – System Characteristics - Project Attributes Form (ERF – SC – PA) to determine if ESE CPE Operational Readiness Review (ORR) is required.

Artifacts Used

ESE Registration Form

ESE Capacity Planning and Engineering (CPE) Services ORR Analysis Report

Enterprise Systems Engineering Capacity Planning and Engineering Operational Readiness Review Analyst

Capacity and Performance Engineering Project Reviews Portal Capacity and Performance Engineering SharePoint site

(17)

ESE Registration Form Portal

Standards

Enterprise Systems Engineering (ESE) Operational Readiness Review Reference

More Info

Files used for ORR are uploaded as zip files to the ESE ORR Checklist.

Process Activity Name: ITE-5.1-DEC01 ORR Required?

If YES, ITE-5.2 Create ESE Testing Checklist OR

If NO, process ends.

Description

The Enterprise Systems Engineering Capacity Planning and

Engineering Operational Readiness Review Analyst determines if ESE CPE Operational Readiness Review (ORR) is required.

Process Activity Name: ITE-5.2 Create ESE Testing Checklist

ITE-5.1-DEC01 ORR Required?

ITE-5.3 Submit/Resubmit for ORR

Description

The Enterprise Systems Engineering (ESE) Capacity Planning and Engineering (CPE)

Operational Readiness Review (ORR) Analyst creates the ESE Testing Checklist as a template for the project team to complete if ESE CPE ORR is required for the project release.

Artifacts Used

(18)

ESE Testing Checklist

Enterprise Systems Engineering (ESE) Testing Checklist Repository ESE Registration Form Portal

Standards

Enterprise Systems Engineering (ESE) Operational Readiness Review Reference Enterprise Systems Engineering Testing Intake Assessment Form User Guide

More Info

None Listed

Process Activity Name: ITE-5.3 Submit/Resubmit ORR

ITE-5.2 Create ESE Testing Checklist

ITE-5.4 Execute ORR Acceptance Review

Description

The Project Manager completes the ESE Testing Checklist and submits for ESE CPE ORR if required for the project release. Once all non-production testing is completed on the build that deploying to Initial Operating Capability (IOC) production or Release (if IOC is not applicable).

If the project team previously submitted for ESE CPE ORR and the ESE CPE ORR was rejected, the project team resubmits for ESE CPE ORR.

Artifacts Used

(19)

Updated ESE Testing Checklist

Project Manager

Enterprise Systems Engineering (ESE) ORR Checklist Repository ESE Registration Form Portal

Standards

Enterprise Systems Engineering (ESE) Operational Readiness Review Reference Enterprise Systems Engineering (ESE) Operational Readiness Review Guide

More Info

None Listed

Process Activity Name: ITE-5.4 Execute ORR Acceptance Review

ITE-5.3 Submit/Resubmit for ORR

ITE-5.4-DEC01 Execute ORR?

Description

The assigned ESE CPE ORR Analyst for the project releases executes an ESE CPE ORR

Acceptance Review. The ESE CPE ORR Analyst analyzes the submitted project release artifacts against the published ESE CPE ORR Acceptance Criteria in order to determine the ESE CPE ORR disposition:

- Rejected: The submitted ESE CPE ORR is grossly deficient (requirements, requirement

traceability, testing results, required items are not provided, etc.). The ESE CPE ORR is rejected and the project team is notified.

- Acceptance Issues: The submitted ESE CPE ORR is not meeting some of the published ESE CPE ORR Acceptance Criteria, but does not qualify as a candidate for rejection. The ESE CPE ORR analyst notifies the project team of the issues and/or questions and the ESE CPE ORR is moved to an acceptance issues status.

(20)

- Accepted: The ESE CPE ORR meets all the published ESE CPE ORR Acceptance Criteria. The project team is notified and the ESE CPE ORR is moved to the accepted status.

Artifacts Used

ESE Testing Checklist

ESE CPE ORR Acceptance Issues/Questions email ESE CPE ORR Accepted Email

ESE CPE ORR Rejection email, or Updated ESE Testing Checklist

Standards

More Info

None Listed

Process Activity Name: ITE-5.4-DEC01 Execute ORR?

If Accepted, ITE-5.6 Execute ORR Or

If Rejected, ITE-5.3 Submit/Resubmit for ORR Or

(21)

If Issues/Questions, ITE-5.5 Address ORR Acceptance Issues/Questions

Description

The Enterprise Systems Engineering Capacity Planning and Project Manager Engineering Operational Readiness Review Analyst determines if the project is ready for Operational Readiness Review (ORR). There are 3 decisions that can occur:

1. If accepted, the ESE CPE ORR meets all the published ESE CPE ORR Acceptance Criteria. The project team is notified and the ESE CPE ORR is moved to accepted status and the next activity is ITE-5.6 Execute ORR.

2. If rejected, the submitted ESE CPE ORR is grossly deficient in that requirements, requirement traceability, testing results, required items, etc. are not provided. The ESE CPE ORR is rejected and the project team is notified and the next activity is ITE-5.3 Submit/Resubmit for ORR

3. If acceptance issues are found, the submitted ESE CPE ORR is not meeting some of the published ESE CPE ORR Acceptance Criteria, but does not qualify as a candidate for rejection. The ESE CPE ORR analyst notifies the project team of the issues and/or questions, the ESE CPE ORR is moved to an acceptance issues status and the next activity is ITE-5.5 Address ORR Acceptance Issues/Questions.

Process Activity Name: ITE-5.5 Address ORR Acceptance

Issues/Questions

ITE-5.4-DEC01 ORR Decision?

Description

The Project Manager addresses the Enterprise Systems Engineering (ESE) Capacity and Performance Engineering (CPE) Operational Readiness Review (ORR) Acceptance Issues/Questions and responds to the assigned ESE CPE ORR Analyst.

Artifacts Used

(22)

ESE CPE ORR Acceptance Issues/Questions email response

Project Manager

Standards

More Info

None Listed

Process Activity Name: ITE-5.6 Execute ORR

ITE-5.4-DEC01 Execute ORR?

Process Ends

Description

The Enterprise System Engineering Capacity and Performance Engineering (ESE CPE) Operational Readiness Review (ORR) Analyst executes the ORR for the accepted project release.

The ORR analyzes and evaluates key technical areas pertinent to ESE, within the context of the service delivery of a high-quality product, especially: requirements definition, design,

traceability, sufficiency of testing, operability (installation and back out), capacity, system resource consumption (including bandwidth), and performance.

Artifacts Used

(23)

ESE Capacity Planning and Engineering (CPE) Services ORR Analysis Report Updated ESE Testing Checklist

Enterprise Systems Engineering (ESE) ORR Checklist Repository

Standards

Enterprise Systems Engineering (ESE) Operational Readiness Review Reference

More Info

None Listed

Process Activity Name: ITE-5.6-DEC01 Open Source Related?

ITE-5.6 Execute ORR

If Yes, ITE-6 Provide to Open Source If No, None

Description

Note: There is a decision dependency that determines the next activity to be performed. If open source related, the next activity is ITE-6 Provide to Open Source.

If not open source related, the process ends.

Project Manager

Process Activity Name: ITE-6 Provide to Open Source

(24)

None

Description

The Project Manager submits a list of required software code, system builds, and artifacts as described in the Open Source Submission Checklist to OI&T PD Open Source, Configuration & Tools Management Division via the mail group OIT Open Source Software Delivery or

[email protected].

Artifacts Used

Open Source Submission Checklist Product and Project Artifacts Software Code

System Build

Updates to Open Source Community

Project Manager

IBM Rational ClearCase

Standards

Open Source Code in Flight Communications Strategy Guide

More Info

The Open Source Submission Checklist lists different artifacts and/or system builds/codes that are to be submitted at certain intervals or milestones to OI&T PD Open Source, Configuration& Tools Management Division, as a product is being built. As project managers are developing their project plans, providing code and artifacts to OI&T PD Open Source, Configuration& Tools Management Division should be considered and incorporated into the plan both based on specific milestones and time intervals basis, as specified in the Open Source Submission

Checklist.

Independent Test and Evaluation

Independent Test and Evaluation

ProPath

Table of Contents

Independent Test and Evaluation Process Maps ... 1

Process: Independent Test and Evaluation ... 3

Independent Test and Evaluation Process Maps

Independent Test and Evaluation

Independent Test and Evaluation: ITE-5 Perform Operational Readiness Review

Process: Independent Test and Evaluation

Independent Test and Evaluation Description and Goals

Description

Goals

Independent Test and Evaluation RACI Information

Independent Test and Evaluation Process

Process Activity Name: ITE-1 Review Testing Intake Assessment

Results

Process Activity Name: ITE-1-DEC01 Independent Testing Required?

Process Activity Name: ITE-2 Conduct Enterprise Systems

Engineering Testing Meeting

Process Activity Name: ITE-3 Perform Enterprise Systems

Engineering Testing

Process Activity Name: ITE-4 Perform SQAS Systems Integration

Testing

Process Activity Name: ITE-5 Perform Operational Readiness Review

Process Activity Name: ITE-5.1 Review ESE ERF Results

Process Activity Name: ITE-5.1-DEC01 ORR Required?

Process Activity Name: ITE-5.2 Create ESE Testing Checklist

Process Activity Name: ITE-5.3 Submit/Resubmit ORR

Process Activity Name: ITE-5.4 Execute ORR Acceptance Review

Process Activity Name: ITE-5.4-DEC01 Execute ORR?

Process Activity Name: ITE-5.5 Address ORR Acceptance

Issues/Questions

Process Activity Name: ITE-5.6 Execute ORR

Process Activity Name: ITE-5.6-DEC01 Open Source Related?

Process Activity Name: ITE-6 Provide to Open Source