Overview of: Mission and System Design Verification and Validation

(1)

Overview of:

Mission and System Design Verification and Validation

SDOE 633

Presented by

With our Academic Partner

(2)

The Challenge...

The US Forest Service needs a more effective means to detect and monitor potentially dangerous wildfires

Turning this...

Into This...

Mission Operations Element

ESA’s Space Operations Center.

Courtesy ESA

Communication Architecture. US Fig. 15-11.

Subject Element:

Wildfires

Space Element

Orbits & Trajectories Launch Element

3.1 Form shall be...

3.2 Fit shall be...

3.3 Function shall be...

System Requirements Document

Valid Baseline Requirements

(3)

The Challenge...

4.1 Form shall be verified by...

Valid Baseline Requirements

Turning this... ...into this

While Managing all of this...

Verification Closure Notices

Test & Verification Requirements (TVRs)

Procedures

Facilities &

Equipment

Personnel Master

Verification Plan

(4)

Course Description

SDOE 633 provides hands-on

opportunities to apply key principals of space systems engineering. Students are given a set of customer expectations in the form of broad mission objectives. Using state- of-the-industry mission design and analysis tools, students apply systems engineering

processes to define top-level system

requirements, design key elements, and conclude with a system design review. In V&V, participants experience system

realization processes first hand by

integrating, verifying, validating, and

delivering the shoe box–sized EyasSAT satellite. From the part-level to the system level, participants implement a rigorous

assembly, integration, verification, and

validation plan on space hardware/software applying “test like you fly, fly like you test”

principles.

(5)

System Engineering Life-cycle Processes

Define, Goals &

Objetctives

Develop Concept of Operations

Engineer &

Manage Requirements Create

Functional

&

Physical Architectures

DESIGN

Integrate Implement

Verify &

Validate

Transition

NEEDS

CAPABILITIES

REALIZE MANAGE

Build-to/Code-to Baseline Mission

Baseline System Baseline

Functional Baseline Design-to

Baseline

As-built Baseline As-deployed

Baseline

We’ll use this model of Systems Engineering Processes to guide our discussion

The purpose of this course is to help bridge the gap between the “left side” and

“right side” of the SE “V”

(6)

SDOE 633 Part A

• Given: a brief Request for Proposal or Announcement of Opportunity...

• Using: Integrated Mission Design Spreadsheet Tool

• Satellite Tool Kit

• Complete the Conceptual Design for an entire mission...

– Orbit design

– Spacecraft design

– Launch vehicle selection – Operations Concept

– Driving Requirements and Key Performance Parameters

4.1 Form shall be verified by...

(7)

SDOE 633 Part A Objectives

• Cultivate a better understanding of the overall space systems engineering process

– Technical processes, tools and information available – Interpersonal skills and distributed collaborative efforts

• Enhance space system engineering skills—system engineering management, technical integrity and technical leadership

• Integrate all elements of a successful mission

• Explore key trajectory constraints on space mission design

• Establish a process to refine requirements and define parameters to meet mission objectives at acceptable cost and risk

• Practical application of the information and processes in a non- threatening environment

• Promote system-thinking by all participants

Start with a blank sheet of paper and develop a detailed Mission Concept Review (MCR) to meet a set of broad objectives

(8)

SDOE 633 Part A Description

• A learning laboratory for participants to explore and use the competencies learned on the job and in the Space Mission Analysis and Design, Designing Cost-Effective Space

Missions, Understanding Space, and Human Spaceflight courses and workshops

– Interpersonal skills

– Space system engineering

– Collaborative design techniques

• During the design exercise, the team…

– Determines the requirements from the customer

– Develops an appropriate mission concept to meet the requirements – Completes the conceptual design of a mission and the associated

space and ground assets to provide the necessary products or services to identified customers and users

(9)

SDOE 633 Part A Description (cont’d)

• Participants…

– Exercise the Space Mission Analysis and Design processes and use space system engineering fundamentals to create a

reasonable science mission opportunity

– Learn the ties between customer requirements, mission and system design and lifecycle cost

– Develop an instinct for the technical concepts and parameters to adjust to make their newly conceived mission more cost-effective – Learn to think and work by way of a practical, end-to-end

example – beginning with customer needs to create a viable space-based product or service

• The design exercise, hence, serves as a concrete and

practical means to apply and test system engineering

techniques in a non-threatening, real-life environment

(10)

SDOE 633 Part A Flow

• Getting Started

– Overview of the SE and conceptual mission design processes – Introduction to Space Space Systems Engineering

– Conceptual Mission Design

– Review of Science Mission & System Design & Operations – Introduce Design Exercise “Announcement of Opportunity”

– Firesat case study example – Team assignments

– Introduction to Space Mission Analysis and Design (SMAD) and STK software tools using Firesat example

– Begin concurrent design sessions (requirements definition)

• Getting Down to Work

– Firesat case study example (cont.)

– Concurrent design sessions and status reviews

• Finishing Up

– Concurrent design sessions – Working lunch

– Team final presentations and feedback (after lunch) – Wrap-up/Critique

(11)

Design Tools...The SMAD Spreadsheet

• Integrates 100’s of equations as well as cost and

operational complexity models from Space Mission Analysis and Design text to create a fully-integrated Excel-based

system modeling tool for rapid trade studies

– Shallow learning curve allows students with basic background in

space mission design to quickly begin analyzing trade-offs between

(12)

SMAD Spreadsheet Navigator

(13)

SMAD Spreadsheet Example Output

(14)

SDOE 633 Part B

Verification Planning

EyasSAT Exercises

REQUIREMENT VERIFY

METHOD (LEVEL)

EVENT(S) SUCCESS CRITERIA VALID REQ’T

? VERIFY STATUS?

VERIFY STATUS?

COMMENTS

3.1 System Characteristics:

EyasSAT System characteristics shall be as refined by the following:

Inspection (SYSTEM)

System Acceptance Review

If verification of all characteristic requirements have been successfully completed.

3.1.1. System Definition:

EyasSAT system major components shall include the following: (1) Structure

& Integration Subsystem (SIS), (2) Electrical Power Subsystem (EPS) Module, (3) Data Handling Subsystem (DHS) Module, (4) Communication Module (Comm), and (5) Attitude Determination & Control Subsystem (ADCS) Module, LED Test Module assembled as per specifications

Inspection (SYSTEM)

Subsystem Baseline Physical Inspections AND System Baseline Physical Inspection

If all specified major components are included

3.1.2. System Mass: Total system mass shall not exceed 3.0 kg, Subsystem mass is allocated as follows:

Inspection (SYSTEM)

System Baseline Physical Inspection

If system mass does not exceed 3.0 kg.

3.1.2.1 SIS Mass: SIS mass shall not exceed 1.5 kg.

Inspection (SUBSYSTEM)

Subsystem Baseline Physical Inspections

If SIS mass does not exceed 1.5 kg

3.1.2.2 EPS Mass: EPS Module mass, including LED Test Module, shall not exceed 0.5 kg.

Inspection (SUBSYSTEM)

Subsystem Baseline Physical Inspections

If EPS mass does not exceed 0.5 kg

EyasSAT Requirements Verification Matrix

Rev. 8.1 August 21, 2009 1

Requirements Validation

Part-level Verification

Software Verification

& Validation

Subsystem Verification System

Integration

• A combination of lecture and extensive hands-on exercises using the EyasSAT Educational Satellite System

– To teach space system verification and validation principles and practices

(15)

SDOE 633 Part B Objectives

• At the end of this course you should be able to:

– Explain the end-to-end SE process and how it applies to system (and lower level) requirements definition, allocation, validation and verification.

– Describe the purpose and scope of key documents required in the

validation and verification processes, and describe typical errors committed.

– Describe various methods of verification, when they are appropriate. and how they are used as part of a verification plan for a system of interest – Determine appropriate circumstances and applicability of verification

methods to prototype and proto-flight systems.

– Analyze representative verification plans, test sequences and activities for an example system of interest (spacecraft).

– Develop, evaluate and implement a master verification plan for a space system including hardware, software and associated ground support

equipment (GSE).

– Apply processes and techniques in a hands-on workshop associated with a system of interest.

– Describe applicable NASA, ECSS, DoD and Industry Standards and lessons learned to support system verification decisions and activities.

(16)

Why Focus on V&V?

Mission/Year Mishap Verification & Validation

Contributing Factors

Genesis/2004 G-switch installed backwards

!parachute not deployed ! hard landing

“no system-level test” (of G-switch)

Columbia/2003 debris damaged thermal tiles ! loss of

crew and vehicle “current [modeling] tools, including the Crater model, are inadequate...”

“flight configuration was validated using extrapolated test data...rather than direct testing”

Comet Nucleus Tour

(CONTOUR)/2002 overheating of s/c by solid rocket motor

plume ! vehicle lost “Project reliance on analysis by similarity”

Wide Field InfraRed

Explorer (WIRE)/1999 electronic startup transient !early cover jettison !cryogen boil-off science mission lost

“failure to correctly identify the source of the signal which caused Electro Explosive

Device (EED) Simulator to ‘latch’ upon Pyrotechnics Box power-up during

spacecraft integration testing”

Mars Polar Lander/

1998

software flaw !descent engine shut-off too soon !vehicle lost

“employed analysis as a substitute for test in the verification and validation of total system performance...tests employed to develop or validate the constituent models were not of adequate fidelity.”

ASSE Table 11-2

(17)

V&V Inputs & Outputs

Inputs Verification, Validation

& Certification Activities

Outputs

•Un-validated Requirements Requirements Validation •Validated Requirements

•Un-validated Mission Critical Models

•Validated Model Requirements Model Validation

•Validated Models

•Model Uncertainty Factors (MUFs)

•List of Model idiosyncrasies

•Validated Requirements

•Un-verified End Product(s)

•Verification Plan (including incompressible test list)

•Verification Enabling Products (e.g.

validated models)

Product Verification

•Verification Plan (as implemented)

•Verified End Product(s)

•Verification Products (data, reports,

verification closure notices, work products)

•Verified End Product(s)

•Customer Expectations (e.g. MOEs and other acceptance criteria)

•Operations Concept

•Validation Plan

•Validation Enabling Products

Product Validation

•Validated Product(s)

•Validation Products (e.g. data, test reports, work products)

•Verified & Validated Product(s)

•Verification & Validation Product(s)

•Real-world Operational Context for End Product(s)

Flight Certification

•Certified Product

•Certification Products (e.g. signed DD250, completed functional config. audit (FCA, physical config. audit (PCA), mission rules)

(18)

SDOE 633 Part B Agenda

LECTURES

• Intro to Space Systems Engineering

• The EyasSAT System of Interest

• Validating Requirements & Models

• Verifying Products

• Verification of COTS/NDI

• Software Verification & Validation

• Validating Products and Flight Certification

Goal: Achieve an ability to analyze, synthesize and critically evaluate V&V plans and real-world implementations

through interactive lectures and hands-on exercises

EXERCISES

• SRPL

• EyasSAT Requirements Validation

• EyasSAT Verification Planning

• EyasSAT Parts-Level Verification Events

• EyasSAT Software V&V Event

• EyasSAT Subsystem Verification Events

• EyasSAT System Verification Events

• EyasSAT System Validation Events

• EyasSAT System Acceptance Review

(19)

SDOE 633 Course Overview

SDOE 633 Admin

• Prerequisites

– Completion of SDOE 632 or 635 (or equivalent space mission design experience)

• Grading

– Mission Concept Review, 40%

» combined oral presentation and written design package in Powerpoint

– Final Exam (online), ^60%

» Focused on key concepts and practices in Space System Verification and

19 G.3 Mission Concept Review

The MCR affirms the mission need and examines the proposed mission’s objectives and the concept for meeting those objectives.

Table G-3 – MCR Entrance and Success Criteria Mission Concept Review

Entrance Criteria Success Criteria

1. Mission goals and objectives.

2. Analysis of alternative concepts to show at least one is feasible.

3. Concept of operations.

4. Preliminary mission descope options.

5. Preliminary risk assessment, including technologies and associated risk management/mitigation strategies and options.

6. Conceptual test and evaluation strategy.

7. Preliminary technical plans to achieve next phase.

8. Defined MOEs and MOPs.

9. Conceptual life-cycle support strategies (logistics, manufacturing, and

operation).

1. Mission objectives are clearly defined and stated and are unambiguous and internally consistent.

2. The preliminary set of requirements satisfactorily provides a system that will meet the mission objectives.

3. The mission is feasible. A solution has been identified that is technically feasible. A rough cost estimate is within an acceptable cost range.

4. The concept evaluation criteria to be used in candidate systems evaluation have been identified and prioritized.

5. The need for the mission has been clearly identified.

6. The cost and schedule estimates are credible.

7. An updated technical search was done to identify existing assets or products that could satisfy the mission or parts of the mission.

8. Technical planning is sufficient to proceed to the next phase.

9. Risk and mitigation strategies have been identified and are acceptable based on technical risk assessments.

G.4 System Requirements Review

The SRR examines the functional and performance requirements defined for the system and the preliminary program or project plan and ensures that the requirements and the selected concept will satisfy the mission.

Table G-4 – SRR Entrance and Success Criteria System Requirements Review

Entrance Criteria Success Criteria

1. Successful completion of the MCR and responses made to all MCR Requests for Actions (RFAs) and Review Item Discrepancies (RIDs).

2. A preliminary SRR agenda, success criteria, and charge to the board have been agreed to by the technical team, project manager, and review chair prior to the SRR.

3. The following technical products for hardware and software system elements are available to the cognizant participants prior to the review:

a. system requirements document;

b. system software functionality description;

c. updated concept of operations;

d. updated mission requirements, if applicable;

e. baselined SEMP;

f. risk management plan;

g. preliminary system requirements allocation to the next lower level

1. The project utilizes a sound process for the allocation and control of requirements throughout all levels, and a plan has been defined to complete the definition activity within schedule constraints.

2. Requirements definition is complete with respect to top- level mission and science requirements, and interfaces with external entities and between major internal elements have been defined.

3. Requirements allocation and

(20)

Overview of:

Mission and System Design Verification and Validation

SDOE 633

Presented by

With our Academic Partner