Implementing a Graded Approach in Selecting Risk Analysis Methods

The spectrum of analysis disciplines involved in the risk analysis of alternatives is as broad as the spectrum of performance measures, spanning the mission execution domains of Safety, Technical, Cost, and Schedule. It is not the intent of this handbook to provide detailed guidance on the conduct of domain-specific analyses. Such guidance is available in domain-specific documents like NPR 8715.3C, NASA General Safety Program Requirements [22] (Chapter 2, System Safety), the NASA Cost Estimating Handbook [23], the NASA Systems Engineering Handbook [2], and the NASA Probabilistic Risk Assessment Procedures Guide [24].

Depending on project scale, life cycle phase, etc., different levels of analysis are appropriate. The rigor of analysis should be enough to assess compliance with imposed constraints and support selection between alternatives. Iteration is to be expected as part of the analysis process, but as a general rule of thumb, the rigor of analysis should increase with successive program/project life cycle phases. In addition for a given phase, parametric, engineering, and logic modeling can commence at a low level of detail; the level of detail can be increased in an iterative fashion based on the requirement to reach a robust decision. Figure 20 indicates the types of analysis that are generally appropriate, as a function of life cycle phase, for cost, technical, and safety estimation. Discussion of uncertainty can be found in Section 3.2.2. Detailed information on methods can be found in discipline-specific guidance, e.g., [2], [23], and [24].

Figure 20. Analysis Methodology Guidance Chart

 Cost and Schedule Estimating Methodologies:

o Analogy Estimating Methodology - Analogy estimates are performed on the basis of comparison and extrapolation to like items or efforts. Cost data from one past program that is technically representative of the program to be estimated serves as the basis of estimate. These data are then subjectively adjusted upward or downward, depending upon whether the subject system is believed to be more or less complex than the analogous program.

o Parametric Estimating - Estimates created using a parametric approach are based on historical data and mathematical expressions relating cost as the dependent variable to selected, independent, cost-driving variables through regression analysis. Generally, an estimator selects parametric estimating when only a few key pieces of data are known, such as weight and volume. The implicit Testing Operating Experience Detailed Simulation First-Order Phase E Phase C/D Phase B Phase A Pre-Phase A Testing Operating Experience Detailed Simulation First-Order Phase E Phase C/D Phase B Phase A Pre-Phase A

Technical Estimating Methodology Guidance Chart

Engineering Build Up Parametric Analogy Phase E Phase C/D Phase B Phase A Pre-Phase A Engineering Build Up Parametric Analogy Phase E Phase C/D Phase B Phase A Pre-Phase A

Detailed Logic Modeling

Typically Not Applicable Applicable Primary Legend: Statistical Methods First-Order Parametric Similarity Phase E Phase C/D Phase B Phase A Pre-Phase A

Detailed Logic Modeling

Typically Not Applicable Applicable Primary Legend: Statistical Methods First-Order Parametric Similarity Phase E Phase C/D Phase B Phase A Pre-Phase A

Safety, Reliability, and Operations Estimating Methodology Guidance Chart Cost Estimating Methodology Guidance Chart

assumption of parametric estimating is that the same forces that affected cost in the past will affect cost in the future.

o Engineering Build Up Methodology - Sometimes referred to as “grass roots” or “bottom-up” estimating, the engineering build up methodology rolls up individual estimates for each element into the overall estimate. This methodology involves the computation of the cost of a work breakdown structure (WBS) element by estimating at the lowest level of detail (often referred to as the “work package” level) wherein the resources to accomplish the work effort are readily distinguishable and discernable. Often the labor requirements are estimated separately from material requirements. Overhead factors for cost elements such as Other Direct Costs (ODCs), General and Administrative (G&A) expenses, materials burden, and fees are generally applied to the labor and materials costs to complete the estimate.

 Estimating Methodologies for Technical Performance Measures:

o First-Order Estimating Methodology - First-order estimates involve the use of closed-form or simple differential equations which can be solved given appropriate bounding conditions and/or a desired outcome without the need for control-volume based computational methods. The equations may be standard physics equations of state or empirically-derived relationships from operation of similar systems or components.

o Detailed Simulation Estimating Methodology - Estimates using a detailed

simulation require the construction of a model that represents the physical states of interest in a virtual manner using control-volume based computational methods or methods of a similar nature. These simulations typically require systems and conditions to be modeled to a high-level of fidelity and the use of “meshes” or network diagrams to represent the system, its environment (either internal, external, or both), and/or processes acting on the system or environment. Examples are computational fluid dynamics (CFD) and finite-element modeling. o Testing Methodology - Testing can encompass the use of table-top experiments all

the way up to full-scale prototypes operated under real-world conditions. The objective of the test is to measure how the system or its constituent components may perform within actual mission conditions. Testing could be used for assessing the expected performance of competing concepts or for evaluating that the system or components will meet flight specifications.

o Operating Experience Methodology - Once the system is deployed data gathered during operation can be analyzed to provide empirically accurate representations of how the system will respond to different conditions and how it will operate throughout its lifetime. This information can serve as the basis for applicable changes, such as software uploads or procedural changes, that may improve the overall performance of the system. Testing and detailed simulation may be

combined with operating experience to extrapolate from known operating conditions.

 Safety, Reliability, & Operations Estimating Methodologies:

o Similarity Estimating Methodology - Similarity estimates are performed on the basis of comparison and extrapolation to like items or efforts. Reliability and operational data from one past program that is technically representative of the program to be estimated serves as the basis of estimate. Reliability and operational data are then subjectively adjusted upward or downward, depending upon whether the subject system is believed to be more or less complex than the analogous program.

o First-Order Parametric Estimation - Estimates created using a parametric approach are based on historical data and mathematical expressions relating safety, reliability, and/or operational estimates as the dependent variable to selected, independent, driving variables through either regression analysis or first-order technical equations (e.g., higher pressures increase the likelihood of tank rupture). Generally, an estimator selects parametric estimating when the system and its concept of operation are at the conceptual stage. The implicit assumption of parametric estimating is that the same factors that shaped the safety, reliability, and operability in the past will affect the system/components being assessed.

o Detailed Logic Modeling Estimation - Detailed logic modeling estimation

involves “top-down” developed but “bottom-up” quantified scenario-based or discrete-event logic models that segregate the system or processes to be evaluated into discrete segments that are then quantified and mathematically integrated through Boolean logic to produce the top-level safety, reliability, or operational estimate. Detailed technical simulation and/or testing, as well as operational data, can be used to assist in developing pdfs for quantification of the model. Typical methods for developing such models may include the use of fault trees, influence diagrams, and/or event trees.

o Statistical Methods - Statistical methods can applied to data collected during system/component testing or from system operation during an actual mission. This is useful for characterizing the demonstrated safety, reliability, or operability of the system. In addition, patterns in the data may be modeled in a way that accounts for randomness and uncertainty in the observations, and then serve as the basis for design or procedural changes that may improve the overall safety, reliability, or operability of the system. These methods are useful for answering yes/no questions about the data (hypothesis testing), describing associations within the data (correlation), modeling relationships within the data (regression), extrapolation, interpolation, or simply for data mining activities.