Probabilistic Modeling of Performance

If there were no uncertainty, the question of performance assessment would be one of quantifying point value performance measures for each decision alternative. In the real world, however, uncertainty is unavoidable, and the consequences of selecting a particular decision alternative cannot be known with absolute precision. When the decision involves a course of action, there is uncertainty in the unfolding of events, however well planned, that can affect the achievement of objectives. Budgets can shift, overruns can occur, technology development activities can encounter unforeseen phenomena (and often do). Even when the outcome is realized, uncertainty will still remain. Reliability and safety cannot be known absolutely, given

finite testing and operational experience. The limits of phenomenological variability in system performance can likewise not be known absolutely, nor can the range of conditions under which a system will have to operate. All this is especially true at NASA, which operates on the cutting edge of scientific understanding and technological capability.

For decision making under uncertainty, risk analysis is necessary, in which uncertainties in the values of each alternative’s performance parameters are identified and propagated through the analysis to produce uncertain performance measures (see Figure 6 in Section 1.5.2). Moreover, since performance measures might not be independent, correlation must be considered. For example, given that labor tends to constitute a high fraction of the overall cost of many NASA activities, cost and schedule tend to be highly correlated. High costs tend to be associated with slipped schedules, whereas lower costs tend to be associated with on-time execution of the program/project plan.

One way to preserve correlations is to conduct all analysis within a common Monte Carlo ―shell‖ that samples from the common set of uncertain performance parameters, propagates them through the suite of analyses, and collects the resulting performance measures as a vector of performance measure values [28]. As the Monte Carlo shell iterates, these performance measure vectors accumulate in accordance with the parent joint pdf that is defined over the entire set of performance measures. Figure 23 notionally illustrates the Monte Carlo sampling procedure as it would be applied to a single decision alternative (Decision Alternative i).

Uncertainties are distinguished by two categorical groups: aleatory and epistemic [29], [30]. Aleatory uncertainties are random or stochastic in nature and cannot be reduced by obtaining more knowledge through testing or analysis. Examples include:

 The room-temperature properties of the materials used in a specific vehicle.

 The scenario(s) that will occur on a particular flight.

In the first case, there is random variability caused by the fact that two different material samples will not have the same exact properties even though they are fabricated in the same manner. In the second case, knowing the mean failure rates for all the components with a high degree of certainty will not tell us which random failures, if any, will actually occur during a particular flight. On the other hand, epistemic uncertainties are not random in nature and can be reduced by obtaining more knowledge through testing and analysis.

Planetary Science Mission Example: Set the Analysis Framework

The figure below shows the risk analysis framework used to integrate the domain-specific analyses. Each alternative is characterized by its performance parameters, some of which are uncertain (shown in red text) and others of which have definite, known deterministic values (shown in black text). In order to calculate the performance measures previously selected for illustrative purposes, four separate performance models have been developed for radiological contamination, data collection, schedule, and cost. Some performance parameters, such as spacecraft structure mass, launch reliability, and science package TRL, are used in multiple models. Some models (e.g., the data collection model) produce outputs (e.g., science package mass) that are inputs to other models (e.g., the schedule model).

Risk Analysis Framework

The analysis framework shown above was used for all four alternatives selected for risk analysis. However, in general, each alternative may require its own analysis framework, which may differ substantially from other alternatives’ frameworks based on physical or operational differences in the alternatives themselves. When this is the case, care should be taken to assure analytical consistency among alternatives in order to support valid comparisons.

Performance

Parameters: Budgeted Science Package Mass

Time to End of Launch Window Spacecraft Structure Mass Launch Reliability Insertion Reliability RTG Load given Insertion Failure Nominal RTG Structural Capacity Science Package TRL Nominal Science Package Life Launch Vehicle Payload Capacity Spacecraft Insertion System Mass Nominal Spacecraft Development Time Nominal Science Package Development Time Domain- Specific Analyses: Radiological Contamination Model Data Collection Model Schedule

Model ModelCost

Performance

Measures: Probability of PuContamination Data CollectionMonths of CompletionTime to MissionCost Science

Package Mass

Uncertain Parameter

Figure 23. Risk Analysis Using a Monte Carlo Sampling Procedure

Examples include:

 The properties of a material at very high temperatures and pressures that are beyond the capability of an experimental apparatus to simulate

 The mean failure rates of new-technology components that have not been exhaustively tested to the point of failure in flight environments.

In both cases, the uncertainty is caused by missing or incomplete knowledge or by limitations in the models used to make predictions.

The assessed risk of not meeting performance requirements during a given mission are affected by both types of uncertainty. For example, if a component were required to operate for 17 years with 90% confidence during a flight to other planets in our solar system, and it had only been tested for 1 year, the evaluation of whether it meets the 90% confidence requirement would have to include both aleatory uncertainty (e.g., the possibility of a premature failure given a known mean failure rate) and epistemic uncertainty (e.g., uncertainty in the mean failure rate due to the limited test time). It is important to include both types of uncertainty in evaluating the performance risk. It is also important to know the relative contribution of each type of failure,

Monte Carlo Shell

Risk Analysis Framework & Models

n npun npu npupu n u pu npu npu pu npu pu npunp npu npunpu npu npunp npu npunpu npunpunpu

npu npupu e o mance a amete 1 e o mance a amete 2 e o mance a amete m e o mance

Measu e 1 Measu e 2e o mance Measu e e o mancen

n npun npun npu npupunpu npupu n u pu npun u pu npu npu pu npunpupu npu pu npunppu npunp npu npunpunpu npunpu npu npunp npu npunp npu npunpunpu npunpu npunpunpunpu

npunpu npu npunpupu npupu e o mance a amete 1 e o mance a amete 2 e o mance a amete m e o mance

Measu e 1 Measu e 2e o mance Measu e e o mancen

Sample Uncertain Performance Parameter 1 1 0 P1 Sample Uncertain Performance Parameter n 1 0 Pn …

Sampled Performance Parameter Values*

Iteration # 1 2 3 N Performance Measure 1 PM1,1 PM1,2 PM1,3 PM1,N Performance Measure n PMn,1 PMn,2 PMn,3 PMn,N … … … … … Add Row Iterate N Times

Risk Analysis Output

since the former source of risk could not be reduced by more testing (without design modification) but the latter source could.