Probabilistic Modeling of Performance

If there were no uncertainty, the question of performance assessment would be one of quantifying point value performance measures for each decision alternative. In the real world, however, uncertainty is unavoidable, and the consequences of selecting a particular decision alternative cannot be known with absolute precision. When the decision involves a course of action there is uncertainty in the unfolding of events, however well planned, that can affect the achievement of objectives. Budgets can shift, overruns can occur, technology development activities can encounter unforeseen phenomena (and often do). Even when the outcome is realized, uncertainty will still remain. Reliability and safety cannot be known absolutely, given

finite testing and operational experience. The limits of phenomenological variability in system performance can likewise not be known absolutely nor can the range of conditions under which a system will have to operate. All this is especially true at NASA, which operates on the cutting edge of scientific understanding and technological capability.

For decision making under uncertainty, risk analysis is necessary, in which uncertainties in the values of each alternative’s performance parameters are identified and propagated through the analysis to produce uncertain performance measures (see Figure 6 in Section 1.5). Moreover, since performance measures might not be independent, correlation must be considered. For example, given that labor tends to constitute a high fraction of the overall cost of many NASA activities, cost and schedule tend to be highly correlated. High costs tend to be associated with slipped schedules, whereas lower costs tend to be associated with on-time execution of the program/project plan.

One way to preserve correlations is to conduct all analysis within a common Monte Carlo “shell” that samples from the common set of uncertain performance parameters, propagates them through the suite of analyses, and collects the resulting performance measures as a vector of performance measure values [25]. As the Monte Carlo shell iterates, these performance measure vectors accumulate in accordance with the parent joint pdf that is defined over the entire set of performance measures. Figure 21 notionally illustrates the Monte Carlo sampling procedure as it would be applied to a single decision alternative (Decision Alternative i).

Uncertainties are distinguished by two categorical groups: aleatory and epistemic [26], [27]. Aleatory uncertainties are random or stochastic in nature and cannot be reduced by obtaining more knowledge through testing or analysis. Examples include:

 The room-temperature properties of the materials used in a specific vehicle.  The scenario(s) that will occur on a particular flight.

In the first case, there is random variability caused by the fact that two different material samples will not have the same exact properties even though they are fabricated in the same manner. In the second case, knowing the mean failure rates for all the components with a high degree of certainty will not tell us which random failures, if any, will actually occur during a particular flight. On the other hand, epistemic uncertainties are not random in nature and can be reduced by obtaining more knowledge through testing and analysis. Examples include:

 The properties of a material at very high temperatures and pressures that are beyond the capability of an experimental apparatus to simulate.

 The mean failure rates of new-technology components that have not been exhaustively tested to the point of failure in flight environments.

Figure 21. Risk Analysis Using a Monte Carlo Sampling Procedure

In both cases, the uncertainty is caused by missing or incomplete knowledge or by limitations in the models used to make predictions.

The assessed performance of an alternative is affected by both types of uncertainty in basically the same way. That is to say, if there were no epistemic uncertainties there would still be uncertainty in the assessed performance because of the unpredictability of performance parameters that are subject to aleatory uncertainty. Likewise, if there were no aleatory

Monte Carlo Shell

Risk Analysis Framework & Models

Input InputInput Input Input Input Input Input Input Input InputInput Input Input Input Input Input Input Input InputInput Input InputInput Input Input Input Input InputInput Performance Parameter 1 Performance Parameter 2 Performance Parameter m … Performance Measure 1 Performance Measure 2 Performance Measure n … Input InputInput InputInput Input Input InputInput Input Input Input Input InputInput Input Input Input InputInputInput InputInput Input Input InputInput Input Input Input Input InputInput Input Input Input InputInput Input InputInput Input InputInputInput InputInput Input Input Input Input Input Input Input InputInputInput InputInput Performance Parameter 1 Performance Parameter 2 Performance Parameter m … Performance Measure 1 Performance Measure 2 Performance Measure n … Sample Uncertain Performance Parameter 1 1 0 P1 Sample Uncertain Performance Parameter n 1 0 Pn …

Sampled Performance Parameter Values*

Iteration # 1 2 3 N Performance Measure 1 PM1,1 PM1,2 PM1,3 PM1,N Performance Measure n PMn,1 PMn,2 PMn,3 PMn,N … … … … … Add Row Iterate N Times

Risk Analysis Output

uncertainties there would still be uncertainty in the assessed performance caused by the possibility of our mischaracterizing reality due to imperfect information.

It has become common in risk analysis to separate these two contributions to uncertainty by using the term risk to reflect the variability caused by aleatory uncertainties alone, and the term risk uncertainty or simply uncertainty to reflect the impreciseness of our knowledge of the risk caused by epistemic uncertainties alone. This distinction is useful for deciding whether additional research is worth the cost that it would entail, but is not as important for distinguishing between different architectural or design alternatives. Therefore, for purposes of the RIDM process, we speak only of uncertainties in the broad sense and do not distinguish between their aleatory and epistemic parts. However, the analyst always has the option of keeping aleatory and epistemic uncertainties separate from one another if he or she desires to do so, and in CRM where mitigation options are considered, this separation can be essential.

Further arguments about the relative advantages of combining aleatory and epistemic uncertainties versus keeping them separate may be found in [28].