Sensitivity Analysis 6

,

~, , ( max )

(

* e₁ E _|1 E_θ| u e₁ x aθ

u _x

e a

x ′′

= ;

but after the formal analysis of e₁is completed, it may be concluded without any formal analysis at all that e₂is not as good ase₁and adopt e₁.

That behavior is perfectly consistent with the principle of choice described in Sections 2 and 3 and readdressed in Section 6. Before making a formal analysis ofe₂, the decision maker can think of the unknown quantity u*(e₂)as a random variablev~. If the number v, resulting from a formal analysis was known, was greater than the known numberu*(e₁), e₂would be adopted rathere₁and the value of the information that

v v =

~ could be measured by the differencev−u*(e₁)in the decision maker’s utility which results from this change of choice. If on the contrary v were less thanu*(e₁), the decision maker would adhere to the original choice ofe₁and the information would have been worthless.

In other words, the random variable can be defined as

=

− *( )}

,~ 0

max{ v u e₁ value of information regarding e₂,

and before expounding such information at the cost of making a formal analysis ofe₂the decision maker may prefer to compute its expected value by assigning a probability measure tov~and then expecting with respect to this measure. If the expected value is less that the cost, the decision maker will quite rationally decide to use e₁without formally evaluatingv=u*(e₂). Operationally one usually does not formally compute either the value or the cost of information one₂: these are subjectively assessed. The computations could be formalized, of course, but ultimately direct subjective assessments must be used if the decision maker is to avoid an infinite regress.

Before closing this section and the subject of incomplete analysis, it should be said that completely formal analysis and completely intuitive analysis are not the only possible methods of determining a utility such asu*(e₂). In many instances it is possible to make a partial analysis in order to gain some insight but at a less prohibitive cost than a full analysis entails.

Section 13.7: Sensitivity Analysis

Sensitivity analysis is an essential element of decision analysis. The principle of sensitivity analysis is also directly applied to areas such as meta-analysis and cost-effectiveness analysis most readily but is not confined to such approaches. This section

6 See Petitti (2000), Hillier and Lieberman (2001), and Clemen (1996).

simply describes the overall purpose of sensitivity analysis and describes one-way and its expansion to n-way analysis as applied to decision analysis on a very general level.

Sensitivity analysis evaluates the stability of the conclusions of an analysis to assumptions made in the analysis. When a conclusion is shown to be invariant to the assumptions, confidence in the validity of the conclusions of the analysis is enhanced.

Such analysis also helps identify the most critical assumptions of the analysis (Petitti, 2000).

An implicit assumption of decision analysis is that the values of the probabilities and of the utility measure are the correct values for these variables. In one-way sensitivity analysis, the assumed values of each variable in the analysis are varied, one at a time, while the values of the other variables in the analysis remain fixed. When the assumed value of a variable affects the conclusion of the analysis, the analysis is said to be

“sensitive” to that variable. When the conclusion does not change, when the sensitivity analysis includes the values of the variables that are within a reasonable range, the analysis is said to be “insensitive” to that variable.

If an analysis is sensitive to the assumed value of a variable, the likelihood that the extreme value is the true value can be assessed qualitatively; perhaps, weighting the benefit of one strategy over the other under the extreme assumption.

An extension of the one-way sensitivity analysis is the threshold analysis. In such a case, the value of one variable is varied until the alternative decision strategies are to have equal outcomes, and there is no benefit of one alternative over the other in terms of estimated outcome. The threshold point is also called the break-even point at the decision is a “too-up”. That is, neither of the alternative decision options being compared is clearly favored over the other. Threshold analysis is especially useful when the intervention is being considered for use in groups that can be defined a priori based on the values of the variable that is the subject of the threshold analysis.

In two-way sensitivity analysis, the expected outcome is determined for every combination of estimates of two variables, while the values of all other variables in the analysis are held constant at baseline. It is usual to identify the pairs of values that equalize the expected or expected utility of the alternatives and to present the results of the analysis graphically. It is simpler to interpret the results of a two-way sensitivity analysis with the aid of graphs.

In n-way sensitivity analysis, the expected outcome is determined for every possible combination of every reasonable value of every variable. N-way sensitivity analysis is analogous to n-way regression and is seemingly difficult to interpret and is not discussed any further in this report.

It is usual to do one-way sensitivity analysis for each variable in the analysis. The highest and the lowest values within reasonable range of values are first substituted for the baseline estimate in the decision tree. If substitution of the highest or the lowest

value changes the conclusions, more values within the range are substituted to determine the range of values.

In an analysis with many probabilities, there are numerous combinations of two and three variables, and the computational burden of doing all possible two-way and three-way sensitivity analysis is large. For this reason, it is not usually feasible to conduct two-way sensitivity analysis for all possible combinations let alone n-way. The choice of variables for multiple way analysis requires considerable judgment, and therefore is not laid down by any fast and hard rules. However, the graphical approach applied in these instances also lends itself to the discussion of dominance considerations. In section 3, the notion of dominant alternatives was introduced and can be considered a type of sensitivity analysis.

A graphical approach, in this instance suggests when one alternative may supercede another and suggests possible implications.

Another area of emphasis for sensitivity analysis is the sensitivity with respect to the prior, when applying Bayesian methodology. These are typically performed when there is little imprecision un the loss because a standard choice of inference loss such as the quadratic was adopted, for instance. It is usually suggested to start with this case as (a) it is simplest and most thoroughly studied and (b) provide many insights for more general problems. It is also notably one of the most difficult elements to assess.

The overview of statistical decision theory provided in this report places has tried to maintain a balance between the classical and the Bayesian approach. In simple terms, the solution of a statistical decision problem proceeds by modeling a decision maker’s judgments by means of the loss function, probability model of the observation process and a prior, and then uses these to identify a ‘good’ decision rule. Thus, this presents a variety of reasons as to why a sensitivity analysis should be conducted – that is, checking the sensitivity of the decision rule (output) with respect to the model and decision maker’s judgments (inputs). In other words, the objective is to check the impact of the loss function, the prior and the model on the Bayes decision rule or Bayes alternative, and their posterior expected loss.