5. THE KEY STEPS IN DECISION ANALYTICAL MODELLING
5.5 EXPLORING UNCERTAINTY IN A MODEL
Sensitivity analysis is vital in exploring the uncertainty of economic evaluation findings. The analysis aids in assessing the reliability of the study conclusions. Sensitivity analysis examines the robustness of the results. Results sensitive to a specific variable are explored to determine the degree of the sensitivity effect and if required, the model is appropriately revised (564).
89
5.5.1 Handling variability, uncertainty and heterogeneity
The results generated in an analytical model are subject to the influences of variability, uncertainty, and heterogeneity, and these must be handled accordingly to ensure that decision-makers are confident about the cost-effectiveness estimates (105, 511).
Variability (1st order uncertainty) represents the random variability in outcomes between similar patients (565). This variability, occasionally referred to as Monte Carlo uncertainty, does not provide any information and is negated by repeatedly running the model, thereby ensuring a stable estimate of the central tendency has been generated (566). Unlike variability, heterogeneity can explain in principle, to some degree, the differences between patients (i.e. differences in mortality between males and females) and does not represent a source of uncertainty (470).
Uncertainty is distinct from variability and heterogeneity. Uncertainty is further considered as parameter or model uncertainty. Parameter uncertainty relates to the uncertainty about the true numerical values of parameters used as inputs (e.g. transition probabilities, costs and health utilities) (508). This is often referred to as second order uncertainty to distinguish it from variability. Standard statistical methods would be used to represent the uncertainty of any estimate, but often does not give the full picture of the effects of joint uncertainty (105). This approach recognises that the data informing the parameter estimate follows a binomial distribution and thus, the standard error of the proportion can be obtained from the binomial distribution (Equation 4):
𝑠𝑒(𝑝̅) = √𝑝̅(1 − 𝑝̅)/𝑛 . (Eq. 4)
Where 𝑝̅ is the estimated proportion and 𝑛 is the sample size and 𝑠𝑒 is the standard error.
Model (or structural) uncertainty addresses the uncertainty occurring in the structure of the model and the assumptions that underpin it (502). The model-structure uncertainty also refers to the mathematical manner in which parameters are combined to estimate costs and/or effects. Model– process uncertainty arises from the collation of decisions applied to the model through the entire process of analysis (508). Model uncertainty is assessed with sensitivity analysis – running the model with alternative structural assumptions (105).
90
5.5.2 Evaluating parameter uncertainty
Briggs et al. proposed three main reasons to consider assessing uncertainty in a model:
(i) Models often combine input parameters in different ways including addition, multiplication and as power functions. This results in models that are nonlinear with regard to those parameters.
(ii) The possibility of uncertainty existing in the results of an analysis implies the possibility of making an incorrect decision, which imposes a cost in terms of benefits forgone. (iii) Policy changes have significant cost implications, with decision reversal being difficult or
not possible.
One-way (univariate) sensitivity analysis
In deterministic sensitivity analysis, parameters are varied manually to test the sensitivity of the model outcome to specific parameter changes (565). This approach examines one variable at a time. The ICER is recalculated after calculating the base-case scenario with only a justified change applied to a single parameter (Figure 11). The process may be repeated with different parameters (473). A second type of one-way analysis is the ‘threshold analysis’ where the input parameters are varied over a range to determine the level below or above which the conclusions of the study change i.e. the ‘threshold’ point where neither of the decisions are favoured over the other (567).
Figure 11. Illustrative one- way sensitivity analysis
The sensitivity of the ICER to different parameters is displayed. The numbers at the end of the bars reflect the range of values assessed in the sensitivity analysis [Source: Shelley, 2015 (568)].
91
Multi-way (multivariate) sensitivity analysis
Two-way analysis assesses two parameters that are common to the intervention being assessed simultaneously. A two-by-two matrix is developed reflecting the ICER for every potential combination for the variables; and the values that approximate a pre-determined willingness-to- pay (WTP) for a unit of effect are identified (473). The ICER is determined for a combination of three parameter estimates in a three-way analysis (473). In this technique, one of the parameters is held at a particular value and the combination of the remaining two parameters is assessed against a pre-determined WTP per unit of effect (473). The process is repeated according to the number of values that needs to be assessed for the first variable (473). This is represented graphically by Figure 12.
Figure 12. Illustrative example of multivariate (three-way) uncertainty analysis
The sensitivity of cost per HIV infection averted (HIA) to unit cost, protective effect and epidemic multiplier is shown per male
circumcision done. This is compared to the base value of $181 per HIA [Source: Kahn, 2006 (569)].
Probabilistic sensitivity analysis
Probabilistic sensitivity analysis (PSA), using Monte Carlo simulations, integrates the probability distribution of key variables and generate a distribution of the anticipated results (570, 571). The PSA is the preferred method of assessing parameter uncertainty as all variables are estimates of the sample mean and sampling error gauged from the best available evidence (511, 565). The PSA is executed by running the model several thousand times (iterations), with the parameter values varied across specified distributions (e.g. for costs and effects) until a distribution has been constructed and confidence intervals can be assessed (502). The information derived from the PSA can graphically represented as cost-effectiveness acceptability curves (CEAC), which demonstrate
92 the probability that an intervention is cost-effective at an assumed maximum WTP for health gains (572). Further, in the event that a model has been derived from a single dataset, bootstrapping can be applied to the model uncertainty by repeatedly re-estimating the model outcomes using randomly drawn subsamples drawn with replacement from the full sample (502). The first two stages of the analysis involve the assigning of distributions to represent the uncertainty followed by the propagation of the uncertainty.
Assigning the distributions
The type of parameter being considered dictates the choice and fitting distribution of the model parameters (Table 8). The idea is to match what is known about the model input with the characteristics of the distribution. Often this requires the use of the standard distributional assumptions employed to estimate confidence intervals. The selection of distributions for probability parameters are governed by two rules regarding the probabilities. The probabilities are limited to a value between zero and one and the probabilities of mutually exclusive events must sum one. However, a HIV-positive state can transition into AIDS, death or remain in the current state. The stages represent a multivariate generalization of the beta distribution with parameters equalling the number of categories in the multinomial distribution (470).
Propagating the uncertainty
The parameters are assumed to be assigned specific probability distributions in a second order Monte Carlo simulation (573). When conducting sensitivity analysis, parameter values are drawn based on the distributions to calculate the estimates required (573). As a stable estimate of the mean is required, the simulation is repeated a large number of times (≥ 1000) to obtain a distribution of the expected outcomes. This is normally achieved by non-parametric bootstrapping, a resampling procedure that randomly selects samples from the original data set with replacements (471). The repetition of this process a large number of times generates a vector of bootstrap replicates which represents the empirical estimate of the statistic’s sampling distribution allows for the generation of a confidence interval for the analyses (573).
93
Table 8. Fitting parameter distributions
Distributions are assigned to match the characteristics of the parameter [Source: Briggs, 2006 (470)].
Distribution Parameters Values Skewness Uses Information Equations Graphic
Normal 2 parameters mean & SD
Continuous unbound
Symmetrical Mean, median & mode are equal
Log odds ratio Central limit theorem1 Log-normal 2 parameters mean & SD ≥ 0 continuous (+) skew; Median to the left of mean; variance dictates skewness Resource use, relative risk Natural log of value generates a normal distribution Gamma 2 parameters Shape & scale ≥ 0 continuous Flexible Symmetrical or (+) skew Cost parameters; mean rate of events
Mean & SD data can be converted to shape an scale
E(θ) = αβ = μ Var (θ) = αβ2 = s2
Beta 3 parameters
α, β & scale ≥ 0 Flexible Symmetrical or (+) or (-) skew
Probabilities, utility
α & β equal the successes and failures in a sample of size n = α + β2 𝐸(𝜃) = 𝛼 𝛼 + 𝛽 𝑉𝑎𝑟(𝜃) = 𝛼𝛽 (𝛼 + 𝛽)2(𝛼 + 𝛽 + 1)
1Central limit theorem: the sampling distribution of the mean will be normally distributed irrespective of the underlying distribution of the data with sufficient sample size.
94