7. Further issues of cost-effectiveness analysis
7.3 SAMPLING ERROR AND UNCERTAINTY ABOUT ESTIMATES
First we look at sampling error for a single variable, i.e., costs, and then look at sampling error when both costs and effects are involved.
7.3.1 Sampling Error and Statistical Theory
We begin the analysis by summarizing basic statistical theory regarding dis-tributions of values for a single variable (refer to any introductory text, such as Weiers (1998)). Any variable’s distribution of values can be sum-marized by two measures: first, a measure of central tendency, and second a measure of dispersion around that central tendency. We will focus on the average (mean) as the measure of central tendency and the variance as a measure of dispersion. To calculate the mean for a variable X, denoted X––, one sums all the values Xiand divides by the number of observations N:
Mean: X––
(7.3)
The variance is defined as the average of the squared residuals (deviations of a value from the mean):
X 兺Xi
N
Further issues of cost-effectiveness analysis 173
Variance: (7.4) Note that in samples (for statistical reasons) the average of the squared deviations for the variance divides the sum by N ⫺1 (and not just N as with the mean). In order to express the measure of dispersion in units that are comparable to the mean, statisticians often replace the variance by its square root, called the standard deviation:
Standard deviation⫽兹 (Variance) (7.5) Table 7.3 gives a simple illustration of how the summary measures are cal-culated for two interventions, the experimental and the control, for a pair of three-person groups. For convenience the individuals are ranked from lowest cost to highest cost. From Table 7.3, one can see why the residuals need to be squared in equation (7.2). The sum of deviations from the mean (as shown in column 3) is always equal to zero. Its value would be the same no matter the shape of the distribution. Table 7.3 shows that the experimen-tal group has the lower average costs, but it also has the higher dispersion.
What this means is that, although the experimental treatment seems to have lower costs, there is a great deal more uncertainty about this result. Perhaps
兺(Xi⫺ X )2
N ⫺ 1
174 CEA and CBA
Table 7.3: Programs with costs that vary by individuals
Program Costs per person ($) Residual Residual squared (Ci) (Ci⫺ C––) (Ci⫺ C––)2 Control
Person 1 100 ⫺120 14 440
Person 2 200 0⫺20 400
Person 3 360 ⫺140 19 600
Total 660 ⫺000 34 440
Mean C–– 220
Standard deviation 131 Experimental
Person 1 40 ⫺160 25 600
Person 2 100 ⫺100 10 000
Person 3 460 ⫺260 67 600
Total 600 ⫺000 103 200
Mean C–– 200
Standard deviation 227
Source: Created by the author
X––
there are more people like person 3 around, whose costs of $460 are over twice the average.
The way that information about the dispersion is factored into judg-ments about the average is through the concept of a confidence interval around the mean. An interval is just a range of consecutive values. What makes an interval for a mean a confidence interval is that one has to be able to tell the probability that the mean lies within that range. For example, one may know that there is a 95% probability that the mean is within this range.
Obviously, the greater the probability, the less uncertainty one has about a particular mean’s true value. The confidence interval is determined by:
Confidence interval for the mean: X–– Z (standard error) (7.6) where Z is a number, typically between 1 and 3, and the standard error can often be assumed to be equal to the standard deviation divided by the square root of the sample size:
Standard error: Standard deviation / N (7.7) The sample size is on the denominator because, the larger the sample we take, the more reliable will be the estimate from the sample and the nar-rower the confidence interval.
The number Z is determined by two considerations: (1) the error that one is willing to accept and (2) the assumed shape of the distribution of values for likely values for X. If one is willing to accept a 5% chance of accepting a wrong value for the mean (one has 95% confidence in the value of an esti-mate), and if the distribution of mean values is assumed to be what statis-ticians call ‘normal’ (i.e., a bell-shaped, symmetric curve), then the corresponding Z value (now properly called a ‘Z-score’) would be 1.96.
For Z values of 1.96, with means and standard deviations as in Table 7.3, and a sample size of N3 (which means that its square root is 1.73), the confidence interval for costs for the experimental group would be
$200 1.96 (131.21) or between $57 and $457. With such a wide interval, the mean of $200 is very unreliable, as the value could with 95% confidence even be as high as $457 (or even negative which is not economically pos-sible). One would not have much confidence that the experimental group had costs $20 less than the control group. (If one used the appropriate sta-tistical formula, one could find the exact probability of observing no differ-ence in the two sample means (the answer is around 90%; it is very probable).)
Before leaving this section, we can use the confidence interval concept to define precisely the statement made in the last chapter (in the context of Further issues of cost-effectiveness analysis 175
regression coefficients in Table 6.7) that some estimates were unlikely to have occurred by chance, i.e., were ‘statistically significant’. If one forms a 95%
confidence interval around finding a zero estimate, say the range is from 20 to20, and one actually comes up with an estimate of 25, then one can say with 95% certainty that the estimate of 25 is unlikely to have been 0. The value 25 is not in the confidence interval around zero and this is what makes this particular estimate statistically, significantly different from zero.
7.3.2 Sampling Error and Cost-Effectiveness Ratios
We have just seen that a costs estimate may have a confidence interval around it indicating a range of uncertainty about its precise value. The same logic applies to any estimate of effects. It will also have a confidence interval around its mean value. This means that when we try to form the ratio C / E we will come up with a whole set of alternatives to the average estimate C––/E––
which was the outcome measure we were dealing with in the last chapter.
Following O’Brien et al. (1994), let us call the lower and upper bounds of a costs confidence interval CU and CL, respectively, and the lower and upper bounds of an effects confidence interval EUand ELrespectively. Then the extreme bounds for the cost-effective ratio can be formed from the extreme values of the two intervals as follows: the highest value for the ratio would pair the highest costs estimate CUwith the lowest effects estimate EL to form CU/ EL; and the lowest value for the ratio would pair the lowest costs estimate CLwith the highest effects estimate EUto form CL/ EU. The result is that there is now a range of ratios for any intervention in which C–– / E–– lies, from CL/ EUto CU/ EL.
In the same way as for the average ratio for a single intervention, one can construct the lower and upper bounds for the incremental cost-effectiveness ratio for the outcomes of an experimental group T over a control group C.
The incremental ratio for costs and effects would be defined as: C/ E
(CTCC ) / (ETEC). Using this definition, the incremental ratio for average costs and effects would be: C––/ E––; the lower bound ratio would be CL/ EU; and the upper bound ratio would be CL/ EU. Figure 7.1, based on O’Brien et al.’s Figure 2, depicts these three ratios graphically.
The result of there being sampling variation is that the incremental cost-effectiveness ratio is unlikely to be a single, unique value. The average ratio given as the slope of 0b in Figure 7.1 is just one possibility. The ratio could be, plausibly, any value between the slope of 0a and 0c. This means that, even if one does come up with the result that the average ratio is lower than the cut-off ratio that emerges from the marginal program in a past cost-effectiveness allocation, the upper ratio may not be lower. Uncertainty in estimation leads to outcome indecisiveness.
176 CEA and CBA
It is important to understand that the lower and upper bounds for the ratios represented in Figure 7.1 need not have the same probability asso-ciated with them as the bounds for a particular probability confidence inter-val applied to incremental costs or effects separately. For example, if the lower bound of a 95% confidence interval for incremental costs is combined with the upper bound of a 95% confidence interval for incremental effects, the ratio of these two values that forms the lower bound ratio for the incre-mental cost-effectiveness ratio need not be the least value for a 95% confi-dence interval for ratios. In particular, as O’Brien et al. emphasize, if the two Further issues of cost-effectiveness analysis 177
Note: When there is sampling variation in both costs and effects, there is not just a single, average ratio C––/ E–– to consider. One can combine the lower bound value for the difference in costs with an upper bound value for the difference in effects to obtain a lower bound for the incremental ratio. Alternatively, one can pair the upper bound value for the difference in costs with a lower bound value for the difference in effects to produce an upper bound for the incremental ratio. The diagram shows that the incremental cost-effectiveness ratio could be any value between the slope of the line 0a and the slope of line 0c. The average ratio given as the slope of line 0b is just one possibility.
Figure 7.1: Incremental cost-effectiveness ratios with sampling variation in costs and effects