• No results found

Confidence intervals are used when you are interested in the uncertainty around a sample estimate such as the mean.

Confidence intervals can be one-sided or two-sided, which means that you either quote one confidence limit from one end of the data or both limits from either end. The confidence interval values are called the lower confidence limit/bound and the upper confidence limit/bound. Therefore, for a one-sided confidence interval you would quote either the lower bound or the upper bound, whereas for a two-sided confidence interval you would quote both bounds The decision about calculating a one-sided or two-sided interval should be decided during the experimental design stage, as your power or sample size calculation would have been carried out using this information. It would be incorrect to quote just one bound from a two-sided confidence interval calcu-lation as the confidence level associated with the statement would be wrong.

The reason you may use a one-sided confidence interval is if you are only interested in one end of the data. For instance, you only care about how bad a detector is but not really how good it can be, in which case you would only want a lower bound.

Figure 5-1 shows the risk, which is the significance level, associated with both a one-sided and two-sided 90% confidence interval for continuous data.

Figure 5-1. Division of risk for two-sided and one-sided confidence intervals

Translating Statistics to Make Decisions 105

For a one-sided interval the risk is put at one end of the distribution with there being no information about the other end. Looking at Figure 5-1, 10% of the risk is at the lower end, so really the upper limit is infinity as there is no information at the upper end, if the risk was all at the higher end the lower limit would be minus infinity.

Thinking about multiple journey times to a location that involves heavily con-gested roads, you may not be interested in how quick the journey could be, but how long it may take. With a one-sided upper interval you would be able to say the average journey time will be no greater than z hours, but you can’t make any kind of statement about how quick the journey could be.

For a two-sided interval the risk is split equally at each end, as you can see in Figure 5-1 there is 5% risk at each end. Therefore the lower bound here would be a smaller number than the lower bound of the one-sided interval.

With a two-sided interval you could state that the average journey time could be as long as x hours but also could be as quick as y hours. However the upper bound of x hours here would be a larger value than that of the z hours in the one-sided interval, as you are not as sure about each end in the two-sided case compared to the one-sided case.

The following sections look at calculating confidence intervals for continuous and binary data, with the information about sides also applying to binary data.

The only difference is that the limits at the ends that are not of interest, in the one-sided interval case, are no longer minus infinity and infinity, but are 0 and 1 as binary data is bounded.

Continuous Data

To calculate a confidence interval for the mean of continuous data that approx-imately follows a normal distribution the following information is required:

• The mean of the sample.

• The variability of the sample: the standard deviation.

• The number of measurements in the sample: the sample size.

• The confidence level required or the level of risk that is acceptable.

• The interval of interest: one-sided or two-sided bounds.

Chapter 5 | Measuring Uncertainty

106

Example 5.1 shows not only how to manually calculate a confidence interval in R to understand the steps involved, but also shows the shortcut using the Rmisc package. This shortcut only works for two-sided intervals unless you adjust the confidence level to compensate.

After creating the data, jump straight to “# Load library” if you are only inter-ested in using the shortcut.

Part of the error calculation uses a critical value from the t distribution, which is based on the specified confidence level, the sample size, and whether the test is one-sided or two-sided.

EXAMPLE 5.1

Calculate a 90% two-sided confidence interval and a 90% one-sided lower confidence bound for the mean on the following data.

# Create some data

data7 = c(26.33, 27.31, 27.38, 26.63, 26.87, 26.67, 28.36, 28.52, 26.91, 28.90, 27.99, 27.17, 28.32, 26.93, 26.93, 26.65, 27.73, 26.93)

# Calculate the mean, standard deviation, and sample size for the data

# Assign the confidence level

# Calculate the lower and upper two-sided confidence intervals

# Calculate the one-sided lower bound

lower.2s.CI = x - error2; upper.2s.CI = x + error2 lower.1s.CI = x - error1

lower.2s.CI; upper.2s.CI

Translating Statistics to Make Decisions 107

# Shortcut to calculate the two-sided interval CI(data7, ci = 0.90)

upper mean lower 27.67382 27.36278 27.05173

# Shortcut to calculate the one-sided interval with amendment CI(data7, ci = 0.80)

upper mean lower 27.60119 27.36278 27.12437

The two-sided confidence interval is (27.05, 27.67) or 27.36 ± 0.31 due to symmetry, and the one-sided lower bound is 27.12.

Note: If using the shortcut to calculate a one-sided interval you will need to double the significance level, here from 0.1 to 0.2, and make sure you only quote one side and not both.

In Example 5-1, if both the lower and upper one-sided bound had been calcu-lated and incorrectly quoted as a two-sided interval, the values would be the same as calculating an 80% two-sided confidence interval, as 10% of risk would be at each end of the distribution totaling 20% risk overall.

Translating the output from a confidence interval is important as you need to remember the interval is around an estimate and not a proportion of the data. Using Example 5.1 you could state that you are “90% confident that the true mean lies within 27.05 and 27.67,” or as the interval is symmetrical you could say that you are “90% confident that the true mean is 27.36 (± 0.31),”

using an appropriate number of decimal points dependent on the required degree of accuracy.

The confidence level used will have a strong effect on the width of the con-fidence interval. For example, if you need a more precise interval around the mean then you could accept a lower confidence level. However this would result in a higher risk of being misled and that would result in a higher risk of the true mean being outside those interval values. If you need a high con-fidence level that would result in a lower risk of being misled, this would give a less precise interval.

Chapter 5 | Measuring Uncertainty

108

The sample size used also will have a strong effect, as a larger sample size will provide a more precise interval whereas a smaller sample size will give a less precise interval.

In addition, the variation in the data also will have an effect with noisier data giving a larger and therefore less precise interval. Note that the variation of the data cannot be controlled.

Example 5.2 uses the same data as Example 5.1, data7, but this time looks at using a lower confidence level of 70%, then using a larger sample size of 25 to highlight the earlier points.

EXAMPLE 5.2

Calculate a 70% two-sided confidence interval for the original data, data7, and then a 90% two-sided interval for the following data, which has a larger sample size.

# Calculate the two-sided confidence interval at the 70% confidence level CI(data7, ci = 0.70)

upper mean lower 27.55392 27.36278 27.17163

# Create the larger sized data

data8 = c(data7, 26.01, 28.33, 26.62, 26.99, 27.48, 27.74, 27.89)

# Calculate the two-sided confidence interval for the larger data CI(data8, ci = 0.90)

upper mean lower 27.60178 27.34360 27.08542

The original confidence interval with a sample size of 18 and a confidence level of 90% was ±0.31, decreasing the confidence level to 70% reduces that to ±0.19;

increasing the sample size to 25 with 90% confidence level (while maintaining a similar mean and variation) reduces that to ±0.26.

The calculations in Example 5-2 can be used to determine confidence inter-vals if one of the three assumptions in the following is satisfied,

• The data approximately follows a normal distribution.

• The data has a reasonable sample size and a symmetrical distribution.

• The data has a large sample size.

If none of these three points can be satisfied, one option is to use nonparametric confidence intervals however they can be quite complicated to compute.

Although they do have fewer assumptions, they will give a less precise answer.

Translating Statistics to Make Decisions 109

A better option to try first would be to transform the data using one of the transformations listed in Chapter 4. The key to remember when doing this is to back transform the information at the end to give the correct confidence interval. It is also worth noting that the interval will no longer be symmetrical around the arithmetic mean but it will be symmetrical around the geometric mean, which is also mentioned in Chapter 4.

Example 5.3 looks at an example where the data needs to be transformed for a confidence interval to be calculated.

EXAMPLE 5.3

Calculate a 95% two-sided confidence interval for the following skewed data and compare that to a confidence interval ignoring the normality assumption requirement.

# Create skewed data

# Check normality – output plot ommited qqnorm(data10); qqline(data10)

# Calculate the two-sided confidence interval for the transformed data ci = CI(data10, ci = 0.95); ci

upper mean lower 1.1183646 1.0246775 0.9309905

# Back-transform values to original scale 10^ci

upper mean lower 13.133019 10.584675 8.530814

# Calculate the incorrect two-sided confidence interval for the skewed data

CI(data9, ci = 0.95)

upper mean lower 15.189851 12.420000 9.650149

The back-transformed confidence interval is (8.53, 13.13), which would be a lower bound of 10.58 (- 2.05) and an upper bound of 10.58 (+2.55), which is clearly not symmetrical due to the skewed distribution.

If you compare that to the incorrectly calculated confidence interval of (9.65, 15.19), which is 12.42 (±2.77) you can see how different the two answers are and how the incorrect confidence interval could be misleading.

Chapter 5 | Measuring Uncertainty

110

If you need to calculate confidence intervals in Excel use the following functions for two-sided and one-sided confidence intervals, respectively:

= AVERAGE(data) – T.INV.2T(alpha, (COUNT(data) – 1)) * STDEV.S(data) / SQRT(COUNT(data)).

Where data is the column of data you are using and where alpha is the significance level you are interested in, remember significance is one minus confidence. Also don’t forget to repeat the line above for the upper confidence limit by replacing the first minus sign with a plus sign.

= AVERAGE(data) – T.INV(alpha, (COUNT(data) – 1)) * STDEV.S(data) / SQRT(COUNT(data))

Again chose the sign to match which one-sided confidence limit you require, minus for lower and plus for upper.

Binary Data

To calculate a confidence interval for a proportion in binary data, known as either a binary confidence interval or a binomial confidence interval, the following information is required,

• The number of successes or equivalent metric of interest.

• The number of trials in the sample: the sample size.

• The confidence level required or the level of risk that is acceptable.

• The interval of interest: one-sided or two-sided bounds.

Example 5.4 shows not only how to manually calculate a binary confidence interval in R to understand the steps involved, but also shows the shortcut using the Hmisc package. This shortcut only works for two-sided intervals unless you adjust the confidence level to compensate.

Jump straight to “# Load library” if you are only interested in the shortcut.

EXAMPLE 5.4

Calculate a 90% two-sided confidence interval and a one-sided upper confidence bound for 20 successes out of 25 trials.

# Assign the successes, trials, and significance level x = 20; n = 25; alpha = 0.10

# Assess the probability of success x/n

[1] 0.8

Translating Statistics to Make Decisions 111

# Assign degrees of freedom for the critical values

# from the exact binomial method df1l = 2*(n - x + 1); df2l = 2*x df1u = df2l + 2; df2u = df1l - 2

# Calculate the two-sided confidence interval

lci = ifelse(x > 0, x / (x + qf(1 - alpha/2, df1l, df2l) *

# Calculate the one-sided upper bound – replace alpha/2 with alpha uci1 = ifelse(x < n, ((x + 1) * qf(1 - alpha, df1u, df2u)) /

# Shortcut to calculate the two-sided interval

binconf(x = 20, n = 25, alpha = 0.1, method = "exact") PointEst Lower Upper

0.8 0.6245949 0.9177091

# Shortcut to calculate the one-sided interval with amendment binconf(x = 20, n = 25, alpha = 0.2, method = "exact") PointEst Lower Upper

0.8 0.6603411 0.8993822

The two-sided confidence interval is (0.625, 0.918) or 62.5% to 91.8%. Note that the probability of success was 80% that would make the intervals minus 17.5% and plus 11.8%, which is not symmetrical due to the probability of success being bound by 100%. The one-sided upper bound is 0.899 or 89.9% which is closer to 80% due to the risk placement.

Note: If using the shortcut to calculate a one-sided interval you will need to double the significance level, here from 0.1 to 0.2, and make sure you only quote one side and not both.

Translating the output from a binary confidence interval is important as you need to remember the interval is around an estimate and not a proportion of the sample. Using Example 5.4 you could state that you are “90% confident that the true success rate lies within 62.5% and 91.8%.” You cannot state the sample success rate ± a value as the confidence limits are not symmetrical due

Chapter 5 | Measuring Uncertainty

112

to the bounding of 0 and 1. For the one-sided intervals you could state that you are “90% confident that the true success rate is less than 89.9%.”

As binary data has far less “detail” than continuous data, the result is 0 or 1 as opposed to being any decimal place between two values. A much larger sample size will be required for a more precise interval. As with the continu-ous data examples, the confidence level and the sample size will have a large effect on the precision of the interval.

Example 5.5 uses the same data as Example 5.4 but this time looks at using a lower confidence level of 70%, then using a much smaller sample size of 5 trials with 4 successes.

EXAMPLE 5.5

Calculate a 70% two-sided confidence interval from the values in the previous example, then a 90% two-sided interval for 4 successes out of 5 trials, which is a much smaller sample size.

# Calculate the confidence interval at 70% confidence binconf(x = 20, n = 25, alpha = 0.3, method = "exact") PointEst Lower Upper

0.8 0.6838798 0.8856574

# Calculate the confidence interval for the smaller sample size binconf(x = 4, n = 5, alpha = 0.1, method = "exact")

PointEst Lower Upper 0.8 0.3425917 0.9897938

The original confidence interval width with a sample size of 25 and a confidence level of 90% was roughly 29.3% (91.8% - 62.5%), decreasing the confidence level to 70%

reduces that width to roughly 20.2% (88.6% - 68.4%); decreasing the sample size to 5 with a confidence level of 90% (while maintaining the same probability of success) increases that width to 64.7% (99.0% - 34.3%).

There are many methods for calculating binary confidence intervals, but argu-ably the three most common ones are the asymptotic normal interval or Wald interval, the Wilson interval, and the exact method, also known as the Clopper–Pearson interval.

The asymptotic normal interval is a normal approximation for binary data, and as such has some strong assumptions. As it is based on the normal distribution the interval will be symmetrical; it could go above 1 and below 0. It should only be used with large sample sizes and when the probability of success is not very small or very large, otherwise the results will be misleading.

The Wilson interval is more robust to deviations from the normal distribution than the asymptotic normal interval, and the intervals are not symmetrical so

Translating Statistics to Make Decisions 113

will converge at 0 or 1. It is easier to compute than the exact method and may be preferable with a medium to large sample size.

The exact method, which was used in the earlier examples, is the recom-mended method to use with small or medium sample sizes. The equations however are complicated to explain and they can use either the F distribution or the beta distribution to calculate the binomial cumulative density function (cdf), as such that will be left to personal investigation if you are interested.

This method also can cope if the probability of success is 0 or 1 however it will produce conservative results meaning the confidence intervals will be wider than the other methods.

If you need to calculate binary confidence intervals in Excel, use the following functions for the lower and upper bounds for the two-sided interval, this uses the exact method:

= IF(x = 0, IF(x = n, (alpha/2)^(1/n), BETA.INV(alpha/2, x, n – x + 1))).

=IF(x = n, 1, IF(x = 0, 1 – (alpha/2)^(1/n), BETA.INV(1 – alpha/2, x + 1, n – x))).

Where x is the number of successes, n is the number of samples/trials, and alpha is the significance level, which is one minus the confidence level.

For the one-sided intervals use the following functions again this uses the exact method:

= IF(x = 0, IF(x = n, alpha^(1/n), BETA.INV(alpha, x, n – x + 1))).

=IF(x = n, 1, IF(x = 0, 1 – (alpha)^(1/n), BETA.INV(1 – alpha, x + 1, n – x))).

Make sure you chose only one of the functions from the two above, either the top line for the one-sided lower bound or the lower for the one-sided upper bound.