Comparison of four methods of computing confidence
intervals for a proportion
Gilles Guillot
June 28, 2017, 12:25:11
Contents
Summary 1
Results 2
Comparing confidence interval bounds returned by the various methods 6
Appendix: R code 17
Summary
We compare four methods of computing confidence intervals for a proportion. These four methods are • the Normal approximation method,
• the Wilson score method,
• the Wislon score method with continuity correction • the Clopper-Pearson method.
They are described on the Wikipedia pageBinomial proportion confidence interval.
We estimate the coverage for the various methods listed above for an intended nominal coverage ofα= 0.05 for various sample sizesnand various probabilitiesp. We do so with Monte Carlo simulations with 10ˆ{6} replicates per combination of parameters (n, p). The Normal approximation fails to achieve the nominal coverage for small sample size, more so when the truepis close to 0 (or 1). The three other methods are more accurate. We note that the Wilson score method with continuity correction and the Clopper-Pearson method tend to be a bit conservative for smallnorp.
Results
0
200
400
600
800
1000
0.0
0.2
0.4
0.6
0.8
1.0
Normal approximation
Sample size
Estimated co
v
er
age
p= 0.001
p= 0.01
p= 0.02
p= 0.05
p= 0.1
p= 0.2
0
200
400
600
800
1000
0.0
0.2
0.4
0.6
0.8
1.0
Wilson score method (no correction)
Sample size
Estimated co
v
er
age
p= 0.001
p= 0.01
p= 0.02
p= 0.05
p= 0.1
p= 0.2
0
200
400
600
800
1000
0.0
0.2
0.4
0.6
0.8
1.0
Wilson score with continuity correction
Sample size
Estimated co
v
er
age
p= 0.001
p= 0.01
p= 0.02
p= 0.05
p= 0.1
p= 0.2
0
200
400
600
800
1000
0.0
0.2
0.4
0.6
0.8
1.0
Clopper−Pearson method
Sample size
Estimated co
v
er
age
p= 0.001
p= 0.01
p= 0.02
p= 0.05
p= 0.1
p= 0.2
Comparing confidence interval bounds returned by the various
methods
Normal
Sample size= 10 , p= 0.001
Width CI w Frequency 0.0 0.1 0.2 0.3 0.4 0.5 0 30000Wilson
Width CI w Frequency 0.30 0.35 0.40 0.45 0 30000Wilson cc
Width CI w Frequency 0.30 0.35 0.40 0.45 0 30000CP
Width CI w Frequency 0.30 0.35 0.40 0.45 0.50 0 30000Normal
Sample size= 10 , p= 0.01
Width CI w Frequency 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0 30000Wilson
Width CI w Frequency 0.30 0.35 0.40 0.45 0.50 0 30000Wilson cc
Width CI w Frequency 0.30 0.35 0.40 0.45 0.50 0 30000CP
Width CI w Frequency 0.30 0.40 0.50 0.60 0 30000Normal
Sample size= 10 , p= 0.05
Width CI w Frequency 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0 20000Wilson
Width CI w Frequency 0.30 0.35 0.40 0.45 0.50 0 20000Wilson cc
Width CI w Frequency 0.30 0.35 0.40 0.45 0.50 0 20000CP
Width CI w Frequency 0.30 0.40 0.50 0.60 0 20000Normal
Sample size= 10 , p= 0.1
Width CI w Frequency 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0 15000Wilson
Width CI w Frequency 0.30 0.35 0.40 0.45 0.50 0 15000Wilson cc
Width CI w Frequency 0.30 0.35 0.40 0.45 0.50 0 15000CP
Width CI w Frequency 0.30 0.40 0.50 0.60 0 15000Normal
Sample size= 20 , p= 0.001
Width CI w Frequency 0.00 0.10 0.20 0 30000Wilson
Width CI w Frequency 0.16 0.20 0.24 0 30000Wilson cc
Width CI w Frequency 0.16 0.20 0.24 0 30000CP
Width CI w Frequency 0.16 0.20 0.24 0.28 0 30000Normal
Sample size= 20 , p= 0.01
Width CI w Frequency 0.00 0.10 0.20 0.30 0 30000Wilson
Width CI w Frequency 0.20 0.25 0.30 0 30000Wilson cc
Width CI w Frequency 0.20 0.25 0.30 0 30000CP
Width CI w Frequency 0.20 0.25 0.30 0.35 0 30000Normal
Sample size= 20 , p= 0.05
Width CI w Frequency 0.0 0.1 0.2 0.3 0.4 0 10000Wilson
Width CI w Frequency 0.20 0.25 0.30 0.35 0 10000Wilson cc
Width CI w Frequency 0.20 0.25 0.30 0.35 0 10000CP
Width CI w Frequency 0.15 0.25 0.35 0 10000Normal
Sample size= 20 , p= 0.1
Width CI w Frequency 0.0 0.1 0.2 0.3 0.4 0 10000Wilson
Width CI w Frequency 0.20 0.25 0.30 0.35 0.40 0 10000Wilson cc
Width CI w Frequency 0.20 0.25 0.30 0.35 0.40 0 10000CP
Width CI w Frequency 0.15 0.25 0.35 0.45 0 10000Normal
Sample size= 50 , p= 0.001
Width CI w Frequency 0.00 0.04 0.08 0 30000Wilson
Width CI w Frequency 0.07 0.09 0.11 0 30000Wilson cc
Width CI w Frequency 0.07 0.09 0.11 0 30000CP
Width CI w Frequency 0.07 0.09 0.11 0.13 0 30000Normal
Sample size= 50 , p= 0.01
Width CI w Frequency 0.00 0.05 0.10 0.15 0.20 0 20000Wilson
Width CI w Frequency 0.08 0.12 0.16 0.20 0 20000Wilson cc
Width CI w Frequency 0.08 0.12 0.16 0.20 0 20000CP
Width CI w Frequency 0.08 0.12 0.16 0.20 0 20000Normal
Sample size= 50 , p= 0.05
Width CI w Frequency 0.00 0.05 0.10 0.15 0.20 0 8000Wilson
Width CI w Frequency 0.10 0.15 0.20 0 8000Wilson cc
Width CI w Frequency 0.10 0.15 0.20 0 8000CP
Width CI w Frequency 0.10 0.15 0.20 0.25 0 8000Normal
Sample size= 50 , p= 0.1
Width CI w Frequency 0.00 0.05 0.10 0.15 0.20 0.25 0 6000Wilson
Width CI w Frequency 0.10 0.15 0.20 0.25 0 6000Wilson cc
Width CI w Frequency 0.10 0.15 0.20 0.25 0 6000CP
Width CI w Frequency 0.10 0.15 0.20 0.25 0 6000Normal
Sample size= 200 , p= 0.001
Width CI w Frequency 0.00 0.01 0.02 0.03 0.04 0 30000Wilson
Width CI w Frequency 0.020 0.030 0.040 0 30000Wilson cc
Width CI w Frequency 0.020 0.030 0.040 0 30000CP
Width CI w Frequency 0.020 0.030 0.040 0 30000Normal
Sample size= 200 , p= 0.01
Width CI w Frequency 0.00 0.02 0.04 0.06 0 8000Wilson
Width CI w Frequency 0.02 0.03 0.04 0.05 0.06 0 8000Wilson cc
Width CI w Frequency 0.02 0.03 0.04 0.05 0.06 0 8000CP
Width CI w Frequency 0.02 0.03 0.04 0.05 0.06 0.07 0 8000Normal
Sample size= 200 , p= 0.05
Width CI w Frequency 0.00 0.02 0.04 0.06 0.08 0 6000Wilson
Width CI w Frequency 0.02 0.04 0.06 0.08 0 3000Wilson cc
Width CI w Frequency 0.02 0.04 0.06 0.08 0 3000CP
Width CI w Frequency 0.02 0.04 0.06 0.08 0.10 0 3000Normal
Sample size= 200 , p= 0.1
Width CI w Frequency 0.04 0.06 0.08 0.10 0 3000Wilson
Width CI w Frequency 0.05 0.07 0.09 0.11 0 6000Wilson cc
Width CI w Frequency 0.05 0.07 0.09 0.11 0 6000CP
Width CI w Frequency 0.05 0.07 0.09 0.11 0 6000Normal
Sample size= 1000 , p= 0.001
Width CI w Frequency 0.000 0.004 0.008 0 10000Wilson
Width CI w Frequency 0.004 0.006 0.008 0.010 0 10000Wilson cc
Width CI w Frequency 0.004 0.006 0.008 0.010 0 10000CP
Width CI w Frequency 0.004 0.006 0.008 0.010 0 10000Normal
Sample size= 1000 , p= 0.01
Width CI w Frequency 0.005 0.010 0.015 0.020 0 3000Wilson
Width CI w Frequency 0.005 0.010 0.015 0.020 0 3000Wilson cc
Width CI w Frequency 0.005 0.010 0.015 0.020 0 3000CP
Width CI w Frequency 0.005 0.010 0.015 0.020 0 3000Normal
Sample size= 1000 , p= 0.05
Width CI w Frequency 0.020 0.025 0.030 0.035 0 3000Wilson
Width CI w Frequency 0.020 0.025 0.030 0.035 0 3000Wilson cc
Width CI w Frequency 0.020 0.025 0.030 0.035 0 3000CP
Width CI w Frequency 0.020 0.025 0.030 0.035 0 3000Normal
Sample size= 1000 , p= 0.1
Width CI w Frequency 0.030 0.034 0.038 0.042 0 3000Wilson
Width CI w Frequency 0.030 0.034 0.038 0.042 0 3000Wilson cc
Width CI w Frequency 0.030 0.034 0.038 0.042 0 3000CP
Width CI w Frequency 0.032 0.036 0.040 0.044 0 4000Appendix: R code
## en.wikipedia.org/wiki/Binomial_proportion_confidence_interval
## functions to compute CI for a proportion CI_prop = function(x ,
alpha = 0.05, # 1-(nominal coverage)
method=c("normal","Wilson","Wilson_cc","CP") )
{
CI_normal = CI_Wilson = CI_Wilson_cc = CI_CP = NULL
p.hat = x/size
if("normal" %in% method) {
z = qnorm(p=1-alpha/2)
delta = z* sqrt(p.hat*(1-p.hat)/size) CI_normal = cbind(p.hat-delta,p.hat+delta) }
if("Wilson" %in% method) {
z = qnorm(p=1-alpha/2)
delta = z* sqrt(p.hat*(1-p.hat)/size + z^2/(4*size^2))
CI_Wilson = cbind((p.hat + z^2/(2*size) - delta)/(1+z^2/size), (p.hat + z^2/(2*size) + delta)/(1+z^2/size)) }
if("Wilson_cc" %in% method) {
z = qnorm(p=1-alpha/2)
delta = z* sqrt(p.hat*(1-p.hat)/size + z^2/(4*size^2))
CI_Wilson_cc = cbind((p.hat + z^2/(2*size) - delta)/(1+z^2/size), ((p.hat + z^2/(2*size) + delta)/(1+z^2/size))) }
if("CP" %in% method) {
lb = qbeta(p=alpha/2,shape1=x , shape2 = size-x+1)
ub = qbeta(p = 1-alpha/2, shape1 = x+1 , shape2 = size-x) CI_CP = cbind(lb,ub)
}
return(list(CI_normal=CI_normal,
CI_Wilson=CI_Wilson,
CI_Wilson_cc=CI_Wilson_cc,
## function to estimate coverage of CI by Monte Carlo simulation ## Normal approximation method
coverage_normal = function(size, #sample size
prob, # actual proportion
rep, # number of Monte Carlo replicates alpha = 0.05 # 1-(nominal coverage)
) {
X = rbinom(size=size,prob=prob,n=rep) p.hat = X/size
z = qnorm(p=1-alpha/2)
delta = z* sqrt(p.hat*(1-p.hat)/size)
mean((prob > p.hat-delta) &
(prob < p.hat+delta )) }
## function to estimate coverage of CI by Monte Carlo simulation ## Wilson score method
coverage_Wilson = function(size, #sample size
prob, # actual proportion
rep, # number of Monte Carlo replicates alpha = 0.05 # 1-(nominal coverage)
) {
X = rbinom(size=size,prob=prob,n=rep) p.hat = X/size
z = qnorm(p=1-alpha/2)
delta = z* sqrt(p.hat*(1-p.hat)/size + z^2/(4*size^2))
mean((prob > (p.hat + z^2/(2*size) - delta)/(1+z^2/size)) & (prob < (p.hat + z^2/(2*size) + delta)/(1+z^2/size))) }
## function to estimate coverage of CI by Monte Carlo simulation ## Wilson score with continuity correction method
coverage_Wilson_cc = function(size, #sample size
prob, # actual proportion
rep, # number of Monte Carlo replicates alpha = 0.05 # 1-(nominal coverage)
{
X = rbinom(size=size,prob=prob,n=rep) p.hat = X/size
z = qnorm(p=1-alpha/2)
delta = z*sqrt(z^2 - 1/size + 4*size*p.hat*(1-p.hat) + (4*p.hat-2)) + 1
lb = (2*size*p.hat+z^2 - delta) / (2*(size+z^2)) subs = lb < 0
lb[lb < subs] = 0
delta = z*sqrt(z^2 - 1/size + 4*size*p.hat*(1-p.hat) - (4*p.hat-2)) + 1
ub = (2*size*p.hat+z^2 + delta) / (2*(size+z^2)) subs = ub > 1
ub[subs] = 1
mean((prob > lb) & (prob < ub)) }
## function to estimate coverage of CI by Monte Carlo simulation ## Clopper-Pearson method
coverage_CP = function(size, #sample size
prob, # actual proportion
rep, # number of Monte Carlo replicates alpha = 0.05 # 1-(nominal coverage)
) {
X = rbinom(size=size,prob=prob,n=rep)
# F = qf(p=alpha/2 , df1=2*X , df2=2*(size-X+1)) # lb = 1/(1+ (size-X+1) / (X*F))
# F = qf(p=1-alpha/2 , df1=2*(X+1), df2 = 2*(size-X) ) # ub = 1/(1+ (size-X) / ((X+1)*F) )
lb = qbeta(p=alpha/2,shape1=X , shape2 = size-X+1)
## lb2 = 1 - qbeta(p=1-alpha/2,shape2=X , shape1 = size-X+1) # just checking SAS formula p.7 OK ub = qbeta(p = 1-alpha/2, shape1 = X+1 , shape2 = size-X)
## ub2 = 1- qbeta(p = alpha/2, shape2 = X+1 , shape1 = size-X) # just checking SAS formula p.7 OK
mean((prob > lb) & (prob < ub))