A COMPARISON OF STATISTICAL METHODS FOR
COST-EFFECTIVENESS ANALYSES THAT USE DATA
FROM CLUSTER RANDOMIZED TRIALS
M Gomes, E Ng, R Grieve, R Nixon, J Carpenter
and S Thompson
Health Economists’ Study Group meeting
Overview
•
CEA can be undertaken alongside CRTs
•
Unit of randomisation is the cluster, not the patient
•
Previous review found that most of CEA of CRTs ignore clustering
•
Methods are available but it is unclear which performs best
•
Aim is to evaluate relative performance of alternative methods
Methods for analyses
Seemingly Unrelated Regressions
• System of regression equations allowing error terms to be correlated
• Implemented with and without robust SE
Generalised Estimating Equations
• Independent estimating equations with robust SE.
Two-Stage Bootstrap
• Non-parametric. Involves resampling clusters as well as individuals
• Unless many clusters, it can overestimate the variance. Implemented using a shrinkage estimator (Davison & Hinkley 1997).
Multilevel Models
Data Generating Process
DGP is general and flexible
Cost-effectiveness data simulated in 2 stages, clusters then individuals
Can mimic a wide range of potential scenarios
varying parameters (e.g. No. of clusters, cluster size dist, ICCs, correlation) allow various distributions of the data at cluster and individual level
‘Fair’ to all methods
Performance measures
- Bias
- Root mean square error (rMSE)
- Confidence interval (CI) coverage and width
Varying parameters for base case and SA
Parameter
Rationale for consideration
Base case
Range for SA
N GEE, SUR, TSB rely on asymptotics 20 3 to 30
CoV GEE, SUR, TSB not tested for imbalance 0 0 to 1
𝑰𝑪𝑪𝒄𝒐𝒔𝒕 See if methods can handle higher ICCs 0.01 0 to 0.3
𝑰𝑪𝑪𝑸𝑨𝑳𝒀 As above 0.01 0 to 0.3
𝛈 GEE,SUR,MLM assume normal errors 0.2 0.25 to 3
- True incremental cost = 500
- True incremental QALY = 0.075
- True INB = £1000 (NICE threshold = £20 000/QALY)
Results for base case
(Parameter of interest – INB)
* MLM estimated by MCMC in WinBUGS produced similar results
SUR GEE 2SB MLM Without robust SE With robust SE With robust SE Without shrinkage correction With shrinkage correction ML* Mean (SE) bias -1.999 (2.45) -1.999 (2.45) -1.999 (2.45) -2.108 (2.45) -2.041 (2.45) -1.999 (2.45) rMSE 109.45 109.45 109.45 109.52 109.52 109.45 CI coverage 0.891 0.940 0.933 0.981 0.943 0.950 Mean CI width 353.6 423.7 417.7 539.1 427.5 440.7
Lower tail coverage 0.048 0.030 0.033 0.009 0.028 0.024
Results for one-way SA
CI coverage
SUR GEE 2SB MLM With robust SE With robust SE Without shrinkage correction With shrinkage correction ML Base case 0.940 0.933 0.981 0.943 0.950Few clusters per arm (M=3) 0.856 0.841 0.962 0.941 0.933
Few individuals per cluster (nm=10) 0.937 0.945 0.991 0.961 0.958
Highly imbalanced cluster size (cvimb=1) 0.919 0.916 0.981 0.963 0.951
High ICC for costs (ICCc=0.3) 0.936 0.935 0.980 0.944 0.953
High ICC for outcomes (ICCe=0.3) 0.941 0.941 0.941 0.943 0.945
Multi-way SA – CI coverage
• From moderate to few clusters (10, 5, 3 clusters per arm)
• From moderate to high cluster size imbalance (CoV=0.5 and 1)
0.70 0.80 0.90 3 4 5 6 7 8 9 10
No. of clusters per arm
moderate imbalance (CoV=0.5)
0.70 0.80 0.90 3 4 5 6 7 8 9 10 CI c o ver ag e
No. of clusters per arm high imbalance (CoV=1)
Multi-way SA - rMSE
• From moderate to few clusters (10, 5, 3 clusters per arm)
• From moderate to high cluster size imbalance (CoV=0.5 and 1)
300 400 500 600 3 4 5 6 7 8 9 10
No. of clusters per arm
moderate imbalance (CoV=0.5)
350 550 750 950 3 4 5 6 7 8 9 10 rMSE
No. of clusters per arm high imbalance (CoV=1)
Multi-way SA – CI coverage
SUR GEE 2SB MLM Without robust SE With robust SE With robust SE Without shrinkage correction With shrinkage correction ML Mean (SE) Bias 6.63 (4.40) 6.63 (4.41) 6.63 (4.40) 7.10 (4.38) 9.08 (4.42) 7.95 (4.33) rMSE 197 197 197 196 198 194 CI coverage 0.858 0.921 0.920 0.978 0.941 0.938 Mean CI width 583 726 724 924 836 754Lower tail coverage 0.072 0.041 0.041 0.014 0.031 0.033
Case study - Outreach
• Case study with a data structure that reflects our DGP.
• 40 clusters; balanced clusters; skewed costs (CoV=1.6)
• Methods perform similarly. TSB without correction shows much larger CIs
SUR GEE 2SB MLM Without Robust SE With Robust SE With Robust SE Without shrinkage correction With shrinkage correction ML Incremental cost (SE) 14.16 (15.84) 14.16 (19.49) 14.16 (19.47) 13.73 (24.67) 15.45 (18.94) 14.78 (19.27) Incremental outcome (SE) -0.057 (0.020) -0.057 (0.046) -0.057 (0.046) -0.061 (0.051) -0.059 (0.045) -0.058 (0.046) INB (SE) -1164 (403.2) -1164 (934.7) -1163 (933.9) -1226 (1031.4) -1198 (908.7) -1170 (917.8)
Summary
- Methods that ignore clustering give poor performance
- MLMs performs well throughout
- GEE and SUR: Perform badly when clusters<20
Worsen with high cluster size imbalance
- 2SB performs well once corrected
