Correction for the Design Effect in School-Based Substance Use
USE OF THE DEFF CORRECTION
Use of the correction for DEFFs has the advantages of enabling use of individual raw scores for individual longitudinal followup, use of all standard statistical hypothesis testing techniques, and testing of inter- actions that are not totally nested in the design. It also is generalizable to the calculation of sample size requirements, as already noted by Donner and colleagues (1981) and Murray and Hannan (1990). As mentioned above, the problem arose in survey research when research- ers sampled clusters (e.g., groups of housing units) rather than using simple random sampling techniques. Cluster sampling results in
increased variances for sample estimates due to the within-class or within-cluster correlations on the dependent variables. This within- class, or intraclass, correlation has been used in social science research for some time, for example, as a method of calculating interrater reliability (Guilford 1954).
The intraclass correlation is simply the ratio of the difference between the dependent variable between- and within-class variances to the total variance, or in its simplest form,
where is the between-class variance, is the within-class
variance, ñg is the harmonic mean group (class or cluster) size, and p
is the intraclass correlation. The two variance terms can be obtained from any one-way ANOVA subroutine using the unit of assignment (e.g., classroom or school) as the independent variable. There are more sophisticated subroutines that complete all of the necessary calculations. For example, SUDAAN (Shah 1990), a set of software produced by Research Triangle Institute, allows one to specify both the model and design in the analysis subroutines. Collins and colleagues (1989) have shown that the usual estimators of the intra-class
correlation, the least squares and the maximum likelihood estimators, are negatively biased (i.e., they are underestimates of the necessary inflation factor). Donoghue and Collins (1990), working with the derivation of the minimum variance unbiased estimator of the
intraclass correlation provided by Olkin and Pratt (1958), have provid- ed a computer program for calculation of the unbiased estimator. The correction was not trivial in the example provided by Donoghue and Collins (1990), and use of the correction is recommended.
The design effect, DEFF = l+[p(ñg-l)], is used to multiply the
standard error of the estimate being used in order to account for the intraclass correlation. Once the standard error is inflated by the DEFF,
the statistic can be computed in the usual way, and the confidence intervals for estimates can be ascertained using standard tabled values for the z- or t-statistic or the F-statistic in a close approximation. In correcting the more complicated research designs, it is more precise to specify the appropriate model (e.g., nested effects) as well as the clustered sampling effect.
Use of the DEFF is directly generalizable to sample size calculations. These generalizations as applicable to school-based prevention research are shown quite adequately in Murray and Hannan (1990) and will not be repeated here except for one example. The calculation of the necessary per-cell sample size for the usual longitudinal study, excluding the DEFF, is:
where Za is the critical value beyond which a tolerable Type I error falls, Zß is the critical value beyond which a tolerable Type II error falls, r is the correlation between scores at two points in time (e.g., pretest and posttest), s2 is the cross-sectional dependent variable
variance, and d2 is the square of the hypothesized magnitude of the
final outcome difference between treatment conditions on the depend- ent variable. Adding the DEFF to the calculation simply requires
multiplying the numerator by 1+( ñg-1)p.
For example, assume that the required n per cell is 100 without taking DEFF into account. In the usual prevention study conducted by the author and his colleagues, the average class size has been around 25, and the usual case involves an average of two classes per grade level within a school, so the average school size has been about 50. The typical intraclass correlation has been around .02 or .03. If the school has been the unit of assignment, and if p = .02, the DEFF calculation usually has resulted in a value of approximately DEFF = 1+(50-1)(.02) = 1.98, and the required per-cell sample size therefore has been 198 rather than 100. When p increases to .03, however, a DEFF of 2.47 and a required per-cell sample size of 247 results, There is a linear increase in the DEFF and the required cell size with further increments in p.
By contrast, using class averages as the dependent variable would require 100 classrooms (rather than 100 subjects), or about 2,500 students (given about 25 students per classroom) per cell, and the disadvantages noted above would remain. The use of hierarchical ANOVA, not allowing for correlated error in the model, also would be subject to the disadvantages noted above. The sample size resulting from the DEFF approach maximizes efficiency and minimizes unten- able assumptions, while providing a correction for the DEFF in the determination of the desired sample size and the data analyses. In practice, the DEFF will differ for each dependent variable and for each subgroup (e.g., gender, ethnic group, or grade level) because of differ- ences in the intraclass correlations, including the average class size, on the variables. When calculating the desired sample size, one can pro- ceed by making conservatively large estimates, basing estimated sample size requirements on the DEFF for the variable with the largest DEFF or on the largest DEFF of the variables that are most important in the study. In estimating the intraclass correlation for sample size determination purposes, most difficulties usually arise when the intra- class correlation may differ by location or grade level or both. If no prior local data are available for this purpose, a local pilot study is recommended in order to estimate the magnitude of the intraclass correlation. The calculations are straightforward for the correction for DEFF in the statistical analyses, as mentioned above, and one can calculate the interclass correlations for each subanalysis directly from the study data.
SUMMARY
The typical school-based substance abuse prevention study uses class- rooms or schools as the unit of assignment to study conditions. It is inappropriate to use individuals as the unit of analysis in this condi- tion, as the individuals probably are not entirely independent observa- tional units. Early studies proposed using class means as the units of analysis, which discards much individual variance. Superior, more recent strategies are available. The correction of the sampling vari- ances for the DEFF has been available for about 25 years and now is becoming more widely used by researchers in the substance abuse prevention field. This correction should be used for both sample size estimation and subsequent analyses.
ACKNOWLEDGMENTS
This research was supported by grants AA06324 and AA08447 from the National Institute on Alcohol Abuse and Alcoholism. It was first presented at a National Institute on Drug Abuse (NIDA) Technical Review, “Scientific Methods for Prevention Intervention Research,” Bethesda, MD, in September 1992.
The author gratefully acknowledges the helpful comments made on this paper by Steve Heeringa of the University of Michigan Institute for Social Research Sampling Section; Patrick O’Malley of the University of Michigan Survey Research Center, Institute for Social Research; and by participants in the NIDA Technical Review. Address requests for reprints to the author at the Department of Postgraduate Medicine and Health Professions Education, G1210 Towsley Center, University of Michigan Medical School, Ann Arbor, MI 48109-0201.
REFERENCES
Altham, P.M.E. Discrete variable analysis for individuals grouped into families. Biometrika 63:263-269, 1976.
Barcikowski, R.S. Statistical power with group mean as the unit of analysis. J Educ Stat 6:267-285, 1981.
Berberian, R.; Gross, C.; Lovejoy, J.; and Paparella, S. The effective- ness of drug education programs: A critical review. Health Educ
Monogr 4:377-398, 1976.
Blair, R.C., and Higgins, J.J. Comment on “Statistical power with group mean as the unit of analysis.” J Educ Stat 11:161-169, 1986. Bryk, A.S., and Raudenbush, S.W. Hierarchical Linear Models.
Newbury Park, CA: Sage Publications, 1992.
Campanelli, P.C.; Dielman, T.E.; Shope, J.T.; Burchart, A.T.; and Renner, D.S. Pretest and treatment effects in an elementary school- based alcohol misuse prevention program. Health Educ Q
16(1):113-130, 1989.
Campbell, D.T., and Stanley, J.C. Experimental and quasi-experimental designs in research on teaching. In: Gage, N.L., ed. Handbook of
Cohen, J.E. The distribution of the chi-squared statistic under clustered sampling from contingency tables. J Am Stat Assoc 71:665-670, 1976.
Collins, L.M.; Donoghue, J.R.; and McGuigan, K.A. “Assessing Dependence Among Observations.” Paper presented at the 97th Annual Convention of the American Psychological Association, New Orleans, LA, August 11-15, 1989.
Cornfield, J. Randomization by group: A formal analysis. Am J
Epidemiol 108:100-102, 1978.
Dielman, T.E.; Shope, J.T.; Leech, S.L.; and Butchart, A.T. Differ- ential effectiveness of an elementary school-based alcohol misuse prevention program. J Sch Health 59(6):255-263, 1989.
Donner, A. An empirical study of cluster randomization. Int J
Epidemiol 11:283-286, 1982.
Donner, A. A regression approach to the analysis of data arising from cluster randomization. Int J Epidemiol 14:322-326, 1985.
Donner, A.; Birkett, N.; and Buck, C. Randomization by cluster: Sample size requirements and analysis. Am J Epidemiol 114(6):906- 914, 1981.
Donoghue, J.R., and Collins, L.M. A note on the unbiased estimation of the intraclass correlation. Psychometrika 55:159-164, 1990. Fletcher, H.J. Possible interpretive problems in analyses using group
means as the experimental unit. Psychol Bull 69:157-160, 1968. Glass, G.V., and Stanley, J.C. Statistical Methods in Education and
Psychology. Englewood Cliffs, NJ: Prentice Hall, 1970.
Guilford, J.P. Psychometric Methods. New York: McGraw-Hill, 1954. Hopkins, K. The unit of analysis: Group means versus individual
observations. Am Educ Res J 19:5-18, 1982.
Kempthome, O. The Design and Analysis of Experiments. New York: Wiley, 1952.
Kinder, B.; Pape, N.; and Walfish, S. Drug and alcohol education: A review of outcome studies. Int J Addict 15:1035-1054, 1980. Kish, L. Survey Sampling. New York: John Wiley and Sons, 1965. Lee, E.S.; Forthofer, R.N.; and Lorimor, R.J. Analyzing Complex
Survey Data Newbury Park, CA: Sage Publications, 1989.
Lindquist, E.F. Statistical Analysis in Educational Research. New York: Houghton Mifflin, 1940.
Lindquist, E.F. Design and Analysis of Experiments in Psychology and
Lumsdaine, A.A. Instruments and media of instruction. In: Gage, N.L., ed. Handbook of Research on Teaching. Chicago: Rand McNally,
1963.
McAlister, A.L.; Perry C.; and Maccoby, N. Adolescent smoking: Onset and prevention. Pediatrics 63:650-658, 1979.
McCullagh, P., and Nelder, J.A. Generalized Linear Models. London: Chapman and Hall, 1983.
McNemar, Q. Review of Statistical Analysis in Educational Research, by E.F. Lindquist. Psychol Bull 37:747, 1940.
Moskowitz, J.M.; Malvin, J.H.; Schaeffer, G.A.; and Schaps, E. An experimental evaluation of a drug education course. J Drug Educ
14:9-22, 1984.
Murray, D.M., and Hannan, P.J. Planning for the appropriate analysis in school-based drug-use prevention studies. J Consult Clin Psychol 58(4):458-468, 1990.
Murray, D.M.; Hannan, P.J.; and Zucker, D.M. Analysis issues in school-based health prevention studies. Health Educ Q 16:315-320, 1989.
Olkin, I., and Pratt, J.W. Unbiased estimation of certain correlation coefficients. Ann Math Stat 29:202-211, 1958.
Peckham, P.D.; Glass, G.V.; and Hopkins, K.D. The experimental unit in statistical analysis. J Special Educ 3:337-349, 1969.
Raths, J. The appropriate experimental unit. Educ Leadership 12:263- 266, 1967.
Shah, B.V. SUDAAN: Professional Software for Survey Data
Analysis. Research Triangle Institute Park, NC: Research Triangle
Institute, 1990.
Staulcup, H.; Kenward, K.; and Frigo, D. A review of federal primary alcoholism prevention projects. J Stud Alcohol 40:943-968, 1979. Winer. B.J. Statistical Principles in Experimental Design. New York:
AUTHOR
T.E. Dielman, Ph.D. Professor
Postgraduate Medicine and Health Professions Education G1210 Towsley Center, Box 0201
University of Michigan Ann Arbor, MI 48109-0201