Sampling for Extensive Studies Barbara M Wildemuth
THE EFFECTS OF NONRESPONSE ON A SAMPLE
You will be selecting your sample at the very beginning of your study. But as you collect your data, some of those participants that you selected for your sample will not accept your invitation to participate in the study. A recent study of surveys published in three core information and library science journals from 1996 to 2001 found that on average, they achieved response rates of only 63 percent (Burkell, 2003). This level of nonresponse (i.e., 37%) can negatively affect the representativeness of the sample you actually include in your study results. While you can oversample to increase the number of participants you include, this practice will not necessarily help you avoid a biased sample. Therefore you should try to do everything you can to improve your response rate.
The primary method for improving your response rate is to use appropriate data collection procedures. In your recruitment materials, use wording that will motivate potential participants. Lower the burden of participation by using well-designed ques- tionnaires or study procedures, making it easy and pleasurable to be a study participant. Offer concrete incentives to the participants, for example, gift certificates, cash, or other items. Dillman’s (2007) tailored design method for surveys has many additional sugges- tions for improving survey response rates, and many of these will apply to other types of studies as well. Once you’ve completed your study, you should assess the sampling bias that might have been caused by nonresponse. You can compare the respondents with the population as a whole to see if they are representative. You can compare the respondents with the nonrespondents to see if they differ. As you try to interpret your results, you should keep in mind that the nonrespondents are likely to have responded
124 APPLICATIONS OF SOCIAL RESEARCH METHODS
more like the late responders than those who responded early in the recruitment period. Finally, if you have a low participation/response rate, be cautious about the conclusions you draw from the results.
EXAMPLES
Three different studies will be discussed here, each using a different approach to defining and recruiting a sample. The first (McKenna et al., 2005) incorporated two samples, one a stratified sample and the other a cluster sample. The second (Uhegbu & Opeke, 2004) combined cluster sampling with quota sampling. The third (Thorpe, 1986) used quota sampling to recruit participants from a particularly challenging population.
Example 1: A Study of Occupational Therapists’ Database Use
McKenna and her colleagues (2005) were interested in studying the extent to which occupational therapists (OTs) were using OTseeker, a database of evidence regarding the effectiveness of various OT practices. They conducted a survey in two phases. In the first phase, they selected a stratified sample of members of OT Australia, a professional organization. The population of interest was all occupational therapists in Australia, but the study population was limited to members of OT Australia. Given that therapists that joined a professional association were also more likely to be incorporating evidence- based practice into their routines, this compromise was a reasonable one. The OT Australia membership list was the sampling frame and included approximately 4,500 members. It was stratified by state/territory. A proportionate random sample was then selected from each state/territory, that is, each stratum. The sample was 400 members; the authors do not provide an explanation of why this sample size was selected. These sampling procedures are very appropriate; unfortunately, a low response rate (31%) made the validity of the study results questionable. Therefore the authors undertook a second phase, drawing a second sample.
In the second sample, the research team focused their efforts in two states. The two states, Queensland and New South Wales, were selected because they were the home states of members of the team. They believed that the occupational therapists in those states were most likely to have used the OTseeker database because they would have heard of it through local sources; thus they were more likely to respond to a questionnaire about their database use. A cluster sampling approach was used for this phase. The sampling frame for Queensland was “a directory of facilities employing occupational therapists who are members of OT AUSTRALIA Queensland” (McKenna et al., 2005, p. 207), and in New South Wales, the sampling frame was “a university’s directory of facilities that provide fieldwork placements for occupational therapy stu- dents” (p. 207). The number of clusters (i.e., facilities) selected from each part of the sample was in proportion to the number of clusters in each sampling frame. Altogether, 95 facilities were selected for inclusion in the sample. Each facility received enough questionnaire packets for all the occupational therapists working at that facility. Of the 326 questionnaires distributed, 27 percent were returned.
Clearly these authors made every attempt to select and recruit a representative sample of occupational therapists in Australia, even going so far as to conduct a follow-up phase of the study to increase their sample size. Their study procedures also included follow- ups with potential respondents in an effort to improve their response rate. Even with
Sampling for Extensive Studies 125
these efforts, it would be hard to argue that the study results are based on a representative sample, as noted in the authors’ discussion of the study’s limitations. In that discussion, they also noted that members of OT Australia are a biased sample of all OTs in Australia and that respondents might have differed from nonrespondents in their familiarity with and use of the OTseeker database. We can conclude that the sampling procedures for this study were quite appropriate but that the conclusions are weakened by the effects of nonresponse.
Example 2: Interactive Interpersonal Networking among Rural Women in Nigeria
As rural Nigeria experiences an increasing flow of information, there is evidence that women are not fully participating in this flow due to barriers to information flow among themselves. They are believed to be “secretive” (Uhegbu & Opeke, 2004, p. 521), to their detriment. Therefore this study was undertaken to understand the factors that affect information flows among rural women in one state in Nigeria and to describe the character of the information dissemination that does occur among them. The methodological challenges presented by this study make it an interesting example to examine in depth.
Uhegbu and Opeke (2004) clearly define their population as “women who are indi- genes of Imo State and resident therein” (p. 523). They go on to further define their target population in terms of the women’s marital status, including only those who are or have been married. Unfortunately, the census data that provides a basis for their sampling plan do not designate the marital status of each person; this issue will be resolved in the final stages of the sampling procedures. The first stage of the sampling procedure was to stratify the 27 local government areas (LGAs) in Imo State into three zones. Two LGAs were then selected from each of the three zones for participation in the study. The next stage of the sampling procedures involved proportionate quota sampling. The number of women to be included in the sample from each LGA was proportional to the female population within that LGA, in relation to the state’s female population. In addition, the quota sampling strategy was intended to eliminate unmarried females, include equal numbers of literate and illiterate women, and include proportionate numbers of rural- and suburban-dwelling women. A total sample of 500 women was selected; of those, 485 participated in the study. It is not clear whether additional women were invited to participate and declined.
Because a quota sampling approach was used in combination with the stratified sampling approach, it is possible that the sample was biased. In addition, nonresponse may have been an issue that was not reported. While these authors addressed many of the challenges associated with studying this population, our understanding of their results would have been strengthened with a more complete consideration of the possibilities for bias in the study sample.
Example 3: Non-users of a Public Library
To identify factors affecting non-use of the Enfield North section of the Enfield (England) Public Libraries, a quota sample of people was selected and interviewed (Thorpe, 1986). They were asked about their reasons for not using the library, whether they knew where it was, and other questions intended to guide the library’s marketing activities.
126 APPLICATIONS OF SOCIAL RESEARCH METHODS
The population for the study was those people who lived in the Enfield North section of town but did not use the library. Three characteristics of the members of the population were used to select the sample. First, the section of town was divided into 10 wards. Each ward was proportionately represented in the final sample. The other two characteristics considered were age and gender. There were four age categories: 16–24, 25–44, 45–64, and 65+. The goal was to recruit three or four individuals from each ward/age/sex category to be broadly representative of the community. Thorpe (1986) considered adding social class as a fourth attribute that could be used for the quota sampling because social class has been found to be related to library use. This possibility was eventually rejected because of the complexity it would have added to the sampling and data collection procedures, possibly lowering the response rate because of the increased burden on participants.
Altogether, 1,049 people were approached and invited to participate in the study. Of those, many did not meet the inclusion criteria for the study: they were users of the library (i.e., they had used at least one library service at some time within the previous year;
n= 253) or they lived outside Enfield North (n = 278). Of those invited to participate,
234 declined. So, of those meeting the inclusion criteria, 54 percent were included in the final sample.
This study is a good example of how quota sampling can be used in a community- based study. In particular, the use of a three-dimensional quota sampling plan im- proved the likelihood that the participants were representative of their community. While both the sampling method and the level of nonresponse weaken the generalizabil- ity of the results, Thorpe (1986) is appropriately cautious in his interpretations of the findings.
CONCLUSION
These three studies illustrate both the challenges that researchers may face in de- veloping an adequate sampling plan and some creative approaches to addressing those challenges. In each case, the authors were very careful to apply the sampling methods correctly. By using several different sampling techniques in combination, they were able to overcome some of the problems. They were aware of, and made the reader aware of, the limitations of their sampling methods.
One weakness that all these examples displayed is the lack of a rationale for their sample size. None of them worked with particularly small samples, so they may not have felt the need to justify their sample size. Nevertheless, it is worthwhile to con- sider the effects of your sample size on the achievement of your research goals while you are planning a study. This process can only strengthen the validity of your study findings.
WORKS CITED
Asraf, R. M., & Brewer, J. K. (2004). Conducting tests of hypotheses: The need for an adequate sample size. Australian Educational Researcher, 31(1), 79–94.
Baker, L. M., Kars, M., & Petty, J. (2004). Health sciences librarians’ attitudes toward the Academy of Health Information Professionals. Journal of the Medical Library Association, 92(3), 323–333.
Sampling for Extensive Studies 127
Bocij, P. (2003). Victims of cyberstalking: An exploratory study of harassment perpetrated via the Internet. First Monday, 8(10). Retrieved August 1, 2007, from http://firstmonday.org/ issues/issue8 10/bocij/index.html.
Bookstein, A. (1974). How to sample badly. Library Quarterly, 44(2), 124–132.
Burkell, J. (2003). The dilemma of survey nonresponse. Library and Information Science Research,
25(3), 239–263.
Case, D. O., Johnson, J. D., Andrews, J. E., Allard, S. L., & Kelly, K. M. (2004). From two- step flow to the Internet: The changing array of sources for genetics information seeking.
Journal of the American Society for Information Science and Technology, 55(8), 660–669.
Cohen, J. (1965). Some statistical issues in psychological research. In B. B. Wolman (Ed.),
Handbook of Clinical Psychology (pp. 95–121). New York: McGraw-Hill.
Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.). Hillsdale, NJ: Erlbaum.
Czaja, R., & Blair, J. (2005). Designing Surveys: A Guide to Decisions and Procedures. Thousand Oaks, CA: Pine Forge Press.
Dillman, D. A. (2007). Mail and Internet Surveys: The Tailored Design Method (2nd ed.). New York: John Wiley.
Fan, X., & Wang, L. (1999, November 3–6). Practical guidelines for evaluating sampling designs
in survey studies. Paper presented at the annual meeting of the American Evaluation
Association, Orlando, FL. (ERIC Document Reproduction Service No. ED435701) Faulkner, L. (2003). Beyond the five-user assumption: Benefits of increased sample sizes in
usability testing. Behavior Research Methods Instruments and Computers, 35(3), 379– 383.
Howell, D. C. (2004). Power. In Fundamental Statistics for the Behavioral Sciences (5th ed., pp. 332–354). Belmont, CA: Brooks / Cole.
Lenth, R. V. (2006). Java Applets for Power and Sample Size [Computer software]. Retrieved July 25, 2007, from http://www.stat.uiowa.edu/∼rlenth/Power/.
Lewis, J. R. (2006). Sample sizes for usability tests: Mostly math, not magic. Interactions, 13(6), 29–33.
Lippmann, W. (2007). Public Opinion. Bibliobazaar. (Original work published 1922.)
Loether, H. J., & McTavish, D. G. (1993). Descriptive and Inferential Statistics: An Introduction. Boston: Allyn and Bacon.
McKenna, K., Bennett, S., Dierselhuis, Z., Hoffmann, T., Tooth, L., & McCluskey, A. (2005). Australian occupational therapists’ use of an online evidence-based practice database (OTseeker). Health Information and Libraries Journal, 22, 205–214.
Nielsen, J. (1993). Usability Engineering. Boston: AP Professional.
Pedhazur, E. J., & Schmelkin, L. P. (1991). Measurement, Design, and Analysis: An Integrated
Approach. Hillsdale, NJ: Erlbaum.
Rea, L. M., & Parker, R. A. (1997). Designing and Conducting Survey Research: A Comprehensive
Guide. San Francisco: Jossey-Bass.
Scheaffer, R. L., Mendenhall, W., III, & Ott, L. (1996). Elementary Survey Sampling (5th ed.). Belmont, CA: Duxbury Press.
Spool, J., & Schroeder, W. (2001). Testing Web sites: Five users is nowhere near enough. CHI
2001 Abstracts, 285–286.
Thorpe, S. (1986). Market research in Enfield Libraries. Public Library Journal, 1(4), 49–54. Uhegbu, A. N., & Opeke, R. O. (2004). Deterrents to effective interactive interpersonal information
networking of rural women in Imo State, Nigeria. Journal of Information Science, 30(6), 520–527.
128 APPLICATIONS OF SOCIAL RESEARCH METHODS
ADDITIONAL RECOMMENDED READING
Hernon, P. (1994). Determination of sample size and selection of the sample: Concepts, general sources, and software. College and Research Libraries, 55(2), 171–179.
Rosenthal, R., & Rosnow, R. L. (1984). Indices of effect size. In Essentials of Behavioral Research:
Methods and Data Analysis (pp. 361–365). New York: McGraw-Hill.