As we approach the end of this chapter, I offer a handful of warnings about the in-ferential connection between samples and populations. I highly suggest that you be-come sensitive to these issues, because many professional journals contain articles in which the researcher’s conclusions seem to extend far beyond what the inferen-tial process legitimately allows. Unfortunately, more than a few researchers get car-ried away with the techniques used to analyze their data—and their technical reports suggest that they gave little or no consideration to the nature of their samples and populations.
My first warning has to do with a possible mismatch between the source of the researcher’s data and the destination of the inferential claims. Throughout this chapter, I have emphasized the importance of a good match between the sample and the population. Be on guard when you read or listen to research reports, because the desired fit between these two groups may leave much to be desired. Consider, for example, the information presented in Excerpt 5.22.
EXCERPT 5.21
• Checking for Attrition Bias
Of these teachers [in the study’s sample], 987 returned a completed questionnaire at Time 1 (T1), and 719 did so at Time 2 (T2). . . . Additional analyses examining the pattern of participant attrition revealed that the T2 sample did not differ signifi-cantly from those who responded at T1 on nine dimensions (gender, age, marital status, school sector, years of teaching experience, number of teachers in the school, number of students in the school, school location, and socioeconomic status of school area).
Source: Bradley, G. L. (2010). Work-induced changes in feelings of mastery. Journal of Psychology, 144(2), 97–119.
EXCERPT 5.22
• Mismatch Between Sample and Intended Population
We sought to develop a dual-process model of sexual aggression by examining the relationships between men’s implicit power–sex association and explicit power–sex beliefs, rape myth acceptance, and rape proclivity. . . . In Study 1, we developed and validated an explicit measure of power–sex beliefs. In Study 2, we used this mea-sure of explicit power–sex beliefs, and an implicit meamea-sure of a power–sex associa-tion, to compare two alternative dual-process models of rape proclivity. . . . Participants [in Study 1] were 131 college students from a Midwestern, Catholic university [who were] enrolled in an upper-level psychology course or an intro-ductory anthropology course. . . . Participants [in study 2] were 108 men from a (continued )
The major concern you ought to have with the passage in Excerpt 5.22 is the vast difference between the declared population of interest and the actual sample.
The researcher’s intention was to use sample data and inferential statistics as a basis for making claims about men’s beliefs. However, data were collected from a sam-ple of male students attending a single, religion-oriented university, with students selected from just two kinds of courses (psychology and anthropology). These fea-tures of the sample made it impossible for the study’s findings to be generalized to the stated population of interest.
My next warning has to do with the size of the sample. If you do not know much about the members of the sample or how the researcher obtained the sample, then the inferential process cannot operate successfully—no matter how large the sample might be. Remember, therefore, that it is the quality of the sample (rather than its size) that makes statistical inference work. Proof of this claim can be seen during national elections when pollsters regularly predict with great accuracy who will win elections even though the samples used to develop these predictions are relatively small.
My third warning concerns the term random. Randomness in research studies is usually considered to be a strong asset, but you should not be lulled into think-ing that an investigation’s results can be trusted simply because the term random shows up in the method section of the write-up. Consider, for example, the mater-ial presented in Excerpts 5.23 and 5.24.
EXCERPTS 5.23–5.24
• The Word Random
[T]he aim of the present study was to evaluate the patterns of rapid weight loss in a large sample of competitive judo athletes. . . . During the [judo] competitions, the participants were approached randomly and invited to participate in the study.
Source: Artigli, G. G., Gualano, B., Franchini, E., Scagliusa, F. B., Takesian, M., Fuchs, M., et al. (2010). Prevalence, magnitude, and methods of rapid weight loss among judo competi-tors. Medicine and Science in Sports and Exercise, 42(3), 436–442.
Surveys, including the School Counselor Self-Efficacy Scale (Bodenhorn &
Skaggs, 2005), questions regarding the school counseling program, achievement (continued ) EXCERPT 5.22
• (continued )
Midwestern, Catholic university who were recruited through an introductory psy-chology course and given course extra credit for their participation. The mean age for this sample was 19.1 years (SD 1.3 years).
Source: Chapleau, K. M., & Oswald, D. L. (2010). Power, sex, and rape myth acceptance:
Testing two models of rape proclivity. Journal of Sex Research, 47(1), 66–78.
In Excerpt 5.23, we are told that the judo competitors were “approached ran-domly” and asked if they would like to participate in the researchers’ study. I am not sure what this means. It is possible, of course, that a subset of all the judo com-petitors were randomly selected. However, it is my hunch that the word casually, if substituted for the word randomly, would more accurately describe how the judo athletes were approached.
In Excerpt 5.24, we see the word randomly used twice in the statistical sense of the term. A large, random sample of members of the ASCA professional association was drawn, and then the sample was randomly divided into two halves. However, this randomness was undermined when nearly half of the study’s intended participants did not return the survey. The sample size of 860 may seem large; however, there is no way to know if the survey results were tainted by a response bias. Although the gender split (85 percent female) and eth-nicity (89 percent European American) among the 860 respondents were re-ported to be “similar to the demographic characteristics of school counselors found in most national studies,” the low response rate destroyed the randomness of the initial random sample.
Regarding another matter, researchers should describe the procedures used to extract samples from their relevant populations. They should do this because the question of whether a sample is a random sample can be answered only by consid-ering the procedure used to select the sample. As indicated earlier in the chapter, one not-too-sophisticated procedure for getting a random sample is to draw slips of paper from a hat. Random samples can also be produced by flipping coins or rolling dice to determine which members of the population end up in the sample.
Most contemporary researchers do not draw their random samples by rolling dice, flipping coins, or drawing slips of paper from a hat. Instead, they utilize either a printed table of random numbers or a set of computer-generated random numbers. To identify which members of the population get into the sample, the re-searcher first assigns unique ID numbers (e.g., 1, 2, 3) to the members of the pop-ulation. Then, the researcher turns to a table of random numbers (or a set of computer-generated random numbers) where the full set of ID numbers appear in a
gap information, and demographics, were sent to a random sample of 1,600 ASCA members. Through random selection of these participants, half of the surveys were sent through postal mail and half through e-mail/Internet. . . . The overall response rate was 54% (860 individuals responded).
Source: Bodenhorn, N., Wolfe, E. W., & Airen, O. E. (2010). School counselor program choice and self-efficacy: Relationship to achievement gap and equity. Professional School Counsel-ing, 13(3), 165–174.
EXCERPTS 5.23–5.24
• (continued)
scrambled order. Finally, the ID numbers that appear at the top of the list (e.g., 27, 4, 9) designate which members of population get into the sample.
In Excerpt 5.25, we see how easy it is for a researcher to indicate that a ran-dom sample was selected via a table of ranran-dom numbers or computer-generated random numbers. These authors deserve credit for clarifying exactly how their random samples were created. All researchers should follow this good example!
EXCERPT 5.25
• Using a Table of Random Numbers
In Trinidad there are seventy nine (79) health centres, most of which have walk-in clinics (where patients can present without an appointment for any medical prob-lem). These clinics were stratified to represent all regional health authorities (ad-ministrative regions) and to capture rural and urban populations. Clinics [n 16]
were then selected using a table of random numbers [to match] the proportion of clinics per administrative region.
Source: Maharaj, R. G., Alexander, C., Bridglal, C. H., Edwards, A., Mohammed, H., Rampaul, T., et al. (2010). Abuse and mental disorders among women at walk-in clinics in Trinidad: A cross-sectional study. BMC Family Practice, 11(26), 1–21.
The final warning is really a repetition of a major concern expressed earlier in this chapter. Simply stated, an empirical investigation that incorporates inferen-tial statistics is worthless unless there is a detailed description of the population or the sample. No matter how carefully the researcher describes the measuring instru-ments and procedures of the study, and regardless of the levels of appropriateness and sophistication of the statistical techniques used to analyze the data, the results are meaningless unless we are given a clear indication of the population from which the sample was drawn (in the case of probability samples) or the sample itself (in the case of nonprobability samples). Unfortunately, too many researchers get carried away with their ability to use complex inferential techniques when analyzing their data. I can almost guarantee that you will encounter technical write-ups in which the researchers emphasize their analytical skills to the near exclusion of a clear explanation of where their data came from or to whom the results apply. When you come across such studies, give the authors high marks for being able to flex their
“data analysis muscles”—but low marks for neglecting the basic inferential nature of their investigations.
To see an example of a well-done description of a sample, consider Excerpt 5.26.
Given this relatively complete description of the 136 children who formed this study’s sample, we have a much better sense of the population to which the statis-tical inferences can be directed. It would be nice if all researchers described their samples with equal care.
EXCERPT 5.26
• Detailed Description of a Sample
We examined the summer employment and community participation experiences of 136 youth with severe disabilities. To be included in this study, students had to (a) be receiving special education services under a primary or secondary disability category of cognitive disability, autism, or multiple disabilities; (b) be eligible for the state’s alternate assessment; and (c) provide assent and parental consent to participate. . . . Youth participating in our study ranged in age from 13.9 to 21.8 (M 18.2; SD 1.8), and slightly more than half were male (52.9%). The majority (85.3%) was European American, 11.8% were African American, and 2.9% reported other races/ethnicities (i.e., Asian American, American Indian). Twenty-six students (19.1%) were in 9th grade, 18 (13.2%) were in 10th grade, 36 (26.5%) were in 11th grade, 37 (27.2%) were in 12th grade, and 19 (14.0%) received services in 18 to 21 programs. More than one quarter (28.7%) of students were eligible for free/reduced lunch (FRL). Most youth were reported to be served under the primary disability category of cognitive disabili-ties (85.3%), followed by autism (10.3%) and orthopedic impairments (4.4%); 61.0%
were reported to have one or more secondary disabilities.
Source: Carter, E. W., Dutchman, N., Ye, S., Trainor, A. A., Swedeen, B., & Owens, L. (2010).
Summer employment and community experiences of transition-age youth with severe disabil-ities. Exceptional Children, 76(2), 194–212.
The Best Items in the Companion Website
1. An interactive online quiz (with immediate feedback provided) covering Chapter 5.
2. Ten misconceptions about the content of Chapter 5.
3. An email message sent from the author to his students to help them under-stand the difference between tangible and abstract populations.
4. A poem about questionnaires (written by a famous statistician).
5. The best passage from Chapter 5 (selected by the author).
To access the chapter outline, practice tests, weblinks, and flashcards, visit the com-panion website at http://www.ReadingStats.com.
Review Questions and Answers begin on page 531.
I
n Chapter 5, we laid the foundation for our consideration of inferential statistics.We did this by considering the key ingredients of this form of statistical thinking and analysis: population, sample, parameter, statistic, and inference. In this chap-ter, we now turn our attention to one of the two main ways in which researchers use sample statistics to make educated guesses as to the values of population parameters.
These procedures fall under the general heading estimation.
This chapter is divided into three main sections. First, the logic and techniques of interval estimation are presented. Next, we examine a second, slightly different way in which estimation works; this approach is called point estimation. Finally, I offer a few tips to keep in mind as you encounter research articles that rely on either of these forms of estimation.
Before beginning my discussion of estimation, I want to point out that the two major approaches to statistical inference—estimation and hypothesis testing—are similar in that the researcher makes an educated guess as to the value of the popu-lation parameter. In that sense, both approaches involve a form of guesswork that might be construed to involve estimation. Despite this similarity, the term estimation has come to designate just one of the two ways in which researchers go about making their educated guesses about population parameters. The other approach, hypothesis testing, is discussed in Chapters 7 and 8.