Within this discussion of reliability and validity, I have not addressed a question that most likely passed through your mind at least once as we considered different pro-cedures for assessing consistency and accuracy: “How high must the reliability and validity coefficients be before we can trust the results and conclusions of the study?” Before leaving this chapter, I want to answer this fully legitimate question.
r2
For both reliability and validity, it would be neat and tidy if there were some absolute dividing point (e.g., .50) that separates large from small coefficients. Un-fortunately, no such dividing point exists. In evaluating the reliability and validity of data, the issue of “large enough” has to be answered in a relative manner. The question that the researcher (and you) should ask is, “How do the reliability and validity of data associated with the measuring instrument(s) used in a given study compare with the reliability and validity of data associated with other available in-struments?” If the answer to this query about relative quality turns out to be “pretty good,” then you should evaluate the researcher’s data in a positive manner—even if the absolute size of reported coefficients leaves lots of room for improvement.
My next comment concerns the possible use of multiple methods to assess in-strument quality. Because there is no rule or law that prohibits researchers from using two or more approaches when estimating reliability or validity, it is surpris-ing that so many research reports contain discussions of one and (if validity is dis-cussed at all) only one kind of validity. That kind of research report is common because researchers typically overlook the critical importance of having good data to work with and instead seem intent on quickly analyzing whatever data have been collected. Give credit to those few researchers who present multiple kinds of evi-dence when discussing reliability and validity.
My third point is simple: Give credit to a researcher who indicates that he or she considered the merits of more than one measuring instrument before deciding on which test or survey to use. Too many researchers, I fear, decide first that they want to measure a particular trait or skill and then latch on to the first thing they see or hear about that has a name that matches that trait or skill. In Excerpt 4.25, we see an example of a better way of going about instrument selection. (In this excerpt, note that the third criterion includes a consideration of reliability and validity.)
EXCERPT 4.25
• Reasons Provided as to Why a Given Instrument Was Selected
Eight survey instruments that measured trust were evaluated using predetermined criteria.
The criteria described an instrument that (1) measures trust on a continuum scale; (2) has a short completion time (<10 min); (3) is available, reliable, and valid;
and (4) has the ability to measure multiple dimensions of trust. The Organizational Trust Index (OTI) survey instrument was selected for use in this study as it best met the established criteria, including the ability to measure the five dimensions of trust identified during the literature review.
Source: Alston, F., & Tippett, D. (2009). Does a technology-driven organization’s culture influence the trust employees have in their managers? Engineering Management Journal, 21(2), 3–10.
My last general comment about reliability and validity is related to the fact that data quality, by itself, does not determine the degree to which a study’s results can be trusted. It is possible for a study’s conclusions to be totally worthless even though the data analyzed possess high degrees of reliability and validity. A study can go down the tubes despite the existence of good data if the wrong statistical pro-cedure is used to analyze data, if the conclusions extend beyond what the data le-gitimately allow, or if the design of the study is deficient. Reliability and validity are important concepts to keep in mind as you read technical reports of research in-vestigations, but other important concerns must be attended to as well.
Accuracy
The Best Items in the Companion Website
1. An interactive online quiz (with immediate feedback provided) covering Chapter 4.
2. Ten misconceptions about the content of Chapter 4.
3. An online resource entitled “Multitrait–Multimethod.”
4. An email message about convergent and discriminant validity sent from the au-thor to his students to help them understand these two measurement concepts.
5. Chapter 4’s best paragraph.
To access the chapter outline, practice tests, weblinks, and flashcards, visit the com-panion website at http://www.ReadingStats.com.
Review Questions and Answers begin on page 531.
I
n Chapters 2 through 4, we considered various statistical procedures that are used to organize and summarize data. At times, the researcher’s sole objective is to describe the people (or things) in terms of the characteristic(s) associated with the data. When that is the case, the statistical task is finished as soon as the data are displayed in an organized picture, are reduced to compact indices (e.g., the mean and standard deviation), are described in terms of distributional shape, are evaluated relative to the concerns of reliability and validity, and, in the case of a bivariate concern, are examined to discern the strength and direction of a relationship.In many instances, however, the researcher’s primary objective is to draw con-clusions that extend beyond the specific data that are collected. In this kind of study, the data are considered to represent a sample—and the goal of the investigation is to make one or more statements about the larger group of which the sample is only a part. Such statements, when based on sample data but designed to extend beyond the sample, are called statistical inferences. Not surprisingly, the term inferential statistics is used to label the portion of statistics dealing with the principles and techniques that allow researchers to generalize their findings beyond the actual data sets obtained.
In this chapter, we consider the basic principles of inferential statistics. We begin by considering the simple notions of sample, population, and scientific guess. Next, we take a look at eight of the main types of samples used by applied researchers. Then we consider certain problems that crop up to block a researcher’s effort to generalize findings to the desired population. Finally, a few tips are offered concerning specific things to look for as you read professional research reports.