The study's main purpose was to develop online education to improve people's use of Web-based health information. The investigators con
vened five focus groups and conducted in-depth interviews with
15
people to identify preferences for learning. They asked participants questions about the value of audio and video. Using the information from the groups and interviews, the investigators developed the education and observed its usability and in a small sample. Once they had evidence that the education was probably ready for use in a larger population, they evaluated its effectiveness by using statistical methods to compare the knowledge, self-efficacy, and Internet use among two groups, one of which received the new education while the other used an already exist
ing on line program.
Mixed methods research reviewers have a daunting task. They must review the reliability and validity of each method's application to the research questions and also appraise if and how well the findings were combined to yield accurate and relevant results and conclusions.
The following is a selected list of mixed methods research studies.
Biasutti, M., & El-Deghaidy, H. (2012). Using Wiki in teacher education: Impact on knowledge management processes and student satisfaction.
Computers
&Education,
59(3), 861-872.Christ, T., Arya, P., & Chiu, M. M. (2012). Collaborative peer video analysis.
Journal of Literacy Research, 44(2), 171-199.
Coles, E., Themessl-Huber, M., & Freeman, R. (2012). Investigating community
based health and health promotion for homeless people: A mixed methods review.
Health Education Research,
27(4), 624-644. doi: 10.1093/her/cys065DeCuir-Gunby, J. T., Marshall, P. L., & McCulloch, A. W. (2012). Using mixed methods to analyze video data. Journal of Mixed Methods Research, 6(3), 199-216. doi: 10.1177/1558689811421174
Hussaini, K., Hamm, E., & Means, T. (2012, December 11 ). Using community
based participatory mixed methods research to understand preconception health in African American communities of Arizona. Maternal and Child
Health Journal. doi: 10.1007/s10995-012-1206-5
Marczinski, C. A., & Stamates, A. L. (2012). Artificial sweeteners versus regu
lar mixers increase breath alcohol concentrations in male and female social drinkers. Alcoholism: Clinical and Experimental Research. doi: 10.1111 I acer.12039
McEwen, A., Hackshaw, L., Jones, L., Laverty, L., Amos, A., & Robinson, J. (2012).
Evaluation of a programme to increase referrals to stop-smoking services using Children's Centres and smoke-free families schemes. Addiction, 107(2), 8-17.
Tan, S. B., Williams, A. F., & Morris, M. E. (2012). Experiences of caregivers of people with Parkinson's disease in Singapore: A qualitative analysis. Journal of Clinical Nursing, 2 7 (15-16), 2235-2246.
Ungar, M., & Liebenberg, L. (2011 ). Assessing resilience across cultures using mixed methods: Construction of the child and youth resilience measure. Journal of Mixed Methods Research, 5(2), 126-149. doi: 10.1177/1558689811400607 Yu, S. (2012). College students' justification for digital piracy : A mixed methods
study. Journal of Mixed Methods Research, 6(4), 364-378.
Zander, K., Stolz, H., & Hamm, U. (2013). Promising ethical arguments for prod
uct differentiation in the organic food sector. A mixed methods research approach. Appetite, 62, 133-142. doi: 10.1016/j.appet.2012.11.015
SUMMARY OF KEY POINTS
• Researchers collect data by administering achievement tests, survey questionnaires, and face-to-face and telephone interviews; analyzing large databases or vital statistics; observing individuals and groups;
reviewing the literature and personal, medical, financial, and other statistical records; performing physical examinations and laboratory tests; and using simulations and clinical scenarios or performance tests.
• No single method of collecting data is inherently better or has more quality than another. Usually, data collection methods are chosen for their practicality as well as for their quality. For the literature reviewer, the deciding factor in determining the quality of a study's
data collection is not the method itself but whether it provides reliable and valid information.
• A reliable data collection method is one that is relatively free from
"measurement error." Because of this error, individuals' obtained scores are different from their true scores. Types of reliability include the following:
Test-retest reliability. A measure has test-retest reliability if the correla
tion between scores from time to time is high. The major conceptual difficulty in establishing test-retest reliability is in determining how much time is permissible between the first and second administration. If too much time elapses, external events might influence responses for the second administration; if too little time passes, the respondents may remember and simply repeat their answers from the first administration.
Equivalence or alternate-form reliability. This type refers to the extent to which two assessments measure the same concepts at the same level of difficulty. As an alternative to establishing equivalence between two forms of the same measure, researchers sometimes compute a split-half reliability. To do this requires dividing a measure into two equal halves (or alternate forms) and obtaining the correlation between the two halves.
Homogeneity. This kind of reliability refers to the extent to which all items or questions assess the same skill, characteristic, or quality.
Sometimes this type of reliability is referred to as internal consistency.
Cronbach's coefficient alpha, which is basically the average of all the correlations between each item and the total score, is often calculated to determine the extent of homogeneity.
Interrater. This type of reliability refers to the extent to which two or more individuals agree.
Intrarater. This type reliability refers to a single individual's consis
tency of measurement, and this, too, can be enhanced by training, monitoring, and continuous education.
• Validity refers to the degree to which a measure assesses what it purports to measure. At least four types of validity are commonly discussed.
Content validity refers to the extent to which a measure thoroughly and appropriately assesses the skills or characteristics it is intended to measure.
Face validity refers to how a measure appears on the surface: Does it seem to ask all the needed questions? Does it use the appropriate lan
guage and language level to do so? Face validity, unlike content valid
ity, does not rely on established theory for support.
Criterion validity is made up of two subcategories: Predictive validity and concurrent validity. Predictive validity refers to the extent to which a measure forecasts future performance. Concurrent validity is demon
strated when two assessments agree or a new measure is compared favorably with one that is already considered valid.
Construct validity is established experimentally to demonstrate that a measure distinguishes between people who do and do not have certain characteristics.
• The appropriateness of each data-analytic method depends on whether the independent variable is measured on a categorical, ordinal, or numerical scale; the number of independent variables; whether the dependent variable is measured on a categorical, ordinal, or numerical scale; the number of dependent variables; and whether the quality and characteristics of the data meet the assumptions of the statistical method.
• Watch for study results that gloss over negative findings.
• Be wary of studies that gloss over findings for the main sample and, instead, provide results on subgroups.
• Check that a study's conclusions come directly from the data collected by the study's researcher.
• Check to make certain that the study's methodological limitations are dis
cussed so that you can judge how much confidence to place in the findings.
• Check editorials and letters to the editor of the publication in which the study appears to make certain that major methods and conclusions are not being challenged.
• Qualitative research takes place in natural social settings rather than in the controlled environments associated with experimental research.
Oriented primarily toward exploration, as well as discovery and induc
tion, this type of research often results in individuals' own accounts of their attitudes, motivations, and behavior.
• When reviewing the literature, you should not focus on whether a study is qualitative or quantitative but concentrate instead on its accuracy and the value of its findings.
• The following checklist can be used to evaluate the presentation and qual
ity of qualitative research. The checklist should be regarded as a supple
ment to the usual criteria for evaluating the quality and value of a study .
./ The researchers should describe their traditions of inquiry .
./ The data collection methods must be reliable and valid and accom-panied by supporting evidence of their accuracy .
./ The study should contain proof of a rigorous research design . ./ Sound sampling methods should be explicit.
./ The analysis methods must be carefully explained.
• Mixed methods is a type of research characterized by collecting, ana
lyzing, and integrating qualitative and statistical or quantitative data into a single study. Two common uses for mixed methods are to assist researchers in better understanding an experimental or quantitative study's findings and to incorporate users' perspectives into the develop
ment and evaluation of a program, service, or policy.
EXERCISES
1.
Read the following excerpts from study reports and tell which concepts of reliability and validity are covered.
a. The self-administered questionnaire was adapted with minor revi
sions from the Student Health Risk Questionnaire, which is designed to investigate knowledge, attitudes, behaviors, and other cognitive variables regarding HIV and AIDS among high school