The influence of primary research methodology on perceptions of good educational research
Kelly D. Bradley1, Jessica D. Cunningham and Jennifer Eli University of Kentucky
Abstract
This study further investigates the findings of Bradley, Royal, Cunningham, Weber, and Eli (2008), where survey items connected to methodology revealed great variation in perceptions of characteristics of good educational research. The purpose is to examine whether differential response patterns exists across the research methodology variable, which is the respondents’ self-identified primary research methodology. Over half of methodology statements in the survey indicated differential response patterns across the research methodology variable. Responses for two methods items were found significantly different across respondents choosing entirely qualitative, mostly qualitative, and mostly quantitative as their primary research methodology.
*** Preliminary paper; do not cite without permission of lead author
1
The influence of primary research methodology on perceptions of good educational research While innovative research in teacher education is a focal point in teacher education, examination of common characteristics of what constitutes good educational research is vital to maintaining high-quality research. This study utilizes results of a survey about perceptions of good educational research completed by College of Education faculty and graduate students at a southeastern university. Prior to distribution, it was assumed common ground and variation in perceptions of what constitutes good educational research could be identified through results coupled with existing literature. While commonality was found in responses to the ethics and theory survey sections, great variation was demonstrated in survey items connected to
methodology (Bradley, Royal, Cunningham, Weber, & Eli, 2008). This finding spawned the question of whether differential responding to survey items existed depending on the variable classifying respondents by their self-reported primary research methodology.
Theoretical Framework
There is general sentiment that educational research “should involve carefully
constructed designs, while implementing the most appropriate methods for data collection and analysis with the purpose of answering research questions aimed at understanding the world of education” (Bradley, Royal, Cunningham, Weber, & Eli, 2008, p. 27). The quality of educational research is often evaluated on the merits of its methods (techniques or procedures for collecting and analyzing data) and methodology. As Teddlie and Tashakkori (2009) posit:
A research methodology is a broad approach to scientific inquiry specifying how research questions should be asked and answered. This includes worldview considerations, general preferences for designs, sampling logic, data collection and analytical strategies,
guidelines for making inferences, and the criteria for assessing and improving quality. (p. 21)
Within the social and behavioral sciences, broadly speaking, educational researchers can be partitioned into three overarching categories based on methodological orientation –
specifically qualitatively leaning, quantitatively leaning, and mixed methodologists (Creswell, 2003; Creswell & Plano-Clark, 2007; Tashakkori & Teddlie, 1998; Teddlie & Tashakkori, 2009). “Different designs and methods are better for different purposes. No one design is the best or most desirable; multiple types of scientific inquiries and methods are required to generate the rich body of scientific knowledge needed to improve education” (Borko & Whitcomb, 2008, p. 566). Given this, attention must be given to differences among paradigms, methodologies and methods that guide researchers toward conducting good educational research.
Paradigms
According to Morgan (2007) paradigms are a system of “beliefs and practices that influence how researchers select both the questions they study and methods they use to study” (p.49). Researchers’ beliefs, practices and philosophical assumptions on the nature of ontology, epistemology, and axiology play a vital role in shaping the methodology, methods and research questions. Is there a single reality or multiple constructed realities? Is the relationship between the researcher and the subject independent or dependent? Is inquiry value free or value bounded?
Quantitative Methods
The constructivist, positivist or post positivist, and pragmatic paradigms are often associated with the qualitative, quantitative and mixed methods research positions, respectively. Quantitative research methods are most often identified with the positivism or post positivism philosophical orientation as these methods propose measuring and analyzing causal relationships within a framework that may or may not be value-free. Positivists and post-positivists believe knowledge is objective; context-free generalizations are possible; causal links can be isolated and
identified. Furthermore, the positivist framework acknowledges personal bias but strives for separation between researcher and subject (Tashakkori & Teddlie, 1998). A quantitative approach generally involves working with numerical data collected from a probability sample that will be analyzed using statistical techniques. Quantitative researchers typically employ deductive logic and reasoning, which is confirmatory in nature.
Qualitative Methods
On the other end of the spectrum, qualitative research methods are often identified with the constructivist philosophical orientation as these methods lend themselves to involve
analyzing descriptive or narrative data. Frequently, an inductive approach is applied. The constructivist framework supports the idea of multiple realities, relationship between researcher and subject being dependent, knowledge as negotiated, value-bounded inquiry, and context-free generalizations as impossible (Tashakkori & Teddlie, 1998, p. 6). The purpose of qualitative methods is often exploratory in nature.
Mixed Methods Research
Considered a middle between quantitative and qualitative research methods, mixed methods research is often identified with the pragmatic philosophical orientation, placing emphasis on a “what works best” approach, supporting both singular and multiple realities (Creswell & Plano-Clark, 2007). The pragmatic approach has “multiple stances” (p. 24) when it comes to axiology to “include both biased and unbiased perspectives” (p. 24), allowing for the use of mixed methods in educational research.
A mixed methods study involves the collection or analysis of both quantitative and/or
qualitative data in a single study in which the data are collected concurrently or
sequentially, are given a priority, and involve the integration of the data at one or more stages in the research process. (Creswell, Plano-Clark, Gutmann & Hanson, 2002, p. 212)
Mixed method research involves the use of both narrative and numeric data; employment of both inductive and deductive logic; integration of categorical and contextual data analysis; and both confirmatory and exploratory in nature.
Research Question
As Feuer, Towne, & Shavelson (2002) argue, “No method is good, bad, scientific, or unscientific in itself: Rather, it is the appropriate application of method to a particular problem that enables judgments about scientific quality” (p. 8). Regardless of researchers’ philosophical orientation the choice of method(s) and methodology should be driven by the research questions which provide organization, focus, coherence, relevance and direction for conducting good educational research (Onwuegbuzie & Leech, 2006). Questions help narrow the scope of the research objective and purpose. In addition, Onwuegbuzie & Leech make the case that research questions “dictate the type of research design used, the sample size and sampling scheme employed, and the type of instruments administered as well as the data analysis techniques” (p. 475). Teddlie and Tashakkori (2009) describe the research question as a dual focal point that “liaises between what is known about the topic before the study and what is learned about the topic during the study. Everything flows through and from the research questions.” (p. 129). Thus, careful consideration for alignment of research questions to primary research methodology is vital for ensuring high quality education research.
Method
This study further investigates the findings of Bradley, Royal, Cunningham, Weber, and Eli (2008), where survey items connected to methodology revealed great variation in perceptions of characteristics of good educational research. The purpose of this study is to explore whether this variation existed across the research methodology variable – respondents’ self-identified
primary research methodology. Specifically, is differential item functioning occurring within the category of primary research methodology?
Response Frame
The target population consisted of 85 faculty and approximately 250 graduate students. The response frame consisted of 126 faculty (n=50; 39.7%) and graduate students (n=76; 60.3%) from the COE. The university’s COE includes programs departmentalized in six areas of
educational research and practice, including: curriculum and instruction; educational leadership studies; educational and counseling psychology; educational policy and evaluation; kinesiology and health promotion; and special education and rehabilitation counseling. A demographic description is presented below for both faculty (see Table 1) and graduate students (see Table 2).
Table 1 Demographic Characteristics of Faculty Respondents (n = 50)
Characteristic n % Faculty Rank Instructor 3 5.9 Lecturer 0 0 Assistant Professor 15 29.4 Associate Professor 12 23.5 Professor 13 25.5 Visiting Professor 2 3.9 Adjunct Professor 1 2.0 Part-time Professor/Instructor 2 3.9 Other 3 5.9
Years as a faculty member (at any higher education institution)
Less than 1 year 6 12.0
1-3 years 12 24.0
4-6 5 10.0
7-9 3 6.0
10-15 7 14.0
16-20 4 8.0
Table 1 (continued)
Characteristic n %
Tenure Status
Tenured 27 52.9
On tenure track, but not tenured 13 25.5
Not on tenure track 11 21.6
Age 71 or older 2 4.3 61-70 6 12.8 51-60 12 25.5 41-50 7 14.9 31-40 15 31.9 Less than 30 5 10.6
Table 2 Demographic Characteristics of Graduate Student Respondents (n = 76)
Characteristic n % Status Graduate Assistant 3 4.1 Research Assistant 19 25.7 Teaching Assistant 14 18.9 Doctoral Student 39 52.7 Master’s Student 8 10.8
Specialist Degree Student 8 10.8
Other 12 16.2
Years as a graduate student (at any institution)
Less than a year 8 11.0
Between 1-2 years 14 19.2 Between 3-4 years 21 28.8 Between 5-6 years 18 24.7 7 or more years 12 16.4 Age Under 20 0 0 20-24 12 16.0 25-29 23 30.7 30-34 10 13.3 35-39 10 13.3 40-44 5 6.7 45-49 5 6.7 50-54 6 8.0 55 or older 4 5.3
The majority of the faculty sample consists of full-time professors distributed across the categorizations of assistant, associate, and professor. Half of the faculty members have been a faculty member at a higher education institute for either 1-3 years or greater than 20 years, with age ranges of either 31-40 or 51-60, and a majority with tenured status. The graduate student sample consists of a majority of doctoral students that are either research or teaching assistants. The majority of the students have been a graduate student for more than three years with ages ranging from 20-39. With regard to the collective sample, 89% (n = 105) of respondents reported being of White/Caucasian ethnicity. Asian and Pacific Islanders comprised 5.9% and African American or Black respondents accounted for 5.1%.
Data Collection
Using a list of emails provided by the dean’s office for all College of Education faculty and graduate students, an invitation to participate in the study was sent to the entire population. The email invitation included the link to the survey and statements regarding purpose and participant rights. Three reminders were subsequently sent, including final notice of a closing date, which was two and a half weeks following the opening date. Responses were collected and stored on a secure web server provided by the survey software program SurveyMonkey (2007). All data were downloaded onto the primary investigator’s computer in aggregate form to ensure respondents were unidentifiable.
Instrumentation
The survey instrument, in its entirety, consisted of 60 items, including demographics. Research statements (see Table 6, appendix) were partitioned into three domains: methodological (17 items), ethical (10 items) and theoretical (12 items). These statements were created from guidelines for “good” or “quality” research set forth by research organizations in educational
disciplines, such as the American Educational Research Association (AERA) and the American Psychological Association (APA). Items utilized a four-point rating scale to endorse level of agreement. In the demographic section, respondents were asked to select the response category best describing their primary research methodology (entirely qualitative, mostly qualitative, mixed methods, mostly quantitative, entirely quantitative).
Table 3 Primary Academic Interest for Faculty and Graduate Student Respondents (n = 137)
Demographic Variable n %
Primary Research Methodology
Entirely qualitative 4 3%
Mostly qualitative 25 21%
Mixed Methods (50/50) 41 35%
Mostly Quantitative 31 26%
Entirely Quantitative 17 14%
Out of the 137 respondents completing the survey, the only category resulting in less than ten respondents for primary research methodology was entirely qualitative.
Analysis
Crucial to determining common characteristics of good educational research is the quality of the instrument used to examine participant responses. In an effort to ensure the quality of the measure, a rating scale model was applied with persons and items to test the overall fit of the data to the model using Winsteps version 3.65.0 software (Linacre, 2006). Missing data were treated as missing, with no values being imputed. The stability of the measurement instrument follows the procedures outlined in Bradley, et al. (2008).
One of the more straightforward approaches to assessing construct bias, particularly for rating scales is the Rasch model (Wright & Masters, 1982; Selner-O’Hagan, Kindlon, Buka, Raudenbush, & Earls, 1998). As with all IRT models, the Rasch model assumes that an additive structure underlies the observed data, that both participants and items can be arrayed on a continuum, and that the items have equal discriminative power (Kan, Breteler, Van der Ven, & Zitman, 1998). Rasch measurement allows for analyses of individual differences in response tendencies, as well as an item’s discrimination (i.e., how well the item is able to discriminate between examinees holding different levels of a latent construct) and an item’s difficulty. Rasch scaling procedures are used to determine equivalence at the item level, and if differences are obtained, to determine the pattern of responding across groups of respondents.
Each person is accompanied with a person label indicating their reported primary
research methodology. Relative frequency tables illustrate a general picture of the characteristics the respondents associated with good educational research separated by primary research
methodology. Differential item analysis within WINSTEPS version 3.65.0 was used to determine if there are statistically significant differences in characteristics of good research across the three domains by self-identified primary research methodology. A separate calibration t-test approach (Wright and Stone, 1979) was used to determine differential response across the subpopulations. While there is not a consensus on statistical significant differences, Smith (2004) suggested using t-statistic estimates greater than 2 or less than -2 for detecting differential item responding.
Results and Discussion
The response categories provided on the survey for respondents to classify their primary research methodology included entirely qualitative, mostly qualitative, mixed methods, mostly quantitative, and entirely quantitative. Participants were asked to select only one. The forced
choice response was difficult for some respondents, as they did not feel comfortable self-classifying. Even so, these categories were selected to construct the boundaries often presented in the literature, the typical quantitative/qualitative split, with mixed methods serving as the balancing point.
To begin the discussion, frequency tables were produced to illustrate the overall picture of item responses by rating scale category as well as separated by primary research methodology. Seven out of ten ethics items, five out of twelve theory items, and nine out of seventeen methods items had less than 20% disagreeing with the statement.
Table 4 Counts and percents of each rating scale category for methods survey items
Item SD D A SA M1 4 (3%) 3 (2%) 59 (43%) 70 (52%) M2 5 (4%) 51 (38%) 59 (44%) 20 (15%) M3 5 (4%) 11 (8 %) 53 (39%) 68 (50%) M4 3 (2%) 39 (28%) 95 (69%) M5 5 (4%) 29 (21%) 66 (49%) 36 (26%) M6 2 (1%) 29 (21%) 106 (77%) M7 2 (1%) 46 (34%) 88 (65%) M8 1 (1%) 2 (1%) 43 (32%) 89 (66%) M9 2 (1%) 39 (28%) 96 (70%) M10 6 (4%) 35 (26%) 58 (43%) 37 (27%) M11 2 (1%) 2 (1%) 54 (40%) 76 (57%) M12 37 (27%) 70 (51%) 22 (16%) 7 (5%) M13 27 (20%) 60 (44%) 42 (31%) 7 (5%) M14 24 (18%) 63 (47%) 41 (30%) 7 (5%) M15 29 (21%) 92 (68%) 13 (10%) 2 (1%) M16 1 (1%) 1 (1%) 80 (59%) 54 (40%) M17 1 (1%) 38 (29%) 58 (44%) 36 (27%)
Table 5 Counts and percents of each rating scale category for ethics survey items Item SD D A SA E1 1 (1%) 23 (18%) 104 (81%) E2 3 (2%) 42 (33%) 83 (65%) E3 23 (18%) 106 (82%) E4 2 (2%) 11 (9%) 60 (47%) 54 (43%) E5 2 (2%) 9 (7%) 48 (38%) 69 (54%) E6 1 (1%) 5 (4%) 48 (38%) 73 (57%) E7 14 (11%) 60 (47%) 53 (42%) E8 17 (13%) 68 (54%) 42 (33%) E9 14 (11%) 83 (65%) 30 (24%) E10 56 (43%) 73 (57%)
Table 6 Counts and percents of each rating scale category for theory survey items
Item SD D A SA T1 4 (3%) 67 (53%) 55 (44%) T2 1 (1%) 3 (2%) 60 (48%) 62 (49%) T3 1 (1%) 8 (6%) 65 (51%) 53 (42%) T4 1 (1%) 31 (24%) 63 (50%) 32 (25%) T5 3 (2%) 21 (17%) 70 (56%) 32 (25%) T6 3 (2%) 33 (26%) 73 (57%) 18 (14%) T7 17 (14%) 79 (63%) 29 (23%) T8 2 (2%) 39 (31%) 68 (54%) 17 (13%) T9 2 (2%) 73 (57%) 52 (41%) T10 4 (3%) 70 (55%) 53 (42%) T11 2 (2%) 13 (10%) 79 (62%) 33 (26%) T12 10 (8%) 88 (72%) 25 (20%)
All respondents either agreed or strongly agreed to ethics items 3 and 10. In contrast, methods item 12 resulted in less than 20% endorsing the statement. For a list of the full statement associated with each item, please see appendix.
Discussing the fit of the data to the Rasch rating scale model is essential prior to ensure meaningful interpretations of differential item functioning across primary research
methodological classifications. The person separation (2.48) and reliability (0.86) estimates were deemed acceptable for the analysis to continue; likewise, the items resulted in acceptable item separation (7.64) and reliability (0.98) estimates. According to Wright and Stone (2004), fit statistics above one standard deviation of the infit and outfit mean are labeled as item misfit. Nine of the ten items resulting in high fit statistics were methods items, indicating unexpected responses are being produced by these items. Items 2, 3, 10, 12, 13, 17, and ethics item 4 resulted in a high outfit statistic while items 1, 3, 4, 6, 10, 12, 13, and 17 resulted in high infit statistics. Methods item 11, ethics item 1, and theory items 2 and 11 resulted in a decrease in the average person measure over the rating scale categories, but this was a result of one or two respondents disagreeing with this item unexpectedly. More problematic decreases in average person measures over the rating scale occurred for items 4 and 6 in ethics, 4 in theory, and 1, 2 and 10 in methods.
Finally, items are inspected for differential item functioning across the variable
classifying respondents by their reported primary research methodology. The second variable of primary research methodology resulted in several significant classes differentially responding across items. Respondents not reporting their primary research methodology were found significantly different for methods items 6, 8, 12, 13, 14, 16 and ethics items 3 and 9. Respondents choosing entirely qualitative and mostly qualitative were found significantly different for methods items 3 and 13. Respondents choosing entirely quantitative were
significantly different for methods items 3, 10, and 13. Mostly quantitative respondents were significantly different for methods item 2. A frequency table was produced for the items
resulting in differential item functioning to depict the distribution of responses across the rating scale by self-identified primary research methodology.
Table 7 Counts of rating scale category by primary research methodology for DIF methods items
Item Method SD D A SA M2 1 2 3 4 5 0 3 0 1 1 0 5 12 19 8 3 11 24 8 3 1 6 4 3 5 M3 1 2 3 4 5 0 2 2 0 0 2 6 2 1 0 2 8 15 16 3 0 9 22 14 14 M6 1 2 3 4 5 0 1 1 0 0 1 4 11 3 1 3 20 29 28 16 M8 1 2 3 4 5 0 0 1 0 0 0 1 0 0 0 2 7 6 12 7 2 17 34 18 10 M10 1 2 3 4 5 0 1 1 2 2 0 6 11 8 5 3 12 16 13 7 1 6 13 8 3
Table 7 (continued) M12 1 2 3 4 5 2 9 11 10 3 1 10 25 19 10 1 5 3 2 3 0 1 2 0 1 M13 1 2 3 4 5 3 8 9 4 1 1 13 22 19 2 0 4 9 8 11 0 0 1 0 3 M14 1 2 3 4 5 3 9 4 5 2 0 10 25 16 8 1 5 11 7 5 0 1 1 2 2 M16 1 2 3 4 5 0 0 1 0 0 0 0 0 0 0 3 14 22 18 9 1 11 18 13 8
Table 8 Counts of rating scale category by primary research methodology for DIF ethics items
Item Method SD D A SA E3 1 2 3 4 5 1 4 3 7 2 3 21 38 24 15 E9 1 2 3 4 5 1 2 4 0 3 3 20 23 23 9 0 3 14 8 5
Conclusion
The intent of educational research is to advance knowledge in the field through inquiry into the world of education. With much agreement within the domain of ethics and theory and considerable variability within the methodology domain, it seemed reasonable to examine the differential item functioning within each domain across the respondents’ reported primary research methodology. Evidence from this study supports the lack of consensus among educational researchers of what constitutes good research. Sadly, it also demonstrates that the great divide of methodological framework is evident – a qualitative/quantitative divide of sorts. Ultimately, the goal is for this research to improve the overall quality of educational research by encouraging researchers to be mindful of characteristics of their own research and hopefully, to be more receptive to the varying perceptions of their peers.
References
Bradley, K.D., Royal, K.D., Cunningham, J.D., Weber, J., & Eli, J.A. (2008). What constitutes good educational research? A consideration of ethics, methods and theory. Mid-Western
Educational Researcher, 21(1), 26-35.
Borko, H., & Whitcomb, J.A. (2008). Teachers, teaching, and teacher education: Comments on the national mathematics advisory panel’s report. Educational Researcher, 37(9), 572.
Creswell, J.W. (2003). Research design: Qualitative, quantitative, and mixed methods
approaches (2nd Ed.). Thousand Oaks, CA: SAGE Publications, Inc.
Creswell, J.W., & Plano-Clark, V.L. (2007). Designing and conducting mixed methods research.
Creswell, J.W., Plano-Clark, V.L., Gutmann, M., & Hanson, W.E. (2002). Advanced mixed methods research designs. In A. Tashakkori, & C. Teddlie (Eds.), Handbook of mixed
methods in social and behavioral research (pp. 209-240). Thousand Oaks, CA: SAGE
Publications, Inc.
Feuer, M.J., Towne, L., & Shavelson, R.J. (2002). Scientific culture and educational research.
Educational Researcher, 31(8), 4-14.
Morgan, D. (2007). Paradigms lost and pragmatisms regained: Methodological implication of combining qualitative and quantitative methods. Journal of Mixed Methods Research, 1,
48-76.
Onwuegbuzie, A.J., & Leech, N.L. (2006). Linking research questions to mixed methods data analysis procedures. The Qualitative Report, 11(3), 474-498.
Tashakkori, A., & Teddlie, C. (1998). Mixed methodology: Combining qualitative and
quantitative approaches. Thousand Oaks, CA: SAGE Publications, Inc.
Teddlie, C., & Tashakkori, A. (2009). Foundations of mixed methods research: Integrating
quantitative and qualitative approaches in the social and behavioral sciences. Thousand
Appendix of Items Ethics
E1 High-quality research abides by ethical standards.
E2 High-quality research informs participants about the consent to research. E3 High-quality research should protect the safety and welfare of participants.
E4 High-quality research minimizes the use of techniques or methodologies that have negative social consequences.
E5 High-quality research adheres to established institutional policies for conducting research.
E6 High-quality research recognizes all researchers who have contributed substantively to
the study.
E7 High-quality research should only be conducted by investigators who have completed
ethics training.
E8 High-quality research findings must be disseminated to the professionals within the discipline.
E9 Results from high-quality research should be disseminated to the public.
E10 High-quality research should abide by the ethical guidelines recognized by the related professional organizations in that field.
Theory
T1 High-quality research reflects the researchers' awareness of both their own and
competing paradigms.
T2 High-quality research should provide the rationale for the conceptual orientation of the study.
T3 High-quality research should provide the rationale for the theoretical orientation of the study.
T4 Educational researchers should adhere to the standards of their own theoretical perspectives to achieve high-quality research.
T5 Reliability, validity and trustworthiness are the most important considerations in high-quality research.
T6 High-quality research provides objective answers to research questions. T7 High-quality research assists in developing theories to explain phenomena. T8 High-quality research aims to develop generalizations based on findings. T9 High-quality research adds to our understanding of the issues in education.
T10 High-quality research should be useful to other professionals within the discipline. T11 High-quality research connects the work to its impact on human well-being. T12 High-quality research merges reason and value.
Methods
M1 High-quality research should be evidence-based. M2 High-quality research should be original.
M3 High-quality research should be reproducible. M4 High-quality research should be attentive to detail.
M5 High-quality research should consider efficiency in choosing research methodology. M6 High-quality research should be methodologically sound.
M7 High-quality research should follow a clear logic of inquiry.
M8 High-quality research should be mindful of differences within the research population (e.g. cultural, religious, gender, etc.).
M9 High-quality research should demonstrate awareness that different types of research call for different data collection techniques.
M10 The appropriate methods ensure the high-quality of research data.
M11 High-quality research requires research methods and techniques based on the nature of the research questions.
M12 High-quality research requires random sampling.
M13 High-quality research requires quantifiable measures of results.
M14 High-quality research consists of experimental studies that yield prescriptions for action.
M15 High-quality research can be determined solely by examining the research methodology.
M16 High-quality research should provide the rationale for the methodological orientation of
the study.
M17 Educational researchers should adhere to the standards of their own methodological perspectives to achieve high-quality research.
