candidates/teachers, during their preservice period and first two years of teaching, think about authentic intellectual work, as a means of promoting the type of learning consistent with the theoretical frame of democratic education, when they create assessments and when they talk about pupil learning, and the extent to which their pupils engage in authentic intellectual work. This dissertation, which draws from a larger longitudinal qualitative case studies project, utilized a qualitatively-driven, concurrent embedded mixed methods approach (Creswell et al., 2003; Morse & Niehaus, 2009) and cross-case study methodology (Stake, 1994) to understand how 11 teacher candidates/teachers did this.
In particular, as a means to examine teacher candidates’ use of authentic intellectual work, this study used the Teacher Assessment/Pupil Learning (TAPL) protocol, a research instrument where participants took part in an interview about assessment and pupil learning in relation to specific samples of assessment
tasks/assignments and pupils’ work that they used in their classroom. This protocol also features an outside examination of these samples of assessments and pupil work. The TAPL protocol, described in detail in a later section, was used with participants in this study at five different points in time during their preservice period and first two years of teaching. This study used an interpretive qualitative approach to explore TAPL interview data to discern how participants thought about concepts related to authentic intellectual work when they talked about assessment and pupil learning and a quantitative rubric to
94
evaluate the quality of authentic intellectual work in the teacher candidates’/teachers’ assessment tasks and pupil work samples. Qualitative and quantitative analyses were integrated to construct a nuanced description and understanding of the extent to which teacher candidates/teachers and their pupils engaged in rigorous authentic intellectual work and how context and conditions influenced this work as beginning teachers developed in the early phase of their career.
This section presents an overview of mixed methods research and describes particular elements of mixed methods research that are related to the dissertation, including qualitatively-driven, concurrent, and embedded designs, followed by an
overview of case study methodology. Next, drawing on the framework of mixed methods research, I describe the research design for this study detailing the larger qualitative case studies project from which this study derives, study participants, and data collection and analysis strategies. I conclude with a discussion of the integrity of the study, considering the issues of rigor, reflexivity, validity, and limitations.
Overview of Mixed Methods Research
To explore how teacher candidates/teachers reflect the ideas of authentic intellectual work in their creation of assessments and understandings of pupil learning, this study utilized a mixed methods approach. This study drew upon data collected during a larger, longitudinal qualitative case studies project and used interpretive qualitative techniques to analyze qualitative interview data from that study. At the same time, this study used a quantitative approach to analyze artifacts collected from the larger study and
95
employed statistical analyses to examine relationships between variables and variation over time.
Mixed method research has been called the “third methodological movement” (Tashakkori & Teddlie, 2003, p. ix) in social science research and is a methodology that draws upon and combines the two general paradigms that preceded the movement: quantitative and qualitative methods. Quantitative and qualitative researchers operate with differing and competing assumptions concerning research ontology, epistemology, and methodology (Guba & Lincoln, 1994), and mixed methods research has been seen, by some, as a way to bridge the two paradigms in a way that takes advantage of the strengths of both traditions to answer new types of research questions and create new understandings.
Qualitative and Quantitative Methods
It has been argued that identifying research by the terms “quantitative” and “qualitative” is a simplistic, and not a very useful, distinction (Howe, 1988). However, in a very general sense, these terms have common and particular meanings. Quantitative research comes from a historical tradition linking science with quantification (Guba & Lincoln, 1994), where social science researchers worked predominantly within the positivist traditions that drew upon numerical analyses and “objective” measurements (Teddlie & Tashakkori, 2003). Scientific objectivity and the construction of replicable research designs are set as ideals for many quantitative researchers (Guba & Lincoln, 1994). Quantitative research is typically seen as using a deductive approach to test the relationship between variables and confirm previous theory (Creswell, 2009; Tashakkori
96
& Teddlie, 2003). Methods used in quantitative research often include experimental design, statistical analyses and interpretation, and instruments, such as surveys, featuring closed-response items or likert scales (Creswell, 2009).
Qualitative research, on the other hand, operates under the assumption that research is not objective and is dependent upon more than just numerical analyses. With this tradition, qualitative researchers see research as value-laden and “stress the socially constructed nature of reality, the intimate relationship between the researcher and what is studied, and the situational constraints that shape inquiry” (Denzin & Lincoln, 1994, p. 4). Also referred to as interpretivist research, qualitative researchers operate under the assumption that it is important to understand the meanings behind people’s actions and beliefs and “that to understand this world of meaning one must interpret it” (Schwandt, 1994, p. 118). In qualitative research the researcher is seen as the instrument, constantly negotiating a relationship between researcher and participant, and research takes place in a natural setting (Denzin, 1994). There are many different forms of qualitative research methods ranging from grounded theory (Charmaz, 2004; Glaser, 1992; Glaser & Strauss, 1967), to ethnography (Eisenhart, 2001; Wolcott, 1999), to phenomenology (Benner, 1994). Qualitative data includes experiencing, enquiring, and examining (Wolcott, 1992) which often take the form of observations, interviews, and material culture. Qualitative research is sometimes seen as exploratory (Teddlie & Tashakkori, 2003) where themes emerge from the data as researchers organize data through an iterative process to construct inferences and generalizations (Creswell, 2009).
97
Mixed Methods
Although quantitative and qualitative approaches are sometimes considered to be dichotomous, mixed methods research attempts to combine the two. Due to the inherent tensions between quantitative and qualitative perspectives, researchers disagree over the degree to which mixed methods can, or even should, be used as a sound research
methodology. Teddlie and Tashakkori (2003) identify six stances towards mixed methods research: 1) methods and paradigms are separate from one another, which makes mixed methods possible because they are unrelated; 2) qualitative and quantitative paradigms are incompatible which makes mixed methods impossible; 3) methods must be kept separate and remain true to their paradigms, but they can work together to answer a research question; 4) one single method can serve as the foundation for mixed methods; 5) different paradigms should engage with one another to examine the tensions between the two; and 6) multiple paradigms can be used, depending on the type of study.
Although some researchers continue to believe in the incompatibility stance, that view “has now been largely discredited, partially because scholars demonstrated that they had successfully employed mixed methods in their research” (Teddlie & Tashakkori, 2003, p. 19). At the other extreme, some researchers advocate for the dissolution of the
identification of research in quantitative or qualitative terms since “all research is interpretive” (Schwandt, 2000, p. 210).
Researchers tend to agree that the most important factor related to choosing a research methodology is the research question (Guba & Lincoln, 1994; Teddlie & Tashakkori, 2003) and that therefore, research methods should be chosen based on the
98
most appropriate way to answer a research question. From this perspective, mixed
methods research is often associated with the philosophy of pragmatism as a practical and applied stance to tackle research questions that either quantitative or qualitative methods alone are unable to address (Howe, 1988; Johnson, Onwuegbuzie, & Turner, 2007; Tashakkori & Teddlie, 1998). Pragmatists suggest that qualitative and quantitative methods are compatible (Howe, 1988). Howe (1988) argues there is a “two-way relationship between methods and paradigms” which makes it possible for methods to inform paradigms and thereby makes the two methods compatible (p. 10).
Although there have been different terms used to describe this combining of methods, such as blended research, multimethod, triangulated studies, multiple method, and mixed research (Creswell, Plano Clark, Gutmann, & Hanson, 2002; Johnson et al., 2007; Schwandt, 2000; Teddlie & Tashakkori, 2003), the term “mixed methods” has become the most commonly used. However, a common definition of mixed methods has been more difficult to come by. In its most basic form, mixed methods research
“intentionally combines different methods” (Greene & Caracelli, 1997, p. 7) and includes combining the “qualitative and quantitative viewpoints, data collection, analysis, [and] inference techniques” (Johnson et al., 2007, p. 123). Mixed methods research “also involves the use of both approaches in tandem so that the overall strength of a study is greater than either qualitative or quantitative research” (Creswell, 2009, p. 4). Building on an analysis of the various definitions that researchers have used, Johnson,
Onwuegbuzie, and Turner (2007) define mixed methods as such:
Mixed methods research is an intellectual and practical synthesis based on qualitative and quantitative research…Mixed methods research is the research
99
paradigm that (a) partners with the philosophy of pragmatism…(b) follows the logic of mixed methods research (including the logic of the fundamental principle and any other useful logics imported from qualitative or quantitative research that are helpful for producing defensible and usable research findings); (c) relies on qualitative and quantitative viewpoints, data collection, analysis, and inference techniques combined according to the logic of mixed methods research to address one’s research question(s); and (d) is cognizant, appreciative, and inclusive of local and broader sociopolitical realities, resources, and needs. Furthermore, the mixed methods research paradigm offers an important approach for generating important research questions and providing warranted answers to those questions. (p. 129)
Therefore, mixing methods provides an opportunity to address research questions in a more complete and powerful way where researchers can both answer questions and generate theory. Greene, Caracelli, and Graham (1989) also point to other purposes of mixed methods research including triangulation to see how different methods answer the same question; complementarity to add to one method with another; development to use one method to inform another; initiation to explore questions; and expansion to add to the field by looking at the topic in a new way. In further defining the field, mixed methods researchers have created nomenclature to refer to the different methods, adopting “qual” for qualitative methods and “quan” for quantitative methods (Tashakkori & Teddlie, 2003).
There are different types of mixed methods research designs, depending primarily upon how the quantitative and qualitative portions work in tandem, particularly in terms of data collection and data analysis. Data design in mixed methods does not have to follow one particular paradigm (Onwuegbuzie & Teddlie, 2003) and the mixing of methods can occur in a single study (Creswell et al., 2002) or across related studies (Johnson et al., 2007). In sequential mixed methods, one method is used to collect and
100
analyze data, the results of which then inform the construction of a new study with the second method (Morse, 1991; Tashakkori & Teddlie, 1998). An example of sequential design is a study where a quantitative survey is given and analyzed and then followed by interviews with a few participants about their responses to further clarify issues related to the survey. In mixed methods nomenclature this is represented by an arrow (i.e., quan → qual). Another example of a sequential design is a study that conducts qualitative
interviews with a focus group and uses that work to inform the construction of a survey; this type of design would be designated qual → quan. It should be noted, however, that Morse and Niehaus (2009) portray qual → quan designs as more advanced than simply using an interview to inform a survey since that practice is an accepted and common component to instrument design; instead, they argue, a qual → quan design should involve either collecting new data for the quantitative piece or transforming qualitative data into a form that can then be quantified. In concurrent mixed methods studies (Creswell et al., 2003), also referred to as parallel or simultaneous mixed methods (Tashakkori & Teddlie, 1998), qualitative and quantitative data are collected and/or analyzed at the same time. For example, a survey might include close- and open-ended responses which are then analyzed with quantitative and qualitative methods respectively. This is represented by a “+” (i.e., quan + qual).
As a developing field, mixed methods research designs have taken a variety of approaches to integrating the qualitative and quantitative processes. Creswell, Plano Clark, Gutmann, and Hanson (2003) argue that researchers have integrated qualitative and quantitative approaches at various points in time including “[d]uring the phases of
101
problem/question specification, data collection, data analysis, and interpretation” (p. 220). Typically in mixed methods research, integration occurs at the analysis and interpretation phases. In fact, Morse and Niehaus (2009) argue that mixing methods can only occur during either of these two times. Qualitative and quantitative data, they claim, should be kept separate and dealt with in an appropriate and rigorous manner according to their respective paradigms until the point of interface which can be at the analysis or the results discussion phase. That is to say, qualitative and quantitative data can be transformed prior to the analytic stage so that both data are used together in the analysis phase or that the two methods can be analyzed separately and then integrated in the discussion section where the two different analyses are compared to create a combined interpretation of the research phenomenon. In this study, integration of qualitative and quantitative data occurred in the question phase (individual questions addressed particular methods), data analysis (qualitative data were translated into quantitative data), and interpretation (results of the various analyses were compared to understand the larger research problem).
Data analysis in mixed methods research can take a variety of forms but involves general procedures. According to Onwuegbuzie and Teddlie (2003), there are seven stages of mixed methods data analysis: (1) data reduction, (2) data display, (3) data transformation, (4) data correlation, (5) data consolidation, (6) data comparison, and (7) data integration. Not all of the stages need to be present and they may occur in a different order. In stage 1, data are reduced through such processes as the computation of
102
stage 2, data are displayed visually in tables, graphs, matrices, or charts. Stage 3 involves transforming qualitative and/or quantitative data into forms that can be used with the other method. The next three stages involve correlating, consolidating, and/or comparing the quantitative and qualitative data, depending upon the research design. Finally, in stage 7, quantitative and qualitative data that have either already been combined or have been kept separate through the analysis process are brought together to interpret as a whole. The data analysis in a later section describes how data in this dissertation follows these stages in the analysis of data.
Embedded Mixed Methods
In both sequential and concurrent designs, one method is often more dominant than the other. Although some researchers see mixed methods as a way to incorporate inductive and deductive theoretical perspectives and assert that the ideal is equality among approaches (Johnson et al., 2007), others argue that it is important to know whether the study has an inductive or deductive theoretical drive (Caracelli & Greene, 1997; Morse & Niehaus, 2009). Morse (1991) argues that “a project must be either theoretically driven by the qualitative methods incorporating a complementary
quantitative component, or theoretically driven by the quantitative method, incorporating a complementary qualitative component” (p. 121). Caracelli and Greene (1997) describe designs where one methodology is located within a larger methodology as “embedded or nested designs” that “feature one methodology located within another, interlocking contrasting inquiry characteristics in a framework of creative tension” (p. 24). From this perspective, the predominant method guides the study in the construction of research
103
questions, data collection, and data analysis. In mixed methods nomenclature, the dominant method is capitalized while the supplemental method is written in lower case (i.e. QUAN → qual) (Morse, 1991). At this point in time, however, there are very few descriptions in the literature about how to conduct an embedded design (Creswell, 2009).
Johnson, Onwuegbuzie, and Turner (2007) define qualitative dominant mixed methods research as “the type of mixed research in which one relies on a qualitative, constructivist-poststructuralist-critical view of the research process, while concurrently recognizing that the addition of quantitative data and approaches are likely to benefit most research projects” (p. 124). The study here is an example of a concurrent embedded design with an inductive, qualitative theoretical drive (QUAL + quan). That is, the larger study from which this dissertation is derived is a qualitative case studies project where observations, interviews, and a collection of artifacts, assessment assignments, and pupil work samples were collected simultaneously at different time-points. As part of the larger qualitative case study, the Teacher Assessment/Pupil Learning (TAPL) protocol was developed to analyze pupil learning using both a qualitative and quantitative component. This dissertation is an analysis of the TAPL data. Both the quantitative and qualitative TAPL data were collected concurrently. The overall drive of the case studies project is inductive qualitative. The study here retains that inductive theoretical drive to see what is happening. The addition of the quantitative component is used to “enhance QUAL studies with measurement” in a way that can enable comparison, enhance description, illustrate, and triangulate (Morse & Niehaus, 2009, p. 99) and can also be used “to enrich the description of the sample participants” (Creswell et al., 2003, p. 230).
104
Figure 3.1 presents a graphic representation of this study’s research design to illustrate how this is a mixed methods study. The outside rectangle represents the larger longitudinal Qualitative Case Studies Project (QCS). The data sources for this larger study included classroom observations, interviews, artifacts (teacher candidates’ teacher preparation coursework), and samples of assessments and pupil work used in their classrooms. Part of the QCS research design included the TAPL protocol which drew from qualitative interviews (Internal TAPL) and a quantitative analysis of assessments and pupil work (External TAPL). This dissertation, which is an analysis of the TAPL protocol, is represented by the center box. A more detailed description follows in the research design section.
105
Quantitizing Qualitative Data
In a qualitatively-driven mixed methods research design, one way to integrate qualitative and quantitative data during the data reduction stage of data analysis is to quantify qualitative data, meaning to represent qualitative data in numerical form. Incorporating numerical analyses in qualitative data is not new and normally occurs, to some extent, in qualitative methodology and in qualitative methods such as ethnography, because, as Dobbert and Kurth-Schai (1992, p. 137) point out, “the ability of
mathematical techniques to clarify patterns has proven valuable” (p. 137). Even before the recent paradigmatic shift legitimizing mixed methods, Mitchell (1979) described how quantitative data could be used in qualitative work and how anthropologists have used quantitative “analytical procedures” as “aids to description” of their fieldwork:
[T]he more detailed knowledge which quantitative methods allow and the correlation between phenomena which statistical reasoning can educe should be the essential foundation on which anthropologists start to erect their
generalizations about the social behavior of the people they study. Quantitative methods are essentially aids to description. They help to bring out in detail the regularities in the data the fieldworker has collected. Means, ratios, and percentages are ways of summarizing the features and relationship in data. Statistical measures based on the theory of probability go beyond the mere quantitative data and use devices to bring out the association between the various social facts the observer has collected. These are essentially analytical procedures and, as Fortes puts it, “are nothing more than a refinement of the crude methods of comparison and induction commonly used.” (p. 20)
From this perspective, quantitative data was used as another way to describe qualitative