This study utilizes quantitative and qualitative data from the larger i3 evaluation for a secondary mixed methods analysis of the ANet program on teachers’ data use and instructional practices. Given the nature of the research questions, this study relies predominantly on year-two survey data. The year-two data are used because of the expectation that an intensive program, such as ANet, would take at least two years to be fully implemented. Due to high levels of school leader survey nonresponse (discussed in chapter three), the primary data source for the quantitative analyses in this study is the year-two teacher surveys (n = 616). The quantitative results for each of the research questions are supported by an analysis of qualitative site visit interview data. This mixed methods approach to data analysis provides a depth of understanding that could not be achieved by quantitative analysis alone (Sammons, 2010).
Scale Validation
This study was conducted in parallel with the larger evaluation on which I was the lead analyst (under the direction of the Principal Investigator). Because of my interest in
19
exploring the relationships between measures of school culture, instructional leadership, and teacher characteristics, and teachers’ data-based instructional practices, I had the opportunity to use data from baseline and year-one surveys to improve upon the
measurement of these focal constructs in the year‐two survey. Although the dissertation does not include a complete discussion of the survey revision work, chapter four reviews the characteristics of the revised year-two scales that measure instructional leadership, school culture, teachers’ attitudes towards and confidence with various data-based practices, and the frequency of teachers’ data use and instructional practices. Details are provided on the items within each scale, the overall scale reliabilities, and their
validation.
Quantitative Analysis
Given the nested design of the study, the analysis of survey data to estimate the effects of ANet uses multilevel regression modeling (MLM). Failure to model the nesting of teachers within schools can lead to violations of the assumptions of homoscedasticity and independence appropriate to the use of ordinary least squares (OLS) regression, increasing the likelihood of type I errors. Multilevel modeling addresses the issue of correlated errors by modeling the relationship at the various levels of the data (e.g., school and teacher) instead of constraining the model to a single level (as in OLS). The estimation procedures used in multilevel modeling generate standard errors that are not inflated due to nesting (Bickel, 2007).
Qualitative Analysis
After completing the quantitative analyses, all leader and teacher interview transcripts were fully coded and analyzed. A first round of coding identified portions of leader and teacher interviews that address the focal measures and research questions in this study (Leech & Onwuegbuzie, 2008; Saldaña, 2009; Hesse-Biber, 2010). The second round of coding entailed both finer-grained coding and analysis. Coding was informed by a conceptual framework developed from the ANet logic model and prior research.
Operationally, the second round of coding utilized a constant comparative approach to extract themes and provide explanations for the quantitative results (Leech &
Onwuegbuzie, 2008).
The mixing of analytic strategies is meant to take advantage of the strengths of both methods (Teddlie & Tashakkori, 2003; Johnson & Onwuegbuzie, 2004) and maximize the likelihood of collecting evidence of the relationship between school culture, teacher characteristics, and teachers’ instructional practices. The “mixing of methods” takes place at the interpretation stage. Given the causal nature of research questions, the results from the quantitative analyses take precedence. The qualitative results serve to explain the quantitative findings and provide explanatory context. In particular, they: 1) provide context for the impacts, or the lack thereof, on the teacher practices and key mediators in this study, 2) explore the validity of the conceptual framework and causal linkages (Yin, 2009), and 3) offer evidence of why ANet may be more effective at changing teachers practices in some contexts than others.
21 SIGNIFICANCE OF THE STUDY
A very considerable amount of time and resources are spent each year on data- based instructional strategies, including interim assessments (Lazarin, 2014; Hart, et al., 2015). In fact, evidence suggests that district-mandated tests – such as interim
assessments – make up a larger proportion of testing time than state tests, especially in urban districts (Lazarin, 2014). This is despite the fact that empirically sound research on the impacts of these practices is sparse and results are varied. Given the recent call by the Obama administration to reduce time spent on testing in American schools and a
provision to allow states to set limits on time spent on testing as part of the Every Student Succeeds Act (ESSA), evidence of the quality of interim assessments and their utility in improving teaching and learning may become more important than ever (U.S. Dept. of Education, Fact Sheet, 2015; ESSA, 2015).
This study addresses two main problems in the current research on interim assessment and data-driven instruction. First, it fills an empirical need for research on interim assessment programs and data-driven instructional practices that combines empirically sound research designs with rich process and outcome data. This design allows for the study to explore the data-based instructional process and address major gaps in our current understanding of whether and how data-based initiatives have an impact on teachers’ practices. In particular, the study explores the oft-cited, but not well- understood roles played by certain school conditions and teacher characteristics. The combination of quantitative and qualitative evidence, collected as part of a randomized
evaluation, provides a unique opportunity to address the empirical and substantive gaps in prior research on teachers’ data-based instructional practices.
To make findings more useable, some have suggested that researchers align their work with the current challenges that administrators are facing (Honig & Coburn, 2007). In terms of its practical importance, the hope is that the results of this study will provide district- and school-level practitioners and policymakers looking to implement data-based instructional strategies with useable insights on where and how to focus their energies in order to foster change without unintended, negative consequences for teachers and students. In the longer term, the results have the potential to inform the development of interim assessment programs; specifically, implementation and training targeted to the conditions in schools and characteristics of educators that support the adoption of effective data-based instructional practices.
23