While the focus of this dissertation is on classroom-level use of interim
data-driven decision making (DDDM) in public education. The phrase is used to describe decision-making processes at all levels of the education system that are informed by various sources of data. DDDM involves the systematic and ongoing collection, analysis, interpretation, and use of educational data for various ends such as improving instruction, better allocating resources (i.e., material and human capital), and informing policies (Mandinach, 2012). Since DDDM can be used at every level and in every role, it
incorporates a variety of educational data from student assessments and demographics, to administrative, financial, personnel, and multiple other data sources (Mandinach, 2012).
Despite its growing prevalence, data use in education is not a new practice and has its roots in the growth of measurement and accountability for student achievement (Dembosky, et al., 2005; Marsh, Pane, & Hamilton, 2006; Christman, et al., 2009; Bulkley, Oláh, & Blanc, 2010; Faria, et al., 2012). However, recent trends in
accountability policies have provided the impetus for a more formal process of data use, including a more systematic use of external, standardized assessments as a key source of data on student learning. Data use and accountability have become “inextricably” linked (Mandinach & Honey, 2008, p. 2).
There has been criticism over the phrase “data-driven” and some of the practices falling under this umbrella, criticism that highlights the range of these practice.
Specifically, critics of the term contend that to be data-driven both oversimplifies the process and implies one in which data drive the focus of education reform at the macro level and the focus of instruction at the micro level (Shirley & Hargreaves, 2006). Instead, some experts in the field propose that the process should be “evidence
27
informed”: the collection of evidence that informs educational decisions (Shirley & Hargreaves, 2006; Hargreaves & Braun, 2013). Furthermore, others contend that
although student data are a useful tool, the process should be combined with and guided by values and professional judgment (Hertiage & Yeagley 2005; Shirley & Hargreaves, 2006; Knapp, Copland, & Swinnerton, 2007; Wayman, Snodgrass Rangel, Jimerson, & Cho, 2010; Wayman, Jimerson, & Cho, 2012; Hargreaves & Braun, 2013; Hargreaves, Morton, Braun, & Gurn, 2014). The fact that this argument is part of the conversation on interim assessment programs illustrates the range of philosophies on which these
programs are based: from programmed and prescribed, to adaptable and open to professional judgment.
It is not the purpose of the dissertation or literature review to evaluate where ANet or other data-based instructional programs fall on this range of data-driven or evidence- informed. Throughout this chapter, the terms data-driven or data-based are used to encompass the range of practices and programs related to instructional data use; from the provision of periodic interim or benchmark assessments, to more comprehensive systems that include tools (e.g., protocols and data systems), professional development and support, and new technology (e.g., data dashboards). Whenever possible, characteristics of the programs examined in prior research are described.
Origins of & Influences on Data Use in Education
Utilizing data has become a key practice in almost any industry that values productivity and continuous improvement: public sectors like health care and
government, or private sector business and finance. In setting the context for instructional data use in education, one could argue its evolution has not only been influenced by the advent of high-stakes educational accountability systems, but also by the increasing role of the business sector in educational management (Marsh, Pane, & Hamilton, 2006; Young & Kim, 2010). The private sector has long promoted management systems that monitor productivity, improve performance, and evaluate systems at all levels (Stecher, Kirby, Barney, Pearson, & Chow, 2004). Data have become an important component of these systems; see, for example, the booming industry around “big data.” Successful businesses are said to empower their employees and one way this can be achieved is by providing real-time, relevant data that allow them to take ownership over decision making (Stecher, et al., 2008; Hargreaves, Morton, Braun, & Gurn, 2014). Given evidence of the success of these practices in other industries (Manyika, et al., 2011), policymakers and reformers have advocated for education to adopt similar processes (Tyack, 1995; Stecher, et al., 2008).
Though the influence of the business sector has had an impact, the proliferation of data use in education has had as much to do with test-based accountability policies that are meant to increase student achievement relative to specific content standards. These accountability systems rely heavily on student assessments which have provided a constant stream of achievement data. From the 1970s through the 1990s assessments were used to monitor whether Title I funds were improving the educational outcomes of disadvantaged students, maintain minimum competency for graduation or grade
29
promotion, and ensure schools were achieving high levels of performance to improve global competitiveness (Linn, 2000).
The modern era of test-based accountability was ushered in when the penultimate reauthorization of the Elementary and Secondary Education Act, the No Child Left Behind Act (2002) (NCLB), was passed in 2001. NCLB maintained a focus on standards and test-based accountability by setting annual achievement goals aiming, ultimately, at proficiency for all students. Whether schools met these goals was determined by annual testing in English language arts (ELA), math, and science. Schools and districts that failed to meet annual targets (i.e., Annual Yearly Progress) were initially subject to sanctions ranging from providing supplemental services to students, to school
restructuring. NCLB was scheduled for reauthorization in 2007. While the U.S. House and Senate debated proposals for reauthorization, states were granted waivers by the federal Department of Education from some portions of the bill’s original requirements in an effort to avoid further sanctions (U.S. Dept. of Education, ESEA flexibility website, 2014).2
In its signature educational reform effort, the Obama administration earmarked grant funding through Race to the Top (RttT) to encourage education reform through improvements in four key interrelated areas: standards and assessment; data systems, collection, and use; teacher effectiveness; and turning around low-performing schools (U.S. Dept. of Education, RttT Executive Summary, 2009). RttT also included a $350
2 President Obama signed the newest iteration of the bill, called the Every Student Succeeds Act, on
December 10, 2015. The new law upholds the testing requirements of NCLB, but allows states more flexibility to set annual accountability targets which are reviewed by the U.S. Department of Education.
million assessment program competition that funded two multi-state consortia to design assessment systems that include a combination of features such as diagnostic, interim, and summative assessments that are, in some cases, administered in a computer-adaptive format (SMARTER Balanced Consortium website, 2014; PARCC Consortium website, 2014). According to the Department of Education, the intention was to
develop assessments that are valid, support and inform instruction, provide accurate information about what students know and can do, and measure student achievement against standards designed to ensure that all students gain the knowledge and skills needed to succeed in college and the workplace. These assessments are intended to play a critical role in educational systems; provide administrators, educators, parents, and students with the data and information needed to continuously improve teaching and learning…. (U.S. Dept. of Education, RttT Assessment Program website, 2014)
The effectiveness of current accountability systems rests on a theory of action that posits that student achievement will be positively impacted by a system that holds
teachers and school leaders accountable to raising student achievement, as measured by student assessments, and through a series of sanctions and incentives (Hamilton, Stecher, & Klein, 2002). The problem is that an accountability system based on summative assessments that measure achievement against a proficiency benchmark cannot provide school leaders and teachers with timely data at the level of detail necessary to draw inferences about student learning, make “actionable” decisions, and adjust instruction as necessary (Mandinach & Jackson, 2012, p. 16). In fact, some argue that such a system actually has limited educational value (Bennett & Gitomer, 2008) and that improvements in teaching and learning will only be realized by “aligning curriculum, instruction, and
31
for educators to analyze data with colleagues in the light of curricular objectives.” (Bulkley, Oláh, & Blanc, 2010, p. 115).
In its 2001 report “Knowing What Students Know,” the National Research Council (NRC) made several recommendations on student assessment. To be
instructionally useful to classroom teachers, they contend that assessment systems should include classroom assessments that are integrated with instruction and make students’ cognitive processes evident; e.g., teachers should be able to infer from students’ assessment results both the strategies students used, as well as their thought processes. Tasks should be developed with consideration to the skills students need to answer an item correctly, the context in which it is presented, and whether it requires transfer of knowledge from other contexts. To increase the likelihood of student learning, results should be timely and teachers should be adequately trained in both theory and practice to use this information (NRC, 2001).
The two consortia’s assessment systems were designed to address the limitations of the current accountability system by shifting from assessment of learning to a system that attempt to include assessment for learning (Pellegrino, 2006; Bennett & Gitomer, 2008; Mandinach & Jackson, 2012) which allows teachers to use the results in some of the ways recommended by the NRC. In a system of assessment for learning, assessments do not merely check on learning summatively, they provide on-going evidence of what students have and have not mastered (see also, Stiggins, 2005). While districts and schools awaited the roll-out of these new assessment systems, many turned to interim assessment programs to improve teaching and learning (Herman & Baker, 2005).