Addition by Subtraction: The Relation Between Dropout Rates and School-Level Academic Achievement

(1)

Between Dropout Rates and School-Level Academic Achievement

ELIZABETH GLENNIE RTI International

KARA BONNEAU

North Carolina Education Research Data Center

MICHELLE VANDELLEN KENNETH A. DODGE Duke University

Background/Context: Efforts to improve student achievement should increase graduation rates. However, work investigating the effects of student-level accountability has consistently demonstrated that increases in the standards for high school graduation are correlated with increases in dropout rates. The most favored explanation for this finding is that high-stakes testing policies that mandate grade repetition and high school exit exams may be the tipping point for students who are already struggling academically. These extra demands may, in fact, push students out of school.

Purpose/Objective/Focus: This article examines two hypotheses regarding the relation between school-level accountability and dropout rates. The first posits that improvements in school performance lead to improved success for ever yone. If school-level accountability sys- tems improve a school for all students, then the proportion of students performing at grade level increases, and the dropout rate decreases. The second hypothesis posits that schools fac- ing pressure to improve their overall accountability score may pursue this increase at the cost of other student outcomes, including dropout rate.

Research Design: Our approach focuses on the dynamic relation between school-level acad- emic achievement and dropout rates over time—that is, between one year’s achievement and

(2)

the subsequent year’s dropout rate, and vice versa. This article employs longitudinal data of records on all students in North Carolina public schools over an 8-year period. Analyses employ fixed-effects models clustering schools and districts within years and controls each year for school size, percentage of students who were free/reduced-price lunch eligible, percent- age of students who are ethnic minorities, and locale.

Findings/Results: This study finds partial evidence that improvements in school-level aca- demic performance will lead to improvements (i.e., decreases) in school-level dropout rates.

Schools with improved performance saw decreased dropout rates following these successes.

However, we find more evidence of a negative side of the quest for improved academic per- formance. When dropout rates increase, the performance composites in subsequent years increase.

Conclusions/recommendations: Accountability systems need to remove any indirect benefit a school may receive from increasing its dropout rate. Schools should be held accountable for those who drop out of school. Given the personal and social costs of dropping out, account- ability systems need to place more emphasis on dropout prevention. Such an emphasis could encompass increasing the dropout age and having the school’s performance composite include scores of zero on end-of-grade tests for those who leave school.

In the 1990s and 2000s, state and federal governments increased their emphasis on measured educational performance for schools, students, and teachers. At the school level, this emphasis took on various forms, such as publishing schoolwide end-of-year test scores in local newspapers, hiring and firing principals for poor student performance, and providing special assistance for failing schools. Schools implemented various strategies to improve schoolwide test scores, and anecdotal reports suggested that schools were pushing out at-risk and low-achieving students as a per- nicious strategy to improve future schoolwide average scores. In contrast, advocates of this emphasis on tested achievement suggested that improvements in schoolwide test scores would create a culture of success that would lead to decreases in school dropout rates. The current manuscript addresses the dynamic relation between schoolwide achievement test scores and schoolwide dropout rates. It does not evaluate accountability programs per se, but it addresses schoolwide strategies that were implemented during an era of accountability. The results of this study have implications for the development and implementation of accountability systems.

Proponents of accountability systems claim that accountability helps all students by providing incentives for teachers and students to perform well. Hanushek and Raymond (2002) argued that “accountability systems should be viewed as an inherent source of incentives designed to push schools toward desired outcomes. The ultimate impact of accountability efforts depends on the precision and force of the incentives they create”

(3)

(p. 80). According to this perspective, in accountability systems, all teachers and students should have a clear definition of academic success and an incentive to perform well. Thus, students will be more likely to perform at grade level. Clear definitions of success and incentives to perform may increase teacher efforts and improve the school’s academic environment. These improvements may lead to a subsequent decrease in dropout rates.

Research on the relation between school achievement and dropout rates has been conducted in the context of accountability programs. Most research on accountability programs investigates the effects of student- level accountability on both positive outcomes (e.g., performance) and negative outcomes (e.g., dropout rates). This research consistently demonstrates that increases in the standards for high school graduation are correlated with increases in dropout rates. The most favored explanation for this finding is that high-stakes testing policies that mandate grade repetition and high school exit exams may be the tipping point for students who are already struggling academically. These extra demands may, in fact, push students out of school. Studies examining between-state dif- ferences have found that states with student-level accountability for graduation (minimum competency exams) have higher dropout rates (Carnoy & Loeb, 2002). Other studies have found that both introducing student-level accountability and increasing accountability standards are associated with increases in dropout rates (Clarke, Haney, & Madaus, 2000; Lillard & DeCicca, 2001). Such accountability practices are particularly likely to influence dropout rates for at-risk schools and low-performing students (Jacob, 2001; McDill, Natriello, & Pallas, 1986).

Because school-level accountability practices are relatively recent, past research has not been able to examine the relation between school-level achievement, as measured through test scores, and school-level dropout rates. Furthermore, past research has not included analyses that reveal the dynamic relation between school-level academic achievement and dropout rates over time—that is, between one year’s achievement and the subsequent year’s dropout rate, and vice versa. Further, few studies include rigorous controls such as fixed-effects analyses at the school and school district level. School-level fixed effects analyses control for time- invariant characteristics of individual schools that might be related to performance. District-level fixed-effects analyses control for factors at the district level, such as a strategy to reassign high- or low-achieving students to different schools in order to concentrate, disperse, or other wise affect performance. Although this manuscript cannot evaluate the impact of accountability programs directly, their presence allows for an investigation of the variables associated with increased performance. Because past

(4)

research on student-level accountability practices shows a relationship between achievement and dropout rates, we focus our investigation on these variables.

In this article, we use longitudinal data of records of all students in North Carolina public schools to examine two hypotheses regarding the relation between school-level accountability and dropout rates. The first, the “rising tide” hypothesis, is that improvements in school performance lead to success for ever yone. If school-level accountability systems improve a school for all students, the proportion of students performing at grade level should increase, and the dropout rate should decrease both concurrently and in subsequent years.

An additional possibility, the “student overboard” hypothesis, is that schools face pressure to improve their overall accountability score and may pursue an increase at the cost of other student outcomes, including dropout rate. The “student overboard” hypothesis is consistent with past research on student-level accountability (referenced earlier).

Furthermore, schools are known to respond to incentives (Hanushek &

Raymond, 2002). If incentives focus exclusively on growth in end-of- grade or end-of-course scores without imposing negative consequences for increases in dropout rates, schools may be tempted to focus attention on increasing test scores at the cost of their dropout rates. In response to the increasing pressure to attain high scores, academically struggling students may feel that they cannot succeed, and teachers may feel pressure to increase test scores; thus, schools might directly or indirectly encourage students to leave. If struggling students drop out and are not counted as failures in the school’s accountability scores, schools may succeed in reaching their accountability goals. In other words, schools can add to their future performance scores by subtracting currently low-performing students.

DROPPING OUT

Dropping out of high school has long-term social and economic consequences. One must have a high school diploma to enroll in postsecondar y schools and even to obtain many minimum-wage jobs (Catterall, 1985; Rumberger, 1987; U.S. Department of Education, 2001). High school completion is even more crucial today as we face the demands of rapidly changing and expanding global markets. From 1970 to 2000, the proportion of jobs in the manufacturing sector declined by 50%, and many low-skilled jobs in the expanding ser vice sector are extremely low paying (Letgers, Balfanz, Jordan, & McPartland, 2002). With few opportunities for advancement, high school dropouts are more likely to live in

(5)

poverty as adults. They exhibit higher levels of alcohol consumption, poorer mental and physical health, and increased likelihood of commit- ting criminal acts and of becoming dependent on welfare and government programs than people with higher educational attainment (Caspi, Wright, Moffitt, & Silva, 1998; Rumberger, 1987; Thornberr y, Moore, &

Christenson, 1985).

Students leave school for any number of reasons, including academic failure, disciplinar y problems, wanting or needing to start work, and having children. Because the process of withdrawing from school is complex, school administrators may not know the specific cause of a particular student’s departure. Research has identified several risk factors for dropping out. Students from low-socioeconomic-status backgrounds are more likely to drop out of school than more affluent students are, and poor academic performance and grade retention are strong predictors of leaving school (e.g., Alexander, Entwisle, & Dauber, 2003; Jimerson, 1999;

Rumberger, 1995). In these cases, students may have had an ongoing academic battle that began before high school. In particular, retained students who are older than their grade peers can legally leave school in earlier grades. In North Carolina, adolescents can legally leave school at age 16. Stearns and Glennie (2006) showed that ninth graders and 16- year-olds are more likely than more advanced, older students to leave school, indicating that students who have had problems with school may leave as quickly as they can.

SCHOOL-LEVEL FACTORS, ACADEMIC SUCCESS, AND DROPOUT Much research has focused on the way that school characteristics influence outcomes for students (e.g., Br yk & Thum, 1989; Goldschmidt &

Wang, 1999; McNeal, 1997a, 1997b; Rumberger, 1995; Rumberger &

Thomas, 2000). In particular, push-out theories for dropping out suggest that certain school practices and policies discourage students from com- pleting high school. These theories focus on aspects of school that make it an unwelcoming place and that diminish teenagers’ connection to school, such as certain disciplinar y policies and accountability systems (Jordan & McPartland, 1996). Schools may have explicit policies con- cerning low grades, attendance, or behavior with consequences of long- term suspensions and referrals to alternative programs under which students withdraw involuntarily from school. Additionally, schools may implement policies and practices that affect the conditions that keep students engaged in school and thus contribute to students’ voluntar y with- drawal from school (Rumberger, 2004). According to push-out theorists, students’ individual attributes do not completely account for the decision

(6)

to leave school. Instead, a combination of individual and school attributes influences them to leave (Fine, 1986, 1991). In this study, we examine whether state and federal accountability policies give schools an incentive to “push” students out of school.

School contextual factors, such as size, poverty, ethnic composition, and locale, have been associated with academic success. In medium-sized schools, students learn more than in smaller and especially in much larger schools, but there are fewer learning gaps in the smaller high schools than in larger schools (Lee & Smith, 1997). In small schools, teachers take more responsibility for their students than teachers in larger schools do (Lee & Loeb, 2000). Thus, in large schools, teachers and counselors may not know the students as well as in small schools, and students may receive less personal attention. The student composition in a school is also associated with its academic profile. Goldschmidt and Wang (1999) found a positive relation between early school dropout rate and the percentage of Latino enrollment. Rumberger (1995) reported a positive relation between the dropout rate and percentage minority in schools where the student body is more than 40% minority. Additionally, he found that students in schools with low socioeconomic status have higher odds of dropping out even when controlling for their individual characteristics, including socioeconomic status. In the analyses that fol- low, we control for aspects of the school structure (size and locale) and student composition (poverty level and ethnic composition).

ACCOUNTABILITY IN NORTH CAROLINA

North Carolina’s ABCs of Accountability program is designed to empha- size the basics and promote high educational standards. Implemented in 1996, this program requires that students in Grades 3–8 take end-of- grade (EOG) reading and math tests each year. In high school, students take end-of-course (EOC) subject tests in up to 10 different subjects, including algebra 1, U.S. histor y, English, and biology. Schools must test at least 95% of enrolled students in those subjects (North Carolina Department of Public Instruction, 1998).

North Carolina’s accountability system was one of the models for the federal No Child Left Behind Act, which was implemented in 2002. This law requires that states assess all public school students in reading and math in Grades 3–8 and once in high school. North Carolina had already established this testing regimen and did not have to introduce new exam- inations in response to this federal law. To date, No Child Left Behind assessments focus on reading and math and do not require assessments in science or social sciences. Although EOC science and social science

(7)

exams count heavily in North Carolina’s ABCs of Accountability program, they are not yet part of No Child Left Behind. This article focuses on school-level performance under North Carolina’s conditions of accountability. It cannot evaluate the impact of an accountability system because data are not available for performance under the counterfactual

“no accountability” system. However, the presence of the accountability system is the reason for the systematic data collection on school-level performance and dropout rates. Data collected under the accountability system provide an informative platform to investigate the relation between school-level test scores and dropout rates.

North Carolina’s system focuses on academic growth as well as levels of proficiency, so each school receives two scores, one performance composite score and one growth score. The performance composite is the percentage of students performing at grade level on the EOC exams, weighted by the numbers of students taking each subject. Growth scores are based primarily on statewide average growth and the previous performance of students in each school. High schools are expected to maintain dropout rates of less than 5%, and the growth score does include the school’s dropout rate. However, the dropout rate does not have much weight in calculating the school’s overall score. All contributors to the growth score are weighted by the number of students affected out of the total number of students in the school. Because the number of dropouts is usually small, its impact on the overall composite is also ver y small. In the calculation, the weight of the dropout component is the largest number of dropouts that occurred in Year 1, 2, or 3 divided by the total number of students across all ABC components (i.e., the number taking algebra 1, the number taking English 1, the number taking biology, and so forth). For example, consider a high school where the highest number of dropouts over 3 years is 29, and the number of students taking all EOC tests is 4,529. The total N is 29 + 4,529 = 4,558. Thus, the weight of dropouts is 29/4,558, or 0.006. If 475 students took algebra 1, then the weight of algebra 1 would be 475/4,558 = 0.104. With 10 EOC exams, the dropout component contributes little to the overall score. Additionally, the growth composite does not include the long-term suspended, expelled, or incarcerated students as part of the dropout rate calculation (North Carolina Department of Public Instruction, 2005). Such students would be excluded from the school’s EOC exams.

Originally, North Carolina’s accountability was designed as a school- level accountability system, but in the early 2000s, the state incorporated a student accountability component. Students who fail EOG reading and math in Grades 3, 5, or 8 are held back to repeat the grade. Although North Carolina does not have an exit exam requirement for high school

(8)

graduation, as of the 2006–2007 year, North Carolina has required students to pass EOC exams in algebra 1, English, biology, U.S. histor y, and civics in order to graduate.¹

How might increased motivation for achievement influence students to drop out? First, under structured systems measuring achievement, teachers are encouraged to give direct feedback to students regarding failure.

Struggling students may tire of receiving negative feedback and come to believe that school is not the right place for them. Catterall (1989) showed that high school students who fail part of a competency exam have increased doubts about graduating and reduced aspirations.

Second, when school success is based on the proportion of students who receive test scores above a threshold, teachers may focus on those whose prior performance is just below grade level and neglect those whose performance is far below grade level. This strategy may help a school reach its proficiency target. Those students just below grade level may indeed pass the exam and increase the school’s overall performance score. Even if those far below grade level do improve, they still may not reach the passing score and would not help the school’s composite. Third, a more direct path from accountability to increased dropout rates might occur if teachers encourage struggling students to transfer, turn to homeschool- ing, or drop out so that school’s score will not include that child’s poor test results.

Furthermore, research suggests that schools respond strategically to school-based accountability systems by excluding some students from taking the high-stakes exams. School-level test scores have increased in schools with the highest rates of grade retention and dropout (Darling- Hammond, 2006). A study of Chicago public schools showed that high- stakes testing led to increases in grade retention in the grades prior to the gateway exam years, particularly among low-achieving students and in low-performing schools. In other words, lower achieving students would be exempt from taking the high-stakes exams for another year (Jacob, 2005). In Florida, after the implementation of a school-based accountability system, low-performing and low-socioeconomic-status students were more likely to be classified into disability categories that are exempt from testing (Figlio & Getzler, 2002). In Texas, schools responded to increases in incentives by classifying more students as special needs and encouraging absences (Cullen & Reback, 2006), and the exit of low- achieving students created an appearance of rising test scores and a nar- rowing achievement gap (McNeil, Coppola, Radigan, & Heilig, 2008).

In this analysis, we examine the relation between the performance composite and the dropout rate. First, we examine whether an increase in the performance composite subsequently leads to a change in the

(9)

dropout rate. If the subsequent dropout rate declines, we would con- clude that the accountability system is operating consistently; that is, as students perform better, more of them stay in school. Next, we consider whether a change in the dropout rate influences the performance composite in subsequent years. If an increase in the dropout rate leads to an increase in the performance composite, then the flaw in the accountability standards allows schools to improve their performance when some students leave.

DATA AND METHOD

Data come from the North Carolina Education Research Data Center at Duke University, which houses information on ever y public school and student in the state. The North Carolina Department of Public Instruction originally collected information on student dropouts and school and performance composites. Our analyses cover the period from 1997–1998, the first year of the high school accountability system, to 2004–2005, the last year for which data are available. The initial sample consists of 258 traditional high schools with grade ranges of 9–12, which were operating in each of these 8 years.

These models include controls for school size, percent of students who were eligible for free/reduced-price lunch, percent of students who are ethnic minorities, and locale. Locale comes from the National Center for Education Statistics Common Core of Data and is coded here as suburban and urban, with rural as the reference categor y. Control measures come from the first year of the performance measurement. For example, in the model where the change in dropout rate from 1998–1999 predicts the change in performance composite from 1998–2000, 1998 school contextual measurements are used. Note that control variables are measured each year. Although the change in performance composite from 1998–1999 uses 1998 school contextual measures, the change from 1999–2000 uses the 1999 school contextual measures.

A dropout is someone who began school in the previous school year and either dropped out during that school year or did not return to school following the summer break. The state does not consider transfers to be dropouts; a dropout has left the school system entirely. Someone who changed schools because of a residential move or school choice option like charter schools would not be counted as a dropout.² Each October, schools report dropouts from the previous year. For example, a student who was enrolled in the ninth grade during the 1998–1999 school year and finished the ninth grade but did not return to school during the fall of 1999 would be counted as a ninth-grade dropout, as

(10)

would a student who left the ninth grade at some point during the 1998–1999 school year. North Carolina follows the federal event count method of reporting dropouts (North Carolina Department of Public Instruction, 2000).

Performance composite is the percentage of students in the high school passing EOC tests, weighted by the number of students who took the tests. The North Carolina Department of Public Instruction calcu- lates this measure each year.

During this period in North Carolina, the school-level academic performance was fairly stable. Year-to-year performance composites are correlated at about 0.9, and year-to-year dropout rates are correlated at about 0.6 (the appendix reports each year-to-year correlation). Thus, high-performing schools continue to do well over time, and low-performing schools continue to do poorly. To control for these patterns, we use a change analysis approach, examining the relation between changes in dropout rates and changes in performance composites. The first set of models examines the influence of the change in the performance composite on the change in dropout rates. This analysis addresses the hypothesis that the accountability system benefits all students. The second set of models examines the influence of the change in dropout rates on the change in the performance composite. This analysis addresses the hypothesis that schools can add to their performance composite by subtracting problematic students.

Note that the state measures dropout rates in October and performance composites in May. Dropout data are collected throughout the year because dropouts may leave school at any time during a 12-month period. The performance composite is a snapshot, calculated once a year from that spring’s EOG and EOC exams. Because these dropout rates and performance composites are reported at different times of year, and the specific dropout date is not available, we calculate these change scores to avoid overlap in these measures. Figures 1a and 1b show the time periods encompassed by these change scores. In these examples,

“year” refers to calendar year (Januar y to December) rather than school year (September to May). October of Year 1 follows May of Year 1.

Figure 1a illustrates the timing of measures for models showing the influence of changes in performance composites on changes in dropout rates. The performance composite change is from May of Year 1 to May of Year 2, and the dropout change is in the subsequent time period, from October of Year 2 to October of Year 3. With this measurement approach, none of these students could have dropped out during the time when performance change is being measured; dropouts could not leave before the second set of exams and performance calculation in Year 2.

(11)

Figure 1b illustrates the timing of measures for models showing the influence of changes in dropout rates on change in performance composite. Here the dropout change was measured from October of Year 1 to October of Year 2. Because students could drop out at any point during the school year or over the summer, some of them did take EOC exams in May of Year 2. Because data do not permit distinguishing those students who took these exams and subsequently dropped out from those who dropped out prior to taking the exams, we must use a two-year change in performance composite to prevent overlap in categories.

Performance composite change is measured in May of Year 1 and May of Year 3. None of the dropouts could have taken exams in May of Year 3.

Year Month Performance composite Dropout rate

Year 1

May June July Aug Sept Oct Nov Dec

Year 2

Jan Feb Mar Apr May Jun Jul Aug Sept Oct Nov Dec

Year 3

Jan Feb Mar Apr May Jun Jul Aug Sep

Figure 1a. Measurement of the change in the performance composite from Year 1 to Year 2 predicting change in dropout rate from Year 2 to Year 3

Note: Bars refer to the period that schools collect these data. Performance composites are based on exams offered once a year, but dropouts could leave school at any time of year.

(12)

Regardless of the schools’ structural factors and student composition, state and federal education policies, such as ABC of Accountability and No Child Left Behind, influence all these schools simultaneously.

Administrators and teachers all learn to work with and adjust to these new initiatives during this time frame. Even within North Carolina’s accountability system, the required tests, the test questions, and the scaling metrics change over time. Those changes influence the percentage of students performing at grade level independent of students’ learning. Because these changes in policy, the economy, and other global factors affect all schools in ways that influence dropout rates and the percentage of students performing at grade level, we estimate two different

Year Month Performance composite Dropout rate

Year 1

May June July Aug Sept Oct Nov Dec

Year 2

Jan Feb Mar Apr May Jun Jul Aug Sept Oct Nov Dec

Year 3

Jan Feb Mar Apr May Jun Jul Aug Sep

Figure 1b. Measurement of the change in dropout rate from Year 1 to Year 2 predicting change in perfor- mance composite from Year 1 to Year 3

Note: Bars refer to period that schools collect these data. Performance composites are based on exams offered once a year, but dropouts could leave school at any time of year.

(13)

sets of fixed-effects models. The first set of models allows for the clustering of schools within years. Each year has its own set of macro-level con- straints affecting all schools. This model is comparable to a traditional ordinar y least squares (OLS) model with dummy variables for each year, thereby controlling for unobser ved, shared characteristics between schools across time.

The school fixed-effects model estimated P_it= X_it + ci+ u_itis, where i indexes years and t indexes schools. Note that this is a somewhat differ- ent specification than is normally seen in such models using panel data;

we believe that the most important clustering occurs among schools within years rather than within schools over time. P is the change in a school’s performance composite from the previous year, and the X vector includes school size, free-lunch-eligible students, non-White students, and urbanicity. The parameter c_iis the effect of unobser ved time-specific characteristics that var y across years but are constant across schools. The u_it is the effect of unobser ved characteristics, which can change across both i and t (Wooldridge, 2002).

In fixed-effects models, c_iis a year-specific constant term, meaning that year-level unobser ved characteristics do not var y across schools (Greene, 2003). This method allows for correlation between obser ved characteristics included in the model and the unobser ved characteristics.

Therefore, a fixed-effects model was employed to deal with heterogene- ity biases, which occur in OLS regressions when unobser ved characteristics, such as policy changes, are correlated both with performance and with dropout rates.

Additionally, we test whether this effect operates in the same way across school academic profiles by classifying schools as having low or high initial dropout rates and low or high initial performance composites, and running models separately for each type.

The second set of models uses district-level fixed effects estimators. For districts that have multiple high schools, it is plausible that district leaders might experiment with student assignment policies to tr y to improve performance, thereby nonrandomly and intentionally altering from one year to the next the student population of high schools. School-level fixed effects models fail to account for the impact of district-level variables. We include both sets of models as the most comprehensive approach to evaluating the relation between achievement and dropout at the school level.

(14)

RESULTS

Table 1 reports descriptive statistics for the measures in the model.

Traditional high schools have an average of 1,069 students. About half of the schools are in urban areas, a third are in suburban areas, and the remaining percentage are in rural locations. Approximately two thirds of students perform at grade level on EOG and EOC exams. In North Carolina high schools, about 36% of the students are ethnic minorities, and 23% of students are eligible for free or reduced-price lunch. The average year-to-year change in performance composite is an increase of 2.59%, and the average two-year change is 5.59. Approximately 6% of students drop out each year, and the year-to-year change in dropout rate is - 0.003. Thus, on average over time, North Carolina schools have had a slight increase in performance composite and decrease in dropout rates.

Table 2 presents the correlations of the school context measures with the performance composite and dropout rates, The dropout rate is negatively correlated with the performance composite; schools with low dropout rates tend to have high percentages of students performing at grade level in EOC exams. Other school factors are strongly associated with the school’s academic profile. Student body composition is associated with both performance composites and dropout rates. Percentage ethnic minority and percentage free or reduced-price eligible are negatively associated with the performance composite and positively associated with the dropout rates; schools with high levels of ethnic minorities

Table 1. Means and Standard Deviations of Analytic Variables (N = 1,514 schools)

Variable Mean SD

School Composition, Size, and Locale

Percent non-White .3450 .2419

Percent free/reduced-priced lunch eligible .2005 .1389

Size (school size) 1,070 454.3

Rural

Suburban .3197 .4665

Urban .5139 .5000

Student Performance

Performance composite 68.83 12.25

1-year change in performance 2.586 3.879

2-year change in performance 5.593 5.198

Dropout Rates

Dropout rate .0569 .0210

Change in dropout rate -.0027 .0162

(15)

and poor students have lower academic profiles. School size is positively associated with performance composite and negatively associated with the dropout rate. Urban schools have a positive correlation with the dropout rate, and rural schools have a negative correlation with both the dropout rate and the performance composite. Now we turn to the ques- tion of how these factors, combined with changes in the dropout rate, influence changes in the performance composite.

Table 3 shows fixed-effect models predicting the year-to-year change in dropout rates from the year-to-year change in performance composite, net of other school factors. Model 1 reports results for the school context measures, and Model 2 adds the prior change in performance composite. School context measures, such as percent minority, size, and locale, are associated with the contemporaneous dropout rate; however, none of these is associated with the change in dropout rate in Model 1. In Model 2, the change in performance composite is negatively associated with the subsequent change in dropout rates. That is, if a school’s performance composite increases in one year, the dropout rate decreases in the next year. For example, a one-year performance increase of 5 points would decrease the dropout rates by 1%. This finding supports the rising tide hypothesis: When schools improve their academic profile, students benefit, and they stay in school. However, when the prior change in performance composite is added to the model, the R²decreases from 0.47 to 0.00, and this model has less explanator y power of the process by which dropout rates change than the model with only control measures. Thus, although the change in performance composite is associated with the change in dropout rates, it is not a good predictor of changes in dropout rates.

To test whether this effect operates similarly across schools with low or high initial dropout rates and low or high initial academic performance,

Table 2. Within-Time Correlations of School-level Variables With Performance Composite and Dropout Rates (N = 258 Schools and 1,546 School Years)

Performance Composite Dropout Rate

Dropout rate -0.41 ***

% Minority -0.68 *** 0.24 ***

% Free/reduced-price lunch eligible -0.55 *** 0.26 ***

Size 0.13 *** -0.18 ***

Suburban 0.05 * -0.01

Urban 0.02 0.09 ***

Rural -0.08 ** -0.10 ***

*p < .05. **p < .01. ***p < .001.

(16)

we estimated Model 2 for four subsets of schools—those with above-aver- age performance (+0.5 – +1.5 SD) at baseline, those with below-average performance (-0.5 to -1.5 standard deviations) at baseline, those with above-average dropout rates at baseline (+0.5 to +1.5 SD), and those with below-average dropout rates at baseline (-0.5 to -1.5 SD). As shown in Table 3, the significant effect of changing performance on dropout rates holds only in schools with high dropout rates at baseline. This finding is not surprising, given that schools with low dropout rates have less room for improvement, and shows that schools with high dropout rates can in fact significantly decrease the dropout rate by improving students’ performance on tests.

Table 4 shows results from the alternative hypothesis that schools improve their performance when students drop out. This school fixed- effect model examines the influence of the change in dropout rate on the change in the performance composite from Year 1 to Year 3. Model 1 includes the control measures, and Model 2 adds the prior change in dropout rate. The student body composition (percent ethnic minority and percentage eligible for free or reduced-priced lunch), school size, and locale were all associated with the dropout rate (Table 2). In Model 1, student body composition is not associated with the change in the performance composite. Although the school’s percentages of students who are ethnic minorities or who are eligible for free or reduced-price lunch are correlated with the academic profile, neither is statistically significant

Table 3. Model Estimation Predicting Change in Dropout Rate

Fixed Effects Model, Model 1 Model 2 +0.5/1.5 -0.5/1.5 +0.5/1.5 -0.5/1.5 Records Grouped by Year

Perfcomp Perfcomp Droprate Droprate

Intercept -0.0029 -0.0020 -0.005 0.0011 0.0022 -0.0029

Control Variables, Values From T1 (Baseline)

Percent non-White -0.0043 -0.0041 -0.0050 0.0080 -0.0014 -0.0141***

Percent free/reduced-price

lunch eligible 0.0089 0.0085 0.0159 -0.0177 -0.0121 0.0123

Membership 0.0000 -0.0000 0.0000 0.0000 0.0000 0.0000**

Suburban -0.0010 -0.0008 -0.0016 0.0006 0.0061 -0.0053***

Urban -0.0002 -0.0001 -0.0004 0.0001 0.0089* -0.0037*

Prior change in

performance composite -0.0002* 0.0002 -0.0002 -0.0014** -0.0000

*p < .05. **p < .01. ***p < .001.

(17)

in predicting the change score. School size and locale are associated with a change in the dropout rate. School size is negatively associated with increasing the performance composite, and, relative to rural schools, both suburban and urban schools are positively associated with changes to the performance composite: Suburban and urban schools were more likely to improve their performance composite over this period.

We posit that the schoolwide change in dropout rate would be positively associated with future change in the academic performance composite: If the dropout rate increases, the performance composite increases the next year. Results depicted in Model 2 show that this hypothesis is empirically supported. Prior year change in dropout rate is positively associated with change in the performance composite, and this measure is the strongest predictor in the model. All else being equal, increasing dropout rates by 1% from 2001–2002 to 2002–2003 would lead to a performance gain of .18 points from 2001–2002 to 2003–2004.

Furthermore, adding the prior change in dropout rate improves the model fit; the R2 increases from 0.05 in Model 1 to 0.24 in Model 2.

Next, we tested this model separately for high-performing, low- performing, high-dropout, and low-dropout schools. The results depicted in Table 4 indicate that all types of schools, regardless of baseline performance and dropout rates, significantly improve in academic performance after there is an increase in the dropout rate.

Figure 2 illustrates the estimated changes in the performance composite from 1999 to 2005 given set year-to-year changes in dropout rates. In

Table 4. Model Estimation Predicting Two-Year Change in Performance Composite

Fixed Effects Model, Records Model 1 Model 2 +0.5/1.5 -0.5/1.5 +0.5/1.5 -0.5/1.5 Grouped by Year

Perfcomp Perfcomp Droprate Droprate

Intercept 6.03*** 6.09*** 5.32*** 2.980* 2.979* 5.907***

Control Variables, Values From T1 (Baseline)

Percent non-White 1.124 1.074 2.725* 4.139* 2.166 -0.031

Percent free/reduced-price

lunch eligible -1.279 -1.300 -0.894 3.517 -4.390 -2.832

Membership -0.002*** -0.002*** -0.001** -0.002* -0.001 -0.001**

Suburban 1.295** 1.219** 0.588 -0.603 1.186 0.112

Urban 1.444** 1.379** 0.765 0.497 1.580 0.177

Prior change in dropout rate 18.395* 17.885* 26.282* 30.243* 33.741**

*p < .05. **p < .01. ***p < .001.

(18)

this example, the first year’s performance composite is set to 65%. The line with blocks shows the change in performance composite given a constant dropout rate. The line with the diamonds shows the change in performance given a dropout rate increasing by 1% each year, and the line with triangles shows the change in performance composite given a dropout rate that decreased by 1% each year. These are estimated changes for 1,000-student suburban schools with 35% non-White students and 20% free-lunch-eligible students.

In each scenario presented in Figure 2, the overall performance composite increases over time. In fact, statewide, the percentage of students performing at grade level has increased since the ABCs of Accountability program began. However, in schools where the dropout rate increased by 1% each year, the performance composite would improve more rapidly than it would in schools where the dropout rate did not change. In schools where the rate was constant, the performance composite would increase faster than in the schools where the dropout rate decreased each year. Given that relatively few students drop out of a school each year, a dropout increase of only a handful of students could make a difference in the performance composite.

Finally, we report the results of district-level fixed-effects models. Table

Note: This analysis begins with schools at the same performance composite.

Figure 2. Fixed-effects estimates predicting year-to-year changes in performance composite for schools with annual dropout rate changes of 0%, a 1% increase, and a 1% decrease

(19)

5 reports model estimations predicting change in dropout rate, including district (local education agencies [LEA]) fixed effects. The impact of prior schoolwide academic performance scores on future dropout rates remains significant and largely unchanged from the school-level fixed effects model.

Table 6 reports model estimations predicting change in academic performance, including district-level fixed effects. The impact of prior dropout rate on future academic performance remains significant, as found in school-level fixed effects models.

Table 5. Model Estimation Predicting Change in Dropout Rate Including LEA Fixed Effects

Fixed Effects Model, Records Grouped by Year Model 1 Model 2

Intercept 0.009* -0.012*

Control Variables, Values From T1 (Baseline)

Percent non-White 0.004 0.005

Percent free/reduced-price lunch eligible 0.001 -0.002

Membership 0.000* 0.000*

Suburban 0.001 0.000

Urban 0.001 0.001

Prior change in performance composite -0.003*

Note. LEA = local education agencies.

*p < .05. **p < .01. ***p < .001.

Table 6. Model Estimation Predicting Two-Year Change in Performance Composite Including LEA Fixed Effects

Fixed Effects Model, Records Grouped by Year Model 1 Model 2

Intercept 4.89*** 5.07***

Control Variables, Values From T1 (Baseline)

Percent non-White -1.94 -1.93

Percent free/reduced-price lunch eligible 1.87 1.65

Membership -0.00 -0.00*

Suburban 0.13 0.11

Urban 0.02 -0.01

Prior change in dropout rate 24.54***

Note. LEA = local education agencies.

* p <.05. **p <.01. ***p < .001.

(20)

CONCLUSION

Generally, schools with high academic performance have low dropout rates, and vice versa. This article uses a dynamic approach to analyze the relation between performance and dropout by examining the influence of the change in dropout rate on academic performance and the change in performance on dropout rate. These results provide insight into the complicated issue of school-level accountability. In this analysis of school- level changes in dropout rates and performance composites, we exam- ined two hypotheses about school-level changes in dropout rates and academic performance under conditions of accountability.

The ABCs of Accountability has not been associated with rapid changes in performance composites and dropout rates. Yet, across high schools in North Carolina, the average performance composite has increased and the dropout rate has decreased slightly over this period. We found support for each hypothesis about the dynamic relation between changes in the dropout rate and changes in the performance composite.

Proponents of accountability systems note that under such systems, with clear expectations, teachers and students have incentives to perform well.

These analyses show some evidence that improvements in school-level academic performance will lead to improvements (i.e., decreases) in school-level dropout rates. School context measures, such as percent minority, size, and locale, were not associated with the change in dropout rate, but the change in performance composite is negatively associated with the subsequent change in dropout rates. That is, in our sample, schools that demonstrated improved performance rates from one year to the next saw decreased dropout rates following these successes. This finding suggests that improvements on the EOC exams can improve the school academic climate and/or student academic self-concept, helping more students to remain in school.

Although the change in performance composite is associated with the change in dropout rates, it is not a good predictor of changes in dropout rates; the model’s explanator y power decreases when this measure is added. The effect of changing performance on dropout rates holds primarily in schools with high dropout rates. Schools with high dropout rates can significantly decrease their dropout rate when students’ test performance improves, whereas schools with low dropout rates have less room for improvement in this area. Policy makers and practitioners should note that in high schools that are “dropout factories,” focusing on academic performance of all students may help reduce the dropout rate as well.

We find more evidence of a negative side effect of the quest for

(21)

improved academic performance. Performance composites increased after the dropout rates increased. The model has greater explanator y power when prior change in dropout is included, and this measure is the strongest predictor in the model. This relation persists in each school type, suggesting that regardless of prior academic performance and dropout rates, schools can add to their performance composite scores by subtracting problematic students.

School leaders may not intentionally act in ways to drive out students who are likely to lower future school composite scores on achievement.

If under the accountability system, schools focus almost exclusively on student performance and neglect dropout rates, the system may have the unintended consequence of pushing academically struggling students out of school. School leaders may not deliberately attempt to discourage struggling students from remaining in school, yet policies may make some students feel unwelcome, and teachers and counselors may not have enough time to attend to both the pressures of performing well on tests and the demands of the most struggling students. In effect, accountability policies may give schools incentives to neglect the most difficult students.

The findings of this study suggest policy changes that could ensure that such practices do not occur. These accountability policies could change by incorporating a “penalty” for dropout students in calculating a school’s composite achievement score. Dropout students who do not take future tests could count as “failures” in achievement rather than being ignored. A school could be held accountable for the future performance of all incoming students, including those who drop out. Another policy reform could incorporate incentives for retaining students more directly by rewarding schools for improving their dropout rate across years. Yet a third policy reform could increase focus on evidence-based dropout prevention efforts. Future research should examine the detailed individual relation between academic histor y and dropout. Our data do not permit linking individual dropout records with their academic histories. We cannot tell how these students scored on past exams or predict how the school’s performance composite would have differed if these dropouts had taken those EOC exams. A link to a dropout’s prior academic performance would be tremendously valuable in highlighting students’ academic trajectories and reasons for leaving school.

Additional research could use qualitative approaches, such as inter- views and focus groups with dropouts, to find out their reasons for leaving school. To what extent did students feel under pressure because of an accountability system? How many of them left because they perceived school as a hostile environment where they could not succeed? Did they

(22)

use any school or community assistance to succeed on the exams? If so, what kind did they use? For those who did not use such ser vices, why not (e.g., no ser vices were available, ser vice schedule conflicted with other activities, student did not believe ser vices would help, student thought ser vices would be boring)? Policy makers could use such information to develop and implement programs to help struggling students succeed on these exams.

Further research should examine the mechanisms through which schools’ scores improve when the dropout rates rise. One reason for improved performance might be that the sample of students taking the end-of-the-year composite tests changed. Those likely to have the worst performance on those exams simply did not take the exams, and the school average performance increased. Is it simply a matter of subtracting those negative test scores from the overall average? Alternatively, per- haps the school environment changes when students with academic and disciplinar y problems leave, and performance improves among those students who remain. Once at-risk students leave school, teachers may be better able to educate the remaining students in their classes such that performance on the next year’s end-of-the-year tests increases. Teachers do not have to spend as much time handling academically struggling students and have more time for those who remain. The remaining students may be less distracted and more focused when they have fewer disruptive peers in the classroom. Researchers could inter view principals, teachers, and at-risk students who graduate to determine whether and how the school environment changes once struggling students leave.

Additional research could examine other exclusions from test taking, such as long-term suspensions and referrals to alternative programs. Do such penalties and referrals occur more often in the spring when these students would be excluded from exams? Do schools’ test scores rise when students with disciplinar y problems are excluded from taking the tests?

Educators and students do seem to respond to incentives, and, particularly in schools with high dropout rates, the dropout rates decline after the performance composite increases. In these schools, the accountability systems may help all students. However, accountability systems seem to provide additional incentives to neglect the most challenging students.

Regardless of the mechanisms by which it occurs, accountability systems need to remove any indirect benefit that a school may receive when its dropout rate increases. Schools should be held accountable for those who drop out of school. Accountability could encompass increasing the dropout age and having the school’s performance composite include scores of zero on EOG tests for those who leave school. Given the personal

(23)

and social costs of dropping out, accountability systems need to place more emphasis on dropout prevention.

Acknowledgments

Authors would like to thank Philip Cook and three anonymous reviewers for their helpful comments on this work. The authors acknowledge the support of the Beyond Test Scores Group at Duke University and its grant from the Smith Richardson Foundation. Dodge acknowledges NIDA Grants K05DA015226 and P30DA023026.

Notes

1. Students can take these courses and pass these exams during any grade and could complete their graduation requirements before their senior year in high school.

2. Ver y few of these students had the opportunity to choose a different type of school.

According to the National Center for Education Statistics, Common Core of Data, Public School Universe, in 2005 (the last year of our study), North Carolina had 14 magnet and 7 charter high schools out of 333 high schools. The earlier years did not have any such high schools.

References

Alexander, K. L., Entwisle, D. R., & Dauber, S. L. (2003). On the success of failure: A reassess- ment of the effects of retention in the primar y school grades (2nd ed.). Cambridge, England:

Cambridge University Press.

Br yk, A., & Thum, Y. (1989). The effects of high school organization on dropping out: An explorator y analysis. American Educational Research Journal, 26, 353–383.

Carnoy, M., & Loeb, S. (2002). Does external accountability affect student outcomes? A cross-state analysis. Educational Evaluation and Policy Analysis, 24, 305–311.

Caspi, A., Wright, B. R., Moffitt, T. E., & Silva, P. A. (1998). Early failure in the labor mar- ket: Childhood and adolescent predictors of unemployment in the transition to adult- hood. American Sociological Review, 63, 424–451.

Catterall, J. S. (1985). On the social costs of dropping out of school. Stanford, CA: Stanford University Press.

Catterall, J. S. (1989). Standards and high school dropouts: A national study of tests required for graduation. American Journal of Education, 98, 1–34.

Clarke, M., Haney, W., & Madaus, G. (2000). High stakes testing and high school comple- tion. National Board on Educational Testing and Public Policy Statements, 1, 2–11.

Cullen, J. B., & Reback, R. (2006). Tinkering toward accolades: School gaming under a perfor- mance accountability system (National Bureau of Education Research Working Paper Series, No. W12286). Cambridge, MA: National Bureau of Economic Research.

Darling-Hammond, L. (2006). No Child Left Behind and high school reform. Harvard Educational Review, 76, 642–667.

Figlio, D. N., & Getzler, L. S. (2002). Accountability, ability and disability: Gaming the system (National Bureau of Economic Research Working Paper Series No. 9307). Cambridge, MA: National Bureau of Economic Research.

Fine, M. (1986). Why urban adolescents drop into and out of high school. Teachers College Record, 87, 393–409.

(24)

Fine, M. (1991). Framing dropouts: Notes on the politics of an urban high school. Albany: State University of New York Press.

Goldschmidt, P., & Wang, J. (1999). When can schools affect dropout behavior? A longitu- dinal multilevel analysis. American Educational Research Journal, 36, 715–738.

Greene, W. (2003). Econometric analysis (5th ed.). London: Prentice-Hall International.

Hanushek, E. A., & Raymond, M. E. (2002). Sorting out accountability systems. In W. E.

Evers & H. J. Wahlberg (Eds.), School accountability (pp. 75–104). Stanford, CA: Hoover Press.

Jacob, B. A. (2001). Getting tough? The impact of high school graduation exams.

Educational Evaluation and Policy Analysis, 23, 99–121.

Jacob, B. A. (2005). Accountability, incentives, and behavior: The impact of high-stakes test- ing in the Chicago public schools. Journal of Public Economics, 89, 761–796.

Jimerson, S. R. (1999). On the failure of failure: Examining the association between early grade retention and education and employment outcomes during late adolescence.

Journal of School Psychology, 37, 243–272.

Jordan, W. J., & McPartland, J. (1996). Exploring the causes of early dropout among race- ethnic and gender groups. Youth and Society, 28, 62–94.

Lee, V., & Loeb, S. (2000). School size in Chicago’s elementar y schools: Effects on teach- ers’ attitudes and students’ achievement. American Educational Research Journal, 37, 3–31.

Lee, V., & Smith, J. (1997). High school size: Which works best and for whom? Educational Evaluation and Policy Analysis, 19, 205–227.

Letgers, N. E., Balfanz, R., Jordan, W., & McPartland, J. M. (2002). Comprehensive reform for urban high schools. New York: Teachers College Press.

Lillard, D. R., & DeCicca, P. P. (2001). Higher standards, more dropouts? Evidence within and across time. Economics of Education Review, 20, 459–473.

McDill, E. L., Natriello, G., & Pallas, A. M. (1986). A population at risk: Potential conse- quences of tougher school standards for student dropouts. American Journal of Education, 94, 135–181.

McNeal, R., Jr. (1997a). Are students being pulled out of high school? The effect of adoles- cent employment on dropping out. Sociology of Education, 70, 206–220.

McNeal, R., Jr. (1997b). High school dropouts: A closer examination of school effects. Social Science Quarterly, 78, 209–222.

McNeil, L. M., Coppola, E., Radigan, J., & Heilig, J. V. (2008). Avoidable losses: High-stakes accountability and the dropout crisis. Education Policy Analysis Archives, 16(3). Retrieved March 17, 2008, from http://epaa.asu.edu/epaa/v16n3/

North Carolina Department of Public Instruction. (1998). 1997–98 ABCs Report Card, Volume I–Executive Summar y. Retrieved from http://www.ncpublicschools.org/

abc_results/results_98/executive_summar y.html#executive

North Carolina Department of Public Instruction. (2000). Dropout data collecting and report- ing: procedures Manual, 2000 Edition. Raleigh: Author.

North Carolina Department of Public Instruction. (2005). ABCs dropout rate calculation, 2005. Retrieved from http://www.ncpublicschools.org/docs/accountability/report- ing/abcsdropoutratecalc.pdf

Rumberger, R. (1987). High school dropouts: A review of issues and evidence. Review of Educational Research, 57, 101–121.

Rumberger, R. (1995). Dropping out of middle school. A multilevel analysis of students and schools. American Education Research Journal, 32, 583–625.

Rumberger, R. (2004). Why students drop out of school. In G. Orfied (Ed.), Dropouts in America: Confronting the graduation rate crisis (pp. 131–155). Cambridge, MA: Har vard Education Press.

(25)

Rumberger, R., & Thomas, S. (2000). The distribution of dropout and turnover rates among urban and suburban high schools. Sociology of Education, 73, 39–67.

Stearns, M. E., & Glennie, E. J. (2006). When and why dropouts leave school. Youth and Society, 38(1), 29–57.

Thornberr y, T., Moore, M., & Christenson, R. (1985). The effect of dropping out of high school on subsequent criminal behavior. Criminology, 23, 3–18.

United States Department of Education: National Center for Education Statistics. (2001).

Dropout rates in the United States: 2000. Washington, DC: U.S. Government Printing Office.

Wooldridge, J. (2002). Economic analysis of cross section and panel data. Cambridge, MA:MIT Press.

APPENDIX

Pairwise Correlations: Performance Composites and Dropout Rates, by Year

ELIZABETH GLENNIE is a senior education research analyst at RTI International. Much of her work focuses on high school reform and programs to enhance opportunities for postsecondar y educational access.

Recent publications: Edmunds, J., Bernstein, L., Glennie, E., Willse, J. T., Arshavsky, N., Unlu, F., et al. (2010). Preparing students for college: The implementation and impact of the Early College High School model.Peabody Journal of Education, 85, 348–364; and Clotfelter, C. C., Glennie, E. J., Ladd, H. F., & Vigdor, J. (2008). Would higher salaries keep teachers in high-poverty schools? Evidence from a policy inter ven- tion in North Carolina. Journal of Public Economics, 92, 1352–1370; and Stearns, M. E., & Glennie, E. (2006). When and why dropouts leave school. Youth and Society, 38(1), 29–57.

KARA BONNEAU is the associate director for data management, North Carolina Education Research Data Center. Her research interests include education policy, race and ethnicity, social stratification, and program evaluation. Recent publication: Stearns, E., Buchmann, C., & Bonneau,

Year Performance Composite Dropout Rate

1998–1999 N/A .4529***

1999–2000 .8611*** .4665***

2000–2001 .9248*** .6756***

2001–2002 .9073*** .6413***

2002–2003 .8990*** .6830***

2003–2004 .9074*** .5923***

2004–2005 .9000*** .6369***

*** p < .001.