First-Year Report: Data

(1)

Impact of the

Payne School Model on Student Achievement

Cannon County Schools Woodbury, Tennessee

First-Year Report:

2004–05 Data

William W. Swan, Ed.D.,

The University of Georgia

Athens, GA.

(2)

Impact of the Payne School Model on Student Achievement Cannon County Schools

Woodbury, Tennessee First-Year Report: 2004–05 Data

William W. Swan, Ed.D.

The University of Georgia Athens, GA

Abstract AbstractAbstract Abstract

Analyses of Covariance (ANCOVA)—using the Tennessee Comprehensive Assessment Program (TCAP) in Mathematics and Reading for Grades 2, 4, 5, 6, 7, and 8 in 2004–05 as dependent variables and the TCAP in Mathematics and in Reading in 2003–04 for the same students in the prior year in each area respectively as the covariates—were conducted for two groups of students. One group was students of teachers scoring in the High Model Fidelity implementation and another was students of teachers scoring in either Low Model Fidelity or no implementation.

Statistically significant results were identified for Grade 2 in both Mathematics and Reading, favoring the High Model Fidelity group, and for Grade 8 in Reading, favoring the Other group. Of the total of 12 ANCOVAs—six for Mathematics and six for

Reading—eight sets of adjusted means favored the High Model Fidelity group. No analyses of disaggregation variables were possible because these student data were not provided. The results support the positive impact High Model Fidelity

Implementation of the Payne School Model on increasing student achievement in elementary and middle schools in Mathematics and Reading.

For more information please contact: Center for Study of Economic Diversity, PO Box 665, For more information please contact: Center for Study of Economic Diversity, PO Box 665, For more information please contact: Center for Study of Economic Diversity, PO Box 665, For more information please contact: Center for Study of Economic Diversity, PO Box 665, Highlands, TX 77562, 800

Highlands, TX 77562, 800 Highlands, TX 77562, 800

Highlands, TX 77562, 800----468468468----9950 (phone), 2814689950 (phone), 2819950 (phone), 2819950 (phone), 281----426426426426----6930 (fax).6930 (fax).6930 (fax). 6930 (fax).

(3)

Introduction and PurposeIntroduction and PurposeIntroduction and Purpose Introduction and Purpose

The federal No Child Left Behind Act (2001) and corresponding state legislation throughout the United States requires that schools use “research based” programs to increase student achievement in all academic areas, with particular emphasis on reading/English/language arts and mathematics. Consistent with these mandated foci on student achievement, Dr.

Ruby K. Payne initiated research to determine the impact of the implementation of the Payne School Model—A Framework for Understanding Poverty, Learning Structures, and Meeting Standards and Raising Test Scores materials and training—on student achievement in the areas of reading/English/language arts and mathematics.

Context for Cannon County Schools Context for Cannon County Schools Context for Cannon County Schools Context for Cannon County Schools

Cannon County Public Schools, located in Woodbury, Tennessee, serves 2,104 students from kindergarten to 12^th grade. There are six elementary schools in the district that serve pre-kindergarten to eighth-grade students and one high school that serves ninth- to 12^th- grade students. All seven schools in the district were involved with aha! Process, Inc.

throughout the 2004–05 school year. All staff members received A Framework for Understanding Poverty, Learning Structures, and Meeting Standards and Raising Test Scores training from aha! Process consultants. During the course of the 2004–05 school year, technical assistance was provided quarterly by aha! Process consultants.

Elementary (K–5)

An aha! Process consultant provided four technical-assistance sessions to selected elementary teachers. Teachers met with the consultant by grade level. During the first semester, technical-assistance sessions focused on:

• Discussing how to effectively work with parents

• Completing student data grids

• Developing time and content grids

• Direct-teaching input strategies

• Identifying interventions for those students who are struggling, based on student data grids

During the second semester, the aha! Process consultant conducted classroom

observations and continued to meet with the teachers for technical-assistance sessions.

These sessions focused on:

• Reviewing and revising time and content grids

• Completing a resource analysis on the four lowest students in the classroom, based on the student data grid

• Incorporating mental models into daily lesson plans

• Sharing strategies among teachers

(4)

Secondary

Districtwide, all sixth- to eighth-grade students are served on one of the six elementary campuses, and all ninth- to 12^th-grade students are served at Cannon County High School.

All sixth- to 12^th-grade teachers received content-specific technical assistance provided by aha! Process consultants.

English/Language Arts

Four technical-assistance sessions were provided to English/language arts teachers by aha!

Process consultants, with each teacher receiving on average two days of technical assistance. During the first semester, technical-assistance sessions focused on:

• Reviewing concepts from Learning Structures

• Developing common bell work to address targeted standards

• Identifying target students using the student data grid

• Introducing the Payne Lesson Design

During the second semester, technical assistance focused on the following—and also included classroom observations by an aha! Process consultant:

• Reviewing the Payne Lesson Design

• Developing common bell work

• Creating step sheets

• Aligning curriculum to standards

Math

An aha! Process consultant provided four technical-assistance sessions to math teachers, with each teacher receiving on average 1.5 days of technical assistance. During the first semester, the technical-assistance sessions focused on:

• Reviewing concepts from Learning Structures

• Creating 10-question tests

• Identifying target students based on data grids

During the second semester, technical-assistance sessions focused on:

• Reviewing input strategies

• Creating 10-question tests

• Designing a consistent plan and label strategy for math problem solving

• Completing a resource analysis of target students

(5)

Science

An aha! Process consultant provided four technical-assistance sessions to science teachers, with each teacher receiving on average two days of technical assistance. During the first semester, the technical-assistance sessions focused on:

• Identifying specific strategies for planning, sorting, and scientific inquiry

• Aligning curriculum with standards

• Teachers’ success with strategies, specifically with mental models, reading strategies, and plan and label strategies

• Incorporating step sheets and question making into daily lessons

• Unpacking the standards

• Developing strategies to improve students’ performance on benchmark assessments

Social Studies

An aha! Process consultant provided four technical-assistance sessions to social studies teachers, with each teacher receiving on average two days of technical assistance. During the first semester, the technical-assistance sessions focused on:

• Embedding instructional strategies from Learning Structures, particularly mental models and plan and label strategies

• Reviewing instructional strategies

• Introducing question-making strategies

• Unpacking the standards

• Using the Payne Lesson Design

Methodology Methodology Methodology Methodology

The research design to determine impact had two dimensions. The first dimension was establishing Model Fidelity at each school. If teachers were not implementing the Payne School Model effectively, differences in student achievement could not be attributed to the use of the model. The second dimension was determining the statistical significance of the impact of the Payne School Model on student achievement in reading and mathematics. The design for this analysis was a post-test-only comparison design for two groups (High Model

(6)

Fidelity vs. Other) using the analyses of covariance to adjust for initial differences between the groups of students.

Model Fidelity

The Instructional Framework Scale—Observation (2003) was used to assess the fidelity of the implementation of the Payne School Model. This instrument consisted of 47 indicators criterion-referenced to key model components/activities and was used by the aha! Process consultants to determine the fidelity of the implementation of the Payne School Model. The median inter-rater reliability for the instrument is .83, with a range from .72 to .95.

Analysis of Student Achievement Data

A post-test-only comparison design for two groups—High Model Fidelity (50% or higher score on the Instructional Framework Scale—Observation) vs. Other (49% or less on the

Instructional Framework Scale—Observation or not observed)—using the analyses of covariance to adjust for initial differences between two groups was employed to determine the statistical impact of the implementation of the Payne School Model on student

achievement. No comparison school district was available.

The independent variable was the implementation or non-implementation of the Payne School Model (High Model Fidelity vs. Other). The expected level of first-year implementation is 50%, which was the criterion used to differentiate the two groups.

The dependent variables were standardized test scores in the TCAP in Reading and Mathematics. (See the following website for more information on the Tennessee

Comprehensive Assessment Program, Accountability, Assessment/Evaluation/ Research, and the 2005 Report Card: www.state.tn.us/education/.) The covariates were 2003–04 prior test scores for students in 2004–05 on the TCAP in Mathematics and Reading, respectively. Analyses were conducted for the total groups at each grade level. No disaggregation analyses could be conducted because demographic descriptors were not available. The level of statistical significance was set at p < .05.

Results Results Results Results

The results are presented in two sections—Model Fidelity and Student Achievement.

Model Fidelity

aha! Process consultants used the Instructional Framework Scale—Observation (2003) to observe teachers who were identified either by the building principal or the aha! Process consultant as potentially implementing the Payne School Model effectively. This resulted in from one to five teachers being observed in each of the five schools in Grades 2–8. The number of teachers per grade who met the criteria for High Model Fidelity group ranged from one to three; the remaining teachers who were observed and those teachers who were not observed ranged from six to nine, and they were placed in the Other group. Of the nine teachers who were in the High Model Fidelity group, six had a score of 100%, and one had a score of 98%.

(7)

Student Achievement

Tables 1 and 2 (see final two pages) contain an overview summary of the statistical analyses for Mathematics and Reading by grade for Grades 2, 4, 5, 6, 7, and 8. The data for Grade 3 had only one student, and thus no analyses were conducted for this grade. No analyses on disaggregations were conducted because no data on gender, race/ethnicity, LEP (Limited English Proficiency), poverty, or IDEA (Individuals with Disabilities Education Act of 2004) disability were provided. A narrative explanation of the results is provided below for each analysis conducted.

MathematicsMathematicsMathematicsMathematics –––– The ANCOVA-adjusted means for Mathematics for the High Model Fidelity group were higher than for the Other group for four four four four of the six gradesof the six gradesof the six grades (Grades 2, 5, 6, of the six grades and 7). There was one statistically significant difference in the means for Grade 2 (F = 14.030, p < .000) in favor of the High Model Fidelity group. While the ANCOVA-adjusted means were not statistically significantly different for other grades, the directional difference was maintained in favor of the High Model Fidelity group.

Reading Reading Reading

Reading –––– The ANCOVA-adjusted means for Reading for the High Model Fidelity group were higher than for the Other Group for four of the four of the four of the four of the gradesgradesgrades (Grades 2, 5, 6, and 7). There grades were two statistically significant differences in the adjusted means—one for Grade 2 (F = 24.554, p < .000) in favor of the High Fidelity Group and one for Grade 8 (F = 4.064, p <

.046) in favor of the Other group.

High S High S High S

High Schoolchoolchoolchool –––– The only data available for the high school were descriptive statistics for Math and Reading in 2005, with the following results:

• 2005 Mathematics: Number of Students = 62; Range = 19–51;

Mean = 36.47; Standard Deviation = 7.764

• 2005 Reading: Number of Students = 183; Range = 0–55;

Mean = 39.19; Standard Deviation = 9.021

The numbers of students are different because of incomplete data on all students. No analyses could be conducted because the high school test is a pass/fail test, and no other assessment was provided. While eighth-grade data could theoretically be used as a

covariate, the time period between eighth grade and 11^th grade precludes a meaningful adjustment.

Discussion Discussion Discussion Discussion

Discussion of these results is constrained by several factors. First, this was the first year of implementation of the Payne School Model. While the Model Fidelity scores for some teachers significantly exceeded the 50% criterion of the first year, this score reflected performance during the middle/end of the Spring Semester and is not an index for performance throughout the year. Even higher performance would be anticipated in the second-year implementation. Second, no comparison schools were available to provide

(8)

direct comparisons to schools not even associated with the Payne School Model. Third, there were no data on demographic variables to assist in interpretation of impact of the Payne School Model on gender, race/ethnicity, LEP, poverty, or IDEA disability.

The pattern of the ANCOVA-adjusted means of the High Model Fidelity group exceeding those of the Other group in eight of the 12 analyses is positive, especially for the first year of implementation. Three patterns deserve additional investigation, perhaps by focusing on the disaggregation variables—the overall pattern across both Mathematics and Reading; Grade 2 and Grade 4; and Grade 4 and 8. The overall pattern of results in Mathematics and Reading in terms of directions of difference is the same; this is relatively rare and should be studied to see if it continues and why. The results for Grade 2 (in favor of the High Model Fidelity group) and for Grade 4 (in favor of the Other group) for both Mathematics and Reading merits further examination in subsequent years to determine if the pattern persists over time for these groups of students. And the results for Grade 4 and Grade 8 that favor the Other group should be examined in the coming year to see if this pattern recurs for these two years and why.

While conducting ANCOVA analyses on the disaggregation variables and making

comparisons to other schools would have increased the value and depth of these findings, these results strongly support the positive impact High Model Fidelity implementation of the Payne School Model on increasing student achievement in all six grades examined for both Mathematics and Reading—except Grade 8 Reading.

Recommendations Recommendations Recommendations Recommendations

Based on these results, two recommendations for continuing improvement should be considered regarding the examination of the impact of the Payne School Model on student academic achievement.

1. Demographic data must be provided, along with achievement data for each student, to determine differential impact on gender, race/ethnicity, LEP, poverty, and IDEA disability. Identifying the differential impact of the Payne School Model on these disaggregate groups is crucial to being able to generalize results.

2. Demographically similar schools that are not using the Payne School Model must be identified in order to more thoroughly examine the positive nature of the results from this study.

(9)

References References References References

Payne, Ruby K. (2005). A Framework for Understanding Poverty (Fourth Revised Edition). Highlands, TX: aha! Process.

Payne, Ruby K. (2003). Instructional Framework Scale—Observation. Highlands, TX: aha! Process. Retrieved from

http://www.ahaprocess.com/School_Programs/Research_&_Development/.

Research design package—analyses, reliability information for instruments. (2005).

Highlands, TX: aha! Process. Retrieved from

http://www.ahaprocess.com/School_Programs/Research_&_Development/.

Tennessee Department of Education. (2005). Tennessee Comprehensive Assessment Program, Accountability, Assessment/Evaluation/Research, 2005 Report Card.

Retrieved from www.sate.tn.us/education/.

(10)

Table 1 Table 1 Table 1 Table 1

Summary of Results for ANCOVA for MathematicsMathematicsMathematicsMathematics—2004–05 Cannon County Schools (Tennessee): High Model Fidelity (HMF) vs. Other

Grade GradeGrade

Grade GroupGroup GroupGroup Sample SizeSample SizeSample SizeSample Size CovariateCovariateCovariateCovariate ANCOVA ANCOVA ANCOVA ANCOVA –––– Adjusted Adjusted Adjusted Adjusted Means

Means Means Means

Direction of Direction of Direction of Direction of Difference Difference Difference Difference

F F F

F ProbabilityProbabilityProbabilityProbability

2^nd HMF 17 1^st Grade 612.541 HMF > Other 14.030 p < .000*

Other 112 577.391

4^th HMF 16 3^rd Grade 470.959 HMF < Other 3.008 p < .085

Other 126 486.815

5^th HMF 13 4^th Grade 514.274 HMF > Other 1.940 p < .167

Other 90 503.827

Other 122 524.505

7^th HMF 32 6^th Grade 536.678 HMF > Other .080 p < .778

Other 101 535.320

8^th HMF 50 7^th Grade 550.204 HMF < Other .217 p < .642

Other 90 551.998

* p < .001

(11)

Table 2 Table 2 Table 2 Table 2

Summary of Results for ANCOVA for ReadingReadingReadingReading—2004–05

Cannon County Schools (Tennessee): High Model Fidelity (HMF) vs. Other

Grade GradeGrade

Grade GroupGroup GroupGroup Sample SizeSample SizeSample SizeSample Size CovariateCovariateCovariateCovariate ANCOVA ANCOVA ANCOVA ANCOVA –––– Adjusted Adjusted Adjusted Adjusted Means

Means Means Means

Direction of Direction of Direction of Direction of Difference Difference Difference Difference

F F F

F ProbabilityProbabilityProbabilityProbability

2^nd HMF 16 1^st Grade 655.790 HMF > Other 24.554 p < .000*

Other 111 620.949

4^th HMF 16 3^rd Grade 476.642 HMF < Other 3.080 p < .081

Other 126 491.784

Other 90 508.540

Other 121 518.076

Other 100 526.813

8^th HMF 90 7^th Grade 535.236 HMF < Other 4.064 p < .046**

Other 49 541.671

* p <.001

** p < .05