The Development and Implementation of Local Assessment Systems in Maine Schools: A Progress Report

(1)

Maine Education Policy Research Institute College of Education and Human Development The University of Maine 5766 Shibles Hall, Orono, ME 04469-5766 207/581-2493 • FAX 207/581-9510 September 2005

The Development and

Implementation of Local Assessment

Systems

in Maine Schools:

A Progress Report

(2)

The Development and Implementation

of Local Assessment Systems

in Maine Schools:

A Progress Report

Prepared by the

Maine Education Policy Research Institute

Co-Evaluators: Janet Fairman Research Team: Debra Allen

Walter J. Harris Barbara Bales

Amy Cates Lori Smith Lisa Tyson

Local Assessment System Implementation Study II Advisory Committee:

Gary MacDonald, MSAD #72 Superintendent Patrick Phillips, Deputy Commissioner of Education Pamela G. Rolfe, Local Assessment Coordinator, Maine

Department of Education

Jill M. Rosenblum, Project Coordinator, Maine Mathematics and Science Alliance

David Silvernail, Co-Director, Maine Education Policy Research Institute

For additional information, contact Dr. Walter Harris, Co-Director, Maine Education Policy Research Institute, The University of Maine, (207) 581-2467 or [email protected].

(3)

Table of Contents

Introduction ... 1

Research Questions... 1

Overview of Methodology ... 2

Organization of this Report ... 3

Part I: Summary of Findings ... 4

Alternative Education... 7

Special Education ... 8

English Language Learners (ELL) ... 8

Career and Technical Education (CTE) ... 9

Conclusions ... 13

Evidence of Progress Made... 13

Mixed Support for LAS... 13

Time as a Barrier... 13

Financial Costs... 14

Human Capacity ... 14

Managing Social Change ... 14

Communications and Public Relations... 14

Technical Assistance ... 15

Readiness to Certify Students’ Achievement of the Learning Results... 15

Professional Development ... 15

Expectations for Student Achievement ... 15

Use of LAS Data... 16

Part II: The Status of LAS Development and Implementation: A Statewide Survey, Spring 2005 ... 17

Demographics ... 17

Leadership of LAS Implementation and Design ... 19

Current Status of LAS Development and Implementation ... 21

English Language Arts ... 21

Mathematics... 23

Overall Progress in Other Content Areas... 24

Number of Certification Assessments Planned ... 26

Distribution of Assessments by Type... 26

Current Use of Local Assessment System Data ... 27

Status of Selected Implementation Activities ... 28

Providing Supplemental Instruction Prior to Replacement Assessments ... 30

Developing a Means of Determining the Value of Prior Education Experiences for Transfers... 30

Collaboration Among SAUs ... 31

Level and Type of Help Needed ... 31

Confidence ... 33

Percent of Graduates Estimated to Meet Learning Results... 33

Understanding the Requirements... 34

Changes to Support the Class of 2008 Meeting the Standards ... 34

Positive and Negative Impacts of LAS Implementation... 34

Boosters and Barriers to LAS Implementation... 35

Supplementary Analysis ... 35

Distribution of Advanced and Less Advanced SAUs by Region ... 36

Leadership in Advanced and Less Advanced SAUs ... 36

(4)

Help Needed by Advanced and Less Advanced SAUs ... 37

Use of LAS data by Advanced and Less Advanced SAUs... 38

Comparison of 2004 and 2005 Survey Items ... 39

Progress in English Language Arts ... 39

Progress in Mathematics ... 43

Summary... 46

Part III: Interviews and Focus Groups: A Qualitative Study of LAS Progress in Fourteen Representative School Administrative Units (SAUs)... 50

Demographics for SAU Sample ... 50

Data Collection for SAU Sample ... 51

Data Analysis ... 52

SAU Organization for LAS Development and Implementation... 52

Personnel Resources for LAS Work... 53

Support and Leadership for LAS... 53

Superintendents... 54

Principals... 55

Teachers ... 56

Progress Toward LAS Development and Implementation ... 57

Curriculum Alignment ... 57

Selecting Performance Indicators... 57

Administering Assessments ... 59

Developing Rubrics ... 60

Single Scoring Assessments ... 60

Double Scoring Assessments... 60

Establishing Reliability ... 61

Professional Development to Support LAS Implementation ... 62

Time Spent on LAS Work... 63

Assessment of Students in Alternative Education Programs ... 63

Assessment of Students in Special Education Programs ... 65

Assessment of English Language Learners (ELL) ... 68

Assessment of Career and Technical Education (CTE) Students... 69

LAS Data Management ... 71

LAS Data Reporting ... 71

LAS Policy Development... 72

Use of Assessment Data ... 72

Impacts of LAS... 73

Impacts on Curriculum... 73

Impacts on Instruction and Assessment ... 74

Impacts on Teachers ... 75

Impacts on Students... 77

Impacts on Parents... 78

Certification of Student Achievement ... 78

Supports and Barriers for LAS Development and Implementation ... 79

Supports... 79

Barriers ... 79

Technical Assistance and Other Resource Needs... 81

Summary ... 82

Part IV: Appendices... 85

Appendix A: 2005 Survey... 86

Appendix B: LASIS II Site Visit Interview Protocols ... 93

(5)

Appendix D: Focus Group Protocols ... 102 Appendix E: Changes to Support the Class of 2008 ... 103 Appendix F: Boosters and Barriers ... 109

(6)

The Development and Implementation of Local Assessment Systems in Maine Schools:

A Progress Report

Introduction

In 1996 the Maine Legislature established the Maine Learning Results, a descriptive statement of the expectations held for what all Maine students are expected to know and be able to do as a result of their public school experience. Accompanying these expectations is the requirement that students’ achievement of the Learning Results be assessed with a combination of the Maine Education Assessment (MEA) and additional assessments that are locally developed. The locally developed assessments or Local Assessment Systems (LASs) are to be valid and reliable and ultimately provide an accurate representation of students’ achievement of the Learning Results.

Technical and methodological resources were provided by the Maine Department of Education (MDOE) to assist school administrative units (SAUs) with the complex tasks of developing LASs. These resources included Measured Measurers: Technical Considerations for

Developing a Local Assessment System, LAS Guide: Principles and Criteria for the Adoption of Local Assessment Systems, Considering Consistency: Conceptual and Procedural Guidance for Reliability in a Local Assessment System, and LAS Guide with Embedded Components for Accountability and Alternate Assessment. In addition, MDOE providedregional workshops, consultation on request, and two web-based inventories of assessment tasks, the Maine Assessment Portfolio (MAP) and Local Assessment Development (LAD), both keyed to the content areas and standards of the Learning Results.

This report summarizes data collected in the spring of 2005 by the Maine Education Policy Research Institute (MEPRI) to assess the current status and progress of Maine SAUs in designing and implementing LASs. It is the third consecutive annual report on this topic and is written for MDOE personnel, school personnel, and policymakers to inform their efforts to assist schools in the creation of valid and reliable assessment systems that contribute to teaching and learning, hold SAUs accountable, and certify students’ achievement of the Learning Results.

Research Questions

Research questions were designed by MEPRI staff and MDOE personnel to guide this study. These questions were developed in a context that included the data from previous studies, the perspectives that MDOE personnel had gained from their work in SAUs on topics related to the design and implementation of LASs, and the needs for information of MDOE personnel as they prepared to design and provide technical assistance to SAUs. The research questions to be addressed by the study included the following:

1. What progress has been made in the design and implementation of Local Assessment Systems (LASs) in Maine SAUs since June 2004 (when the results of the 2004 statewide survey were reported)?

(7)

2. What is the current status and technical quality of SAUs’ efforts to select and score assessments in individual content areas and grade spans?

3. What progress has been made, what best practices exist, and what issues need to be resolved in SAUs’ efforts to develop LASs that will assess the learning of students in programs in Alternative Education, Special Education, Career and Technical

Education and programs for English Language Learners?

4. How is LAS data being collected, managed, reported, and communicated? 5. What policies relevant to LASs have been created or modified by SAUs?

6. How are SAUs responding to the systemic use of LAS data for formative assessment of curriculum and instruction?

7. What progress are SAUs making toward certification of student achievement of the

Learning Results, and how is LAS data informing this progress? 8. What technical assistance or other resources are needed?

9. What factors, such as leadership structures and professional development opportunities, have been helpful in creating and implementing LASs?

10. How have SAUs used time and other resources to meet both the state and local requirements of implementing LASs?

Overview of Methodology

Three methods were used to gather sufficient data to address all 10 research questions:

1. A survey was mailed to all SAUs in May 2005 with a request for it to be completed by the individual in each SAU who was most familiar with the status of LAS development (Appendix A). During the time that respondents were completing the survey potential policy changes effecting LASs were being discussed in the MDOE and the Education Committee of the Legislature and were reported in the news media. These discussions may have impacted responses to the survey. The overall return rate for the survey was 83% with a range of 52% to 97% across the nine superintendent regions. A detailed analysis of the survey results constitutes Part II of this report. 2. Interviews of LAS leadership personnel, and teachers who were not directly involved

in LAS leadership were conducted in 14 SAUs that represented all geographic regions (except the unorganized territories) and types of administrative structures. This

sample also represented both large and small SAUs and SAUs with both high and low percentages of students in free-and-reduced-lunch programs. A total of 185

(8)

included in Appendix B. Like the survey, these interviews were conducted during a time in which policy changes were being discussed by MDOE personnel, the

Education Committee of the Legislature and reported in the news media. These discussions may have affected responses to interview questions. Focus groups were conducted with representatives of Career and Technical Education, Alternative Education and Special Education programs. Due to scheduling difficulties, a focus group was not conducted with representatives of English Language Learner (ELL) programs, but three Directors of ELL programs were interviewed, and ELL was a focus of interviews in all 14 SAUs. The interview protocols for focus groups can be found in Appendix D.

3. Documents, policies, assessment templates and checklists of LAS tasks completed were requested of all fourteen SAUs. Analysis of these documents informed the study. A detailed analysis of data gathered through interviews and focus groups constitutes Part III of this report.

Organization of this Report

This report is divided into four parts:

• Part I draws on data from the survey, interviews, and focus groups and provides a summary

of the findings related to each research question and overall conclusions.

• Part II describes and analyzes the data resulting from the survey and includes a

Supplementary Analysis that divides responding SAUs into an Advanced Group and a Less Advanced Group for comparison on several key variables. Part II also includes a

comparison of the 2005 survey with a similar survey conducted in spring 2004.

• Part III summarizes the interview and focus group data collected in 14 representative

SAUs.

• Part IV, Appendices, includes the survey, summaries of narrative responses to survey

(9)

Part I: Summary of Findings

1. What progress has been made in the design and implementation of Local Assessment Systems (LASs) in Maine SAUs since June 2004 (when the results of the 2004 statewide survey were reported)?

SAUs have continued to make progress in developing and implementing their LASs. While most SAUs were developing their LASs in spring 2004, (e.g., selecting performance indicators and selecting assessments), in spring 2005 SAUs were beginning to implement their LASs (e.g., administering and scoring assessments). SAUs have spent most of this year piloting and revising assessments, rubrics, and LAS templates. The piloting of assessments has revealed the need to revise assessment tasks and LAS templates. In some SAUs, educators are revisiting decisions about the number and type of assessments to be administered and the schedule of assessments. Some SAUs had selected a number of assessments and performance indicators in excess of the number recommended by MDOE, and are reducing these numbers. In other cases, educators realized that selected assessments did not work well, and they needed to select different assessments or revise curriculum and instruction to allow for better integration of curriculum, instruction and assessment.

Most SAUs have single scored the assessments administered this year, and were just beginning to double score some of these assessments this past spring. Interview data indicate that many SAUs made the scoring of assessments in English Language Arts and Mathematics in grades 1, 5 and 9 a priority. Most SAUs have provided training for teachers on developing and scoring assessments but more work needs to be done.

Continued progress is also indicated by a comparison of the results of the LASIS surveys conducted in 2004 and 2005 on four tasks related to assessment in English Language Arts and Mathematics (Tables 37-42). In English Language Arts, progress has been made in all grade spans in developing curriculum documents aligned with the Learning Results, aligning

instruction and curriculum with the Learning Results, and providing professional development in common scoring for teachers. Progress has also been made in selecting performance indicators for assessment in the K - 4 grade span, while less progress was made on this task in the 5 - 8 and 9 - 12 grade spans.

In Mathematics, progress has been made in all grade spans in developing curriculum documents aligned with the Learning Results, aligning instruction and curriculum with the Learning Results, and providing professional development in common scoring for teachers. Less progress was made in selecting performance indicators for assessment in all grade spans.

Analysis of the 2005 survey data indicates that the level of progress made by SAUs could not be predicted by any of the following variables: superintendent region, the job role responsible for LAS coordination, attending enrollment, percent of students receiving free-and-reduced lunch, per pupil expenditures, and student/teacher ratios (Tables 31-34).

Analysis of interview data from the sample of 14 SAUs indicates that good progress was made in six SAUs, while 6 SAUs showed some progress and 2 showed limited progress. Those SAUs

(10)

showing good progress can be characterized as having strong, stable administrative leadership for curriculum and assessment work, strong levels of agreement with the concept of LAS, and support for LAS work expressed by both administrators and teachers. These SAUs had also scheduled more time to do LAS work than SAUs that had made limited progress.

Reviewing all sources of data, it appears that progress has been made on the tasks required to build and implement LASs. A considerable amount of revising curriculum, instruction, and assessment is occurring and adjustments are being made as LASs are being implemented. Most educators have indicated a strong need to continue the development of their expertise and capacity to do this work. However, SAUs are finding that this work is requiring more time to complete than expected because the changes needed to implement LASs impact a wide spectrum of constituents, belief systems, and expectations which have historical roots, and instructional practices that are firmly established.

Respondents to both surveys and interviews were asked to describe boosters and barriers that have facilitated or impeded the process of implementing a LAS. The following categories of responses were apparent in these data. A more detailed list of boosters and barriers can be found in Appendix F.

Boosters:

• Leadership structures for LAS work that were previously created for curriculum work • A specified coordinator for LAS work (curriculum coordinator or other person) • MDOE workshops and MDOE documents

• Additional time for LAS work by restructuring school day or calendar • Teacher stipends for non contractual time and summer work

• Collaboration and partnerships that result in sharing ideas and assessment models • Dedicated, professional staff who are open and willing to learn

• New data management tools and processes

• Support, including consultation from MDOE, financial and moral support from local

administrators and school committees and support from consulting organizations

• Professional development

Barriers:

• Insufficient time (in particular, to score assessments and reflect on results) • Incomplete work on curriculum alignment

• Poor communication among multiple schools within an SAU, parents, and community. • Late and inconsistent information from MDOE

• Size and scope of LAS requirements and tasks (particularly for small school districts) • Lack of compensation or release time needed to accomplish tasks

• Reluctance to give up instructional units that were deemed to be successful • Lack of adequate data systems for reporting and analyzing data

• Lack of available assessments that are reliable and valid

• Lack of direction for assessing students in special education, career and technical

(11)

2. What is the current status and technical quality of SAUs’ efforts to select and score assessments in individual content areas and grade spans?

Most SAUs were just beginning to double score assessments in the spring of 2005, and therefore it was too soon to determine the technical quality of the scoring process. Survey data indicate that 28% of SAUs reported that work was complete for double scoring common assessments in English Language Arts in grade span K - 4, 25% in grade span 5 - 8, and 33% in grade span 9 - 12 (Tables 8-10). In Mathematics, 37% reported that work was complete for double scoring common assessments in grade span K - 4, 34% in grade span 5 - 8, and 39% in grade span 9 - 12 (Tables 11-13). These data indicate that more SAUs had completed double scoring in

Mathematics in all grade spans than had completed double scoring in English Language Arts. They also indicate that a higher percentage of SAUs reported work complete on double scoring in grade spans 9 - 12 in both English Language Arts and Mathematics than in grade spans 5 - 8 and K - 4. The smallest percent of SAUs reported work complete in double scoring in both English Language Arts and Mathematics for grade span 5 - 8.

Survey data indicate that 28% of SAUs reported that work was complete on the task of

establishing and documenting reliability among scorers in English Language Arts in grade span K - 4; 25% in grade span 5 - 8; and 33% in grade span 9 - 12 (Tables 8-10). In Mathematics, 28% of SAUs reported work complete on this task in grade span K - 4; 23% in grade span 5 - 8, and 33% in grade span 9 - 12 (Tables 11-13). These data indicate that the highest percent of SAUs reported work complete in the 9 - 12 grade span and smallest percentage in grade span 5 - 8, a pattern similar to the results reported above for the task of double scoring common

assessments.

Interviews in the 14 SAUs revealed that teachers often described the processes of reaching consensus over benchmarks and scoring of students’ work as producing conflict and stress within work groups. These elements sometimes slowed progress on LAS tasks. Some SAUs assigned administrators or other educators to facilitate work groups to overcome the divisiveness and to keep these groups moving forward.

Respondents indicated that MDOE’s LAS documents were used to guide the technical aspects of the scoring process; however, some departed from the suggested protocols to some degree. Both the survey and interview data confirm that most SAUs believe that the technical aspects of double scoring assessments and establishing reliability are within their capability and are not tasks for which a high level of technical assistance is needed. Instead, time to score assessments and skilled group leaders for some work groups were cited as important needs for completing this task.

3. What progress has been made, what best practices exist, and what issues need to be resolved in SAUs’ efforts to develop LASs that will assess the learning of students in programs in Alternative Education, Special Education, English Language Learners (ELL), and Career and Technical Education (CTE)?

(12)

The overall focus on assessment and accountability has brought to light significant issues related to specifying common standards for all students. Educators have become acutely aware of the challenge of aligning the goals and philosophies of these educational programs and Maine’s

Learning Results.

Survey data reveal that only 21% of SAUs indicated that work was complete on creating

assessments for students with 504 plans. Nineteen percent indicated that work was complete for students requiring accommodations, and 25% of the SAUs reported that work was complete for creating alternative assessments (Table 24). Although 54% of SAUs reported that no action had been taken on creating assessments for students in programs for English Language Learners, this may be a misleading result. It cannot be determined if SAUs responding to this item had ELL programs for their students. Regarding CTE programs, 46% of the SAUs indicated that no action had been taken yet on developing LASs to assess students in this group, and 40% reported that no action had been taken yet related to alternative education programs (Table 25).

Interview data confirm that little work has been completed in developing assessments for

students in alternative education, special education, ELL, and CTE programs. Many educators in regular education programs reported that the design and implementation of LASs has increased their focus on educational outcomes and have generally increased the expectations they hold for students with disabilities. However, many educators in special education, alternative education, CTE, and ELL programs expressed skepticism about the appropriateness of the Learning Results

standards for their respective students. Educators in alternative education, special education, and ELL programs said the Learning Results grade level performance indicators and LAS

assessments were often too challenging for their students. CTE representatives said that very few of the Learning Results performance indicators apply to the content of their programs, which are organized around national standards for trade organizations.

Best practices related to LAS development and implementation are not apparent at this time, as educators are struggling with prerequisite issues such as curriculum alignment and aspects of LAS assessments, scoring and reporting. However, researchers generally found that where CTE and alternative education programs were co-located in or adjacent to a regular education facility, more collaboration and coordination occurred between regular education programs and special programs. Although administrators in some SAUs made the involvement of educators in special programs a priority, most educators in these programs felt they had not been included at both state and local levels in the design and development of LASs.

Alternative Education. In both a focus group and in on-site interviews, alternative education representatives described the wide range of needs of students in their programs. Some students required minimal support from alternative education (e.g., an altered school day schedule or altered program of study) while other students attended an alternative education program on a full-time basis that may be located at some distance from the regular school. Respondents described the difficulty of collaborating with regular education teachers and programs about the unique needs of their students and participating in the work of designing LAS assessments appropriate for their students given the small size of most alternative education programs.

(13)

Survey data indicate that among K - 12 and 9 - 12 SAUs, 65% indicated that help in building the capacity to assess students in alternative education programs was among their highest priority needs for assistance (Tables 26 and 27).

In alternative education, unresolved issues related to LAS development and implementation included the following:

• the incongruity of holding alternative education students accountable to standards and

assessments developed within and for regular education programs

• the requirement to administer assessments that are beyond students’ capabilities, forcing

repeated failure before students can take replacement assessments

• remediation and replacement assessments have not yet been developed and/or do not meet

the needs of students in alternative education

• ambiguity about what high school diploma options will be available to alternative

education students who cannot meet Learning Results standards as measured by a LAS Special Education. Assessment and accountability in special education have been part of the educational context for many years. Having specific learning objectives and specifying the means to achieve and measure those objectives have been part of students’ IEPs. The LAS requirements have added a complicating layer, which has had both positive and negative impacts. On the positive side, some special educators reported that the Learning Results have provided an organizational structure and new learning goals for students with disabilities.

However, common assessments did not take into account students’ unique learning needs and the difficulty of assessing students’ progress.

In special education, unresolved issues include all of the points listed above for alternative education. In addition, special educators reflected confusion about the role of the IEP and whether below-grade-level assessments could be administered to special education students. On the survey, 20% of the Advanced SAUs said that help developing alternate assessments (PAAPs) was one of the top five priority needs (Table 36).

English Language Learners (ELL). In most SAUs in which interviews were conducted, issues related to the assessment of ELL students were not viewed as a priority because these SAUs had few or no students in this subgroup, and because most were engaged in more fundamental tasks related to developing their LASs. For larger SAUs with sizable ELL populations, and for SAUs that were in a more advanced stage of LAS development and implementation, issues related to assessing ELL students were identified as a priority area. On the survey, 25% of the Advanced SAUs indicated that their need for assistance in developing the capacity to assess ELL students was among the top five areas of help needed (Table 35).

ELL teachers and coordinators reported that ELL students had great difficulty taking common assessments not only because of their limited English skills, but because many of them are also illiterate in their native languages. ELL students’ academic knowledge and skills were often significantly below grade-level expectations due to the lack of formal schooling experience in their native countries.

(14)

Career and Technical Education (CTE). The central issue related to the assessment of CTE students centers on the lack of congruence between the Learning Results and CTE programs that are guided by trade and industry standards. CTE representatives describe their programs as attempting to focus on the Learning Results but finding little overlap with national trade and industry standards. CTE representatives also expressed the concern that CTE students may become frustrated when their LAS assessments do not take into consideration their different learning styles, which tend to be more hands on and applied. CTE representatives suggested that the LAS model might prompt some CTE students to drop out of high school.

The LAS requirements have created some unique problems for CTE programs. With few exceptions, CTE programs are located at some distance from regular high schools and serve multiple SAUs, each with their own LAS. Different school calendars within a CTE region are a barrier to coordination of LAS assessment schedules in some SAUs. Respondents also indicated that SAUs rarely count assessments of students’ work in CTE courses as evidence of

achievement that is valid in LASs. Building capacity to assess CTE students was among the top five priority needs for help of SAUs that were Advanced in LAS work (Table 35).

4. How is LAS data being collected, managed, reported and communicated?

On the survey, building capacity for developing systems to analyze LAS data and developing systems to record LAS data were identified as being among the top priority needs of SAUs. Interviews in the 14 SAUs indicated that although many SAUs had put new software systems in place, many were transitioning to these systems, upgrading their systems, and in the early stages of training teachers and entering assessment scores. A few SAUs had not yet selected a software system and were waiting to determine what systems might be compatible with the Maine

Educational Data Management System (MEDMS). Even in SAUs that were implementing new data management systems, it was common for administrators to collect and report assessment data using simple spreadsheets. Analysis of these data for any purpose is limited.

Respondents in the 14 SAUs reported that assessment scores were typically translated into letter grades for the purpose of reporting student achievement on report cards. Although some SAUs are beginning to introduce standards-based reporting in some grades, most student achievement was reported using traditional report cards.

On the survey, 39% of the SAUs indicated that work on the task of developing a report card describing progress toward meeting content standards at the K - 4 level was either partially complete or complete. The same degrees of work completion were indicated by 32% of SAUs at the 5 - 8 level, and 24% at the 9 - 12 level (Tables 21-23).

Interview data indicate that administrators and teachers were providing some general information to parents about LAS requirements through parent meetings and newsletters. Parent meetings were described as infrequent and often not well attended. More effort had been made to meet with parents of eighth- and ninth-grade students. Administrators indicated that they were somewhat apprehensive about communicating with parents about assessment and graduation guidelines because LAS requirements had been frequently revised, and they were concerned that additional revisions were possible.

(15)

On the survey, the percentage of SAUs that indicated work complete or partially complete on the task of Developed a procedure for public reporting of scores ranged from 42% to 50% across the grade spans, with the highest percentage in the 9 - 12 grade span (Tables 21, 22, and 23). Only 22% of SAUs indicated that LAS data is used to inform the community about students’ progress

(Table 20). It appears that SAUs were in the early stages of developing ways to report assessment results, but the data were not yet ready for reporting.

Several administrators and teachers suggested that a more intense information campaign related to standards-based education be mounted at the state level to increase public awareness about state educational standards and to support local efforts to measure and report students’

achievement of the standards. As evidence of this need, on a 4-point scale ranging from Do not understand at all to Completely understand, only28% of SAUs indicated parents’ level of understandingto be in the top half of this scale. On the same item, 54% of SAUs indicated that freshmen had the same level of understanding (Table 30).

5. What policies relevant to LASs have been created or modified by SAUs?

Survey results indicate that 40% of Less Advanced SAUs perceived that help in developing policies related to assessment was among the top five areas of assistance needed (Table 35). On the task of developing a policy for replacement assessments, 45% of SAUs reported work was

partially complete or complete in the K - 4 and 5 - 8 grade spans, while 55% of SAUs reported the same level of completed work in the 9 - 12 grade span (Tables 21-23). This finding is consistent with interview data, which indicate that most SAUs were in the early stages of planning and drafting policies for remediation and replacement assessments. Many SAUs were waiting for policy models from the Maine School Management Association at the time of the site visits.

6. How are SAUs responding to the systemic use of LAS data for formative assessment of curriculum and instruction?

Survey data indicate that in 83% of SAUs teachers were using LAS data in an informal way to make decisions about curriculum and instruction. This finding is consistent with interview data. Somewhat less frequent was the use of LAS data to identify students’ needs for remediation (60%), the revision of curriculum (59%), the identification of curriculum strengths and areas of need (51%), and to inform priorities for professional development (48%) (Table 20). The lack of data management systems and lack of time to score all assessments and enter them into new data software prevented educators from using data in more formal ways. Educators interviewed in the 14 SAUs also reported that the lack of time to reflect on assessment results was a significant barrier to using data to inform curriculum and instruction.

7. What progress are SAUs making toward certification of student achievement of the Learning Results and how is LAS data informing this progress?

Respondents were asked how confident they were that their SAU would have assessments in place to certify that graduates of 2008 would meet the Learning Results in English Language

(16)

Arts and Mathematics. In addition, they were asked what percent of their graduates they estimated would meet the Learning Results standards in 2008.

Survey results indicate that on a scale of 1, Not at all confident, to 4, Very confident, 75% of SAUs indicated confidence levels of 3 (32%) and 4 (43%) that assessments will be in place to certify that 2008 graduates will meet the Learning Results in 2008 in English Language Arts. Seventy-eight percent of SAUs indicated a confidence level of 3 (38%) and 4 (40%) in

Mathematics (Table 28).

When asked to project what percent of 2008 graduates would achieve the Learning Results in English Language Arts, 79% of SAUs reported that between 51% and 100% of their graduates would meet this criterion. Sixty-nine percent of SAUs indicated that between 51% and 100% of graduates would achieve the Learning Results in Mathematics (Table 29).

These two questions required that respondents select answers based on their best guess about future developments in the implementation of LASs in Maine. The reader is cautioned in the use and interpretation of these data. During the time period in which the survey was being

conducted discussions about the future of LASs were occurring among MDOE personnel and the Education Committee of the Legislature. Responses to these questions, which required

speculation, may also have been influenced by reports of these discussions in the media.

8. What technical assistance or other resources are needed?

Survey results indicate that K - 5 and K - 8 grade spans were most in need of assistance in building capacity to develop systems for analyzing LAS data (77%), to use data to inform curriculum sequence (70%), to use data to inform curriculum content (68%), and to develop systems for recording LAS data (68%). K - 12 and 9 - 12 SAUs indicated that their highest priority needs for assistance include help building capacity to assess CTE students (70%), to develop systems to analyze LAS data (69%), to assess students in alternative education programs (65%), and to assess students in programs for English Language Learners (62%) (Tables 26 and 27).

Data from interviews and focus groups highlighted other areas of need that were more global. Time, staff, and data management resources were nearly universal needs. Financial resources were identified as a need in some SAUs. Many other comments by respondents can be characterized as a need for knowledgeable leadership. These respondents were frequently critical of SAUs decisions to send a small number of representatives to MDOE workshops about LAS implementation who were then expected to return to the SAU and train others. These respondents indicated that the designated trainers could not answer questions at more than a surface level and/or they were spread too thinly throughout the SAU.

Respondents also commented that, although MDOE workshops were informative, they often did not match the needs of SAUs that were at different levels of LAS development and

implementation. Personnel from more Advanced SAUs often attended MODE workshops that appeared to be appropriate for their needs only to find that the level of presentation and

(17)

A need for leadership toward an integrated curriculum also emerged from the interviews. Some respondents indicated that when curriculum was integrated, assessment tasks were more aligned with curriculum and could serve multiple content areas. While this condition existed in some SAUs, many respondents in other SAUs viewed it as a goal that required leadership that was not available.

Finally, many respondents voiced a need for clear guidance from MDOE personnel and LAS documents on how to accomplish LAS tasks. Educators working with special education students, alternative education students and English Language Learners had specific questions about assessing their students and expressed the need to have MDOE personnel visit their schools.

9. What factors, such as leadership structures and professional development opportunities have been helpful in creating and implementing LASs?

It is clear from interview data that SAUs that assigned LAS leadership and coordination

responsibilities to teams that already existed for curriculum alignment with the Learning Results

were advanced in LAS work. More Advanced districts viewed LAS work as an extension of earlier work on the Learning Results. Less Advanced districts were those that started more slowly, had changes in leadership, or formed totally new committees or teams to do LAS work. Interview data indicate that superintendents played an important role in more Advanced SAUs through supporting the concept of LASs, providing resources, stimulating policy development and communicating with the public. However, curriculum coordinators and building principals were more significant leaders in getting the work done.

Professional development opportunities were frequently listed as a booster to progress and one of the positive impacts of LAS work. LAS work was attributed with encouraging high-quality focused professional development with specific outcomes or products. For example, one superintendent, while concerned about the resources demanded by LAS implementation, indicated that LAS had caused “the best professional development my teachers have had in years.” On the negative side, topics related to LAS development were viewed by some respondents as excluding other important topics related to instruction.

Further analyses of these data reveal no clear difference between Advanced and Less Advanced SAUs in the job role identified as having primary responsibility for LAS work (Table 32).

10. How have SAUs used time and other resources to meet both the state and local requirements of implementing LASs?

SAUs varied widely in the amount of time dedicated to LAS work including professional development. In SAUs in which interviews were conducted the total number of days scheduled for LAS work varied from .5 to 7 full days. Of these, the number of days scheduled for scoring assessments varied from 0 to 4 days. These figures do not include the personal time many teachers devoted to LAS tasks. Overall, SAUs making the most progress used professional development days and early release days for LAS development and scoring. Some SAUs hired

(18)

substitute teachers to allow content or grade-level teams to meet for full days. Many respondents reported that using brief periods of time for LAS work, such as after school hours, and before and after school meetings, was a frustrating and unproductive approach.

Conclusions

Evidence of Progress Made

Survey results indicate that progress is being made on the tasks of developing and implementing LASs. Throughout 2004-2005 most SAUs were refining their assessment tasks and schedules after

administering some assessment tasks and discovering that the assessment plans created earlier were in need of adjustment. Few SAUs were involved in refining the scoring process, reporting data, or using data to inform curriculum and instruction. More SAUs reported a higher level of work completion in English Language Arts and Mathematics. The content areas of Science and Technology, Social Studies, and Health and Physical Education were rated as partially complete or work complete by more than half of the SAUs responding to the survey with the largest percent in grade span 9-12. Few SAUs reported work to be partially complete or complete in the areas of Modern and Classical Languages, Visual and Performing Arts and Career Preparation.

Mixed Support for LAS

Support for the concepts of standards based education, accountability, and high expectations for students emerges from the interviews of teachers and administrators in regular education though most expressed concerns about the timelines for implementing LASs and the scope of the overall changes required. Many respondents indicated that LAS work has provided highly valuable professional development experiences and stimulated collaborative planning among teachers. As a result, curriculum has been more clearly defined in each grade span and there is more consistency, coherence, and common practices in curriculum and assessment across grade spans and throughout K-12. This is not true of educators who are responsible for programs in special education, career and technical education, alternative education, or programs for English Language Learners. The issues raised by these special groups are related to the unique characteristics of the students they serve, the concept of holding the same standards for all students regardless of their capabilities, the means of assessing achievement through LAS, and the criteria for receiving a high school diploma. The data suggest that these issues should be examined if the Learning Results and the LAS model are to apply to all students.

Interview data also suggest that some teachers are resistant to LAS implementation because it may require that they eliminate, modify or reschedule units of instruction that have been developed and refined, sometimes over many years.

Time as a Barrier

The primary impediment to the development and implementation of LASs in most SAUs is the lack of time to do the work, particularly the scoring of assessments, processing the scores in data management systems and analyzing and reflecting on the results. Large blocks of time (full days or 2 days together) are viewed as most productive. More successful SAUs have found additional time by hiring substitutes,

(19)

scheduling common planning periods for teachers, using early release days, and paying stipends to teachers for the additional workload inherent in LAS planning and scoring assessments. Not all SAUs had the financial flexibility or flexibility with the school calendar to use these strategies.

Financial Costs

There are significant concerns about the cost of developing, implementing, and sustaining LASs among SAU administrators. Frequently mentioned costs include the creation of additional time for LAS work, stipends for teachers and teacher leaders, providing remedial services and developing and managing data systems. Methods of funding these activities within the Essential Programs and Services model are of particular interest.

Human Capacity

The need to develop the human capacity to fully implement LASs emerges from the qualitative data in several different forms. Interview data suggest that the training model used by many SAUs in which a very small number of representatives attend MDOE workshops and then return to train other SAU personnel is minimally effective. Teachers report that their representatives often find themselves inadequately prepared to respond to substantive questions upon returning to their schools. Alternative ways of developing this capacity in SAUs should be explored.

Survey and interview data indicate that one knowledgeable person, typically a Curriculum Coordinator, is critical to successful LAS work in most SAUs. Some SAUs may need financial support or funding to create or support this position.

Professional development is needed to build SAUs capacity to support strategies such as interdisciplinary approaches and the seamless integration of instruction and assessment. Managing Social Change

Interview data confirm that the role of teacher is changing. Teachers are being asked to change

curriculum, instruction and assessment practices and work in a collaborative mode with other teachers. Leadership personnel with group facilitation skills and time are needed to support major changes that are required in both beliefs and behaviors.

Communications and Public Relations

Interview and survey data indicate that there is a need for increased communication about LAS, among and between schools and between SAUs and MDOE.

SAUs are in need of public support from beyond their boundaries as they develop LASs. Interview data indicate that some school leadership personnel believe that increased publicity about standards-based education and performance-based assessment sponsored by MDOE would assist SAUs in

(20)

LAS goals and requirements should be communicated effectively and consistently by MDOE. This is especially needed for the assessment of students in special education, alternative education, career and technical education programs and programs for English Language Learners.

Technical Assistance

Both survey and qualitative data indicate a need to see models of LASs that have been fully

implemented and used to inform curriculum and instruction. Components of particular interest include data management systems, replacement assessments, alternative assessments, policies, and remediation programs. Assistance is also needed in K-12 and 9-12 SAUs in building capacity to assess CTE

students, students in Alternative Education programs and for English Language Learners. Technology can assist SAUs in recording, managing, analyzing, and using LAS data. However, few SAUs have devised or purchased the perfect system and many are waiting for MEDMS to be functional before they invest in technology for assessment. Leadership from the MDOE is needed to help school districts make choices that will meet their LAS data needs.

Readiness to Certify Students’ Achievement of the Learning Results

Thirty percent of the SAUs responding to the survey indicate that they anticipate that less than 50% of their students will meet the Learning Results in 2008. Although this data is based on respondents’ speculation, it may suggest the need to examine the opportunities that students have to gain the

knowledge and skills encompassed by the Learning Results and to identify and explore barriers that may impede learning.

Professional Development

Survey and interview data indicate that the development and implementation of LASs have been a very positive professional development experience for teachers. Both leadership personnel and teachers described that more time was being devoted to discussions of students, curriculum and assessment and that LAS work was requiring more professional discussion among teachers. Both teachers and

administrators reported that their knowledge of basic measurement concepts such as validity and reliability, and of assessment concepts and practices such as rubrics, benchmarks, and double scoring had increased.

Professional development topics related to LASs were valued by both teachers and administrators. However, many expressed the concern that other important topics related to instruction were not being considered due to LAS deadlines.

Expectations for Student Achievement

Some teachers and administrators commented that the process of developing and implementing a LAS had raised teachers’ expectations for students performance and that students were meeting those expectations.

(21)

Use of LAS Data

Survey and interview data confirm that LAS data is being used informally by teachers in discussions about students and curriculum. Few SAUs have been able to use data in more formal ways (e.g. identifying implications of assessment data for systemic changes in curriculum and instruction) due to lack of time for scoring, and lack of technology for recording, analyzing and reporting LAS data.

(22)

Part II: The Status of LAS Development and Implementation: A Statewide Survey, Spring 2005

A survey was conducted to assess the status of LAS implementation in May 2005. In order to gain information about the progress made by all SAUs in developing and implementing LASs, some items that had been used in a similar survey in 2004 were included. The 2005 survey may be found in Appendix A.

The survey was designed to obtain information from the entire population of Maine SAUs and, therefore, no sampling strategy was used. However, it was necessary to determine the maximum number of LASs that might be created in the State since a LAS might be designed and

implemented for more than one SAU. This information was gained through e-mail and telephone dialog with representatives of school unions, CSDs, and other SAUs that share a superintendent. As a result of these communications, it was determined that the total population of 225 SAUs would generate a maximum of 159 LASs.

Demographics

The survey was mailed to all 225 SAUs in Maine in early May of 2005 with a request that it be returned by May 20. The tables below represent the responses of 187 SAUs representing a return rate of 83%.

One hundred and thirty-five different LASs are represented in the returned surveys (some LASs are implemented in more than one SAU). Based on the assumption that a maximum of 159 LASs are expected to be designed and implemented in Maine, 85% of all LASs are represented in the data.

Table 1indicates the number of SAUs that returned surveys from each region and the percent of the total number of SAUs, in each region that responded. Since some LASs might apply to more than one SAU these data might not represent precisely the number and percent of LASs actually being implemented.

Table 1. Percent of SAUs Responding in Each Region

16 66.7% 19 95.0% 29 93.5% 29 93.5% 15 55.6% 32 96.9% 6 100.0% 17 81.0% 12 52.1% 12 75.0% Aroostook County Cumberland County Hancock County Kennebec Valley Midcoast Penquis Unorganized Territories Washington Western Maine York n %

(23)

Table 1 indicates that all regions were well represented in the data. The Western Maine and Midcoast regions were least represented at 52.1% and 55.6% response rates. The highest response rates were from the Penquis region (96.9%), Cumberland County (95%), Hancock County (93.5%), and Kennebec Valley (93.5%).

Table 2 indicates the number and percent of school administrative structures represented in the data, and Table 3 displays the number of individuals who completed or assisted in completing the survey. As indicated by the data in Table 3, more than one person participated in responding to the survey in many SAUs.

Table 2. Administrative Structures, Sample and State

Table 3. Roles of Individuals Completing the Survey

Data were collected regarding the grade configuration and enrollment of each of the responding SAUs. Tables 4 and 5 present these data. Table 5 provides a breakdown of responding SAUs by grade configuration and enrollment category.

35 18.7% 100 53.5% 3 1.6% 0 .0% 28 15.0% 31 16.6% 7 3.7% 13 7.0% Superintendent Curriculum Coordinator Assistant Principal Headmaster Assistant Superintendent Principal Teacher Leader Other job role

n % 68 36.4 87 37.8 38 20.3 45 19.6 58 31.0 71 30.9 10 5.3 14 6.1 6 3.2 6 2.6 7 3.7 7 3.0 Union

Town with Individual Supervision MSAD CSD Unorganized Territory* Other Sample Sample Percent Districts with Schools

in State State Percent

(24)

Table 4. Grade Configuration, Sample, and State

Table 5. Enrollment Category, Sample and State

The data in Tables 4 and 5 indicate that over 90% of the SAUs that responded to the survey were K - 12 and K - 8 SAUs with approximately a third of all responding SAUs enrolling fewer than 200 students and slightly more than a third enrolling more than 851 students.

Leadership of LAS Implementation and Design

Respondents were asked to indicate which job roles have the primary responsibility for leading the process of developing a LAS in their SAU. Respondents were asked to limit their selection to two role titles. Table 6 displays these data.

6 3.2 6 2.6 91 48.7 111 48.3 4 2.1 4 1.7 86 46.0 109 47.4 6-12 or 9-12 K-12 K-5 K-8 Sample Sample Percent Districts with Schools

in State State Percent

67 35.8 82 35.7 25 13.4 31 13.5 26 13.9 39 17.0 35 18.7 42 18.3 34 18.2 36 15.7 Fewer than 200 201 - 400 401 - 850 851 - 2000 More than 2000 Sample Sample Percent Districts with Schools

(25)

Table 6. Job Roles with Primary Responsibility of Leading LAS Development

Table 6 indicates that that the role of Curriculum Coordinator was most frequently indicated as a role with leadership responsibility for designing the LAS followed by principals, teacher leaders, and superintendents. These data were further analyzed by categories of district size (attending enrollment), and results are displayed in Table 7.

Table 7. Job Roles with Primary Responsibility of Leading LAS Development by SAU Size

155 82.9% 32 17.1% 84 44.9% 103 55.1% 186 99.5% 1 .5% 187 100.0% 159 85.0% 28 15.0% 142 75.9% 45 24.1% 154 82.4% 33 17.6% 156 83.4% 31 16.6% Superintendent leading development of LAS Curriculum Coordinator leading development of LAS Assistant Principal leading development of LAS Headmaster leading development of LAS Assistant Superintendent leading development of LAS Principal leading

development of LAS Teacher Leader leading development of LAS Other job role leading development of LAS n % Not checked n % Checked 13 19.4 3 12.0 4 15.4 8 22.9 4 11.8 34 50.7 10 40.0 16 61.5 19 54.3 24 70.6 1 4.0 8 11.9 5 20.0 2 7.7 3 8.6 10 29.4 20 29.9 5 20.0 4 15.4 14 40.0 2 5.9 11 16.4 3 12.0 8 30.8 6 17.1 5 14.7 9 13.4 4 16.0 6 23.1 5 14.3 7 20.6

Superintendent leading development of LAS

Curriculum Coordinator leading development of LAS

Assistant Principal leading development of LAS

Headmaster leading development of LAS Assistant Superintendent leading development of LAS

Principal leading development of LAS Teacher Leader leading development of LAS

Other job role leading development of LAS n % Fewer than 200 n % 201 - 400 n % 401 - 850 n % 851 - 2000 n % More than 2000

(26)

Curriculum Coordinator was the role most frequently selected in all size categories. Principals were frequently indicated as leaders in the larger 851-2,000 enrollment category of SAUs while teacher leaders were selected in the 401-850 category.

Current Status of LAS Development and Implementation

Respondents were asked to rate the progress of LAS work on nine different activities related to designing and implementing a LAS in two content areas, English Language Arts and

Mathematics. A scale of 1 to 4 was employed where 1 signified No action on this yet, 2 indicated Planning in progress, 3 signified Partially complete,and 4 indicated Work complete.

Although respondents were asked to rate “progress,” their ratings on the scale resulted in a representation of the current status of their work. Progress in LAS development and

implementation is described in Tables 37-42 where data from 2004 and 2005 are compared. English Language Arts. Data representing the status of tasks related to LAS development and implementation in English Language Arts for each grade span is presented in Tables 8, 9, and 10 below.

Table 8. Status of LAS Tasks in English/Language Arts K - 4

1 .5% 10 5.4% 86 46.7% 87 47.3% 1 .5% 10 5.5% 116 63.4% 56 30.6% 7 3.8% 7 3.8% 58 31.2% 114 61.3% 4 2.2% 4 2.2% 59 31.9% 118 63.8% 5 2.7% 6 3.3% 62 33.7% 111 60.3% 5 2.7% 16 8.6% 61 33.0% 103 55.7% 11 5.9% 15 8.1% 83 44.9% 76 41.1% 11 6.0% 25 13.6% 96 52.2% 52 28.3% 32 17.3% 62 33.5% 65 35.1% 26 14.1% Developed curriculum documents aligned with

MLR

Aligned instruction with curriculum and MLR Selected a sample of performance indicators for assessment

Used scoring rubrics for each assessment Administered common assessments

Provided professional development in common scoring for teachers

Double scored common assessments

Established and documented reliability among scorers

Used LAS data to inform the design of curriculum and instruction n % No action taken on this yet n % Planning in progress n % Partially complete n % Work complete

(27)

Table 9. Status of LAS Tasks in English/Language Arts 5 - 8

Table 10. Status of LAS Tasks in English/Language Arts 9 - 12

The data in Tables 8, 9, and 10 indicate that the work of developing curriculum documents for English Language Arts that are aligned with the Learning Results was complete in nearly 50% of responding SAUs in all grade spans. However, the alignment of instruction with the Learning Results was less complete in grade spans K - 4 (30.6%) and 5 - 8 (30.1%), and more complete in the 9 - 12 grade span (42.0%). It appears that although curriculum has been aligned in document format in nearly 50% of SAUs, changing instructional practices has proceeded at a slower pace. More than 50% of the respondents in all grade levels reported that work was complete in the selection of assessments, use of scoring rubrics, administration of assessments and providing

1 .5% 14 7.6% 85 46.2% 84 45.7% 1 .5% 17 9.3% 110 60.1% 55 30.1% 7 3.8% 8 4.3% 60 32.4% 110 59.5% 3 1.6% 4 2.2% 64 34.4% 115 61.8% 4 2.2% 8 4.4% 68 37.2% 103 56.3% 5 2.7% 14 7.5% 67 36.0% 100 53.8% 12 6.5% 14 7.6% 88 47.8% 70 38.0% 10 5.4% 22 11.9% 106 57.3% 47 25.4% 33 18.0% 63 34.4% 69 37.7% 18 9.8% Developed curriculum documents aligned with MLR

Used LAS data to inform the design of curriculum and instruction n % No action taken on this yet n % Planning in progress n % Partially complete n % Work complete 0 .0% 3 2.6% 53 46.1% 59 51.3% 0 .0% 6 5.4% 59 52.7% 47 42.0% 5 4.3% 4 3.4% 23 19.8% 84 72.4% 0 .0% 2 1.7% 29 25.2% 84 73.0% 0 .0% 3 2.6% 43 37.4% 69 60.0% 2 1.7% 6 5.2% 42 36.5% 65 56.5% 2 1.7% 12 10.3% 54 46.6% 48 41.4% 5 4.3% 17 14.8% 55 47.8% 38 33.0% 25 21.2% 32 27.1% 45 38.1% 16 13.6% Developed curriculum documents aligned with MLR

Established and documented reliability among scorers Used LAS data to inform the design of curriculum and instruction n % No action taken on this yet n % Planning in progress n % Partially complete n % Work complete

(28)

professional development for teachers in common scoring. A higher percentage (60% to 72%) of SAUs reported that work was complete on this task in the 9 - 12 grade span. Use of LAS data to inform the design of curriculum and instruction was most frequently rated as Planning in progress or No action taken yet by more than 50% of all SAUs.

Mathematics. Data representing the status of tasks related to LAS development and implementation in Mathematics for each grade span is presented in Tables 11, 12, and 13. Table 11. Status of LAS Tasks in Mathematics K - 4

Table 12. Status of LAS Tasks in Mathematics 5 - 8

2 1.1% 7 3.8% 78 42.4% 97 52.7% 1 .5% 13 7.1% 101 54.9% 69 37.5% 7 3.8% 9 4.9% 59 32.2% 108 59.0% 4 2.2% 6 3.3% 52 28.3% 122 66.3% 4 2.2% 12 6.6% 60 33.0% 106 58.2% 8 4.4% 18 9.9% 60 33.1% 95 52.5% 18 9.9% 20 11.0% 77 42.3% 67 36.8% 13 7.1% 26 14.2% 93 50.8% 51 27.9% 32 17.6% 59 32.4% 59 32.4% 32 17.6% Developed curriculum documents aligned with MLR

Established and documented reliability among scorers Used LAS data to inform the design of curriculum and instruction n % No action taken on this yet n % Planning in progress n % Partially complete n % Work complete 2 1.1% 7 3.8% 77 41.8% 98 53.3% 1 .5% 16 8.7% 100 54.3% 67 36.4% 7 3.8% 8 4.4% 66 36.1% 102 55.7% 3 1.6% 5 2.7% 65 35.3% 111 60.3% 3 1.7% 11 6.1% 62 34.3% 105 58.0% 8 4.4% 17 9.4% 71 39.4% 84 46.7% 19 10.5% 16 8.8% 85 47.0% 61 33.7% 14 7.7% 22 12.1% 104 57.1% 42 23.1% 34 18.8% 58 32.0% 72 39.8% 17 9.4% Developed curriculum documents aligned with MLR

Aligned instruction with curriculum and MLR

Selected a sample of performance indicators for assessment Used scoring rubrics for each assessment

Administered common assessments

Established and documented reliability among scorers Used LAS data to inform the design of curriculum and instruction n % No action taken on this yet n % Planning in progress n % Partially complete n % Work complete

(29)

Table 13. Status of LAS Tasks in Mathematics 9 - 12

The data in Tables 11, 12, and 13 reflect a pattern of work completion for Mathematics similar to that of English Language Arts. In each grade span, a higher percentage of SAUs rated the

development of curriculum documents aligned with the Learning Results as Work complete than rated the alignment of instruction as Work complete. Again, it appears that the work of creating aligned documents has moved more quickly than the alignment of instruction. It may be that the alignment of documents was the work responsibility of a few, while aligning instruction with curriculum and Maine Learning Results required participation of many.

More than 50% of the respondents in all grade levels reported that the selection of assessments, the use of scoring rubrics, and administration of assessments as Work complete. A higher percentage (60% to 72%) of SAUs reported that work was complete on this task in the 9 - 12 grade span.

In all grade spans, the use of LAS data to inform the design of curriculum and instruction was indicated as Work complete by the smallest percentage of respondents indicating an area of high need for technical assistance in this or other prerequisite LAS tasks.

Overall Progress in Other Content Areas. English Language Arts and Mathematics were the content areas that were the primary focus of the 2005 survey. However, the survey also

requested that respondents indicate the overall status of LAS development and implementation in the additional content areas of Science and Technology, Social Studies, Health and Physical Education, Modern and Classical Languages, Visual and Performing Arts, and Career

Preparation. Respondents’ overall status of work in these content areas is displayed in Tables 14, 15, and 16. 1 .9% 3 2.6% 39 33.9% 72 62.6% 0 .0% 5 4.3% 57 49.6% 53 46.1% 5 4.3% 2 1.7% 20 17.4% 88 76.5% 0 .0% 2 1.7% 27 23.5% 86 74.8% 0 .0% 2 1.8% 35 31.0% 76 67.3% 1 .9% 12 10.7% 37 33.0% 62 55.4% 4 3.5% 12 10.5% 54 47.4% 44 38.6% 7 6.1% 16 13.9% 54 47.0% 38 33.0% 22 19.5% 31 27.4% 46 40.7% 14 12.4% Developed curriculum documents aligned with

MLR

Used LAS data to inform the design of curriculum and instruction

n % No action taken on this yet n % Planning in progress n % Partially complete n % Work complete

(30)

Table 14. Overall Status of LAS Development and Implementation K - 4

Table 15. Overall Status of LAS Development and Implementation 5 - 8

Table 16. Overall Status of LAS Development and Implementation9 - 12

The data in Tables 14, 15, and 16, which pertain to content areas other than English Language Arts and Mathematics, indicate that the largest percentage of SAUs rated the status of their work as complete in the content areas of Science and Technology, Social Studies, and Health and Physical Education in all grade levels but especially in the 9 - 12 grade span. Modern and

Classical Languages, Visual and Performing Arts, and Career Preparation reflected a higher level of completion in grade spans 9 - 12 and 5 - 8 and a lower level of completion in grades K - 4.

5 2.7% 24 13.1% 103 56.3% 51 27.9% 13 7.1% 41 22.3% 91 49.5% 39 21.2% 5 2.7% 41 22.3% 91 49.5% 47 25.5% 102 56.4% 63 34.8% 16 8.8% 0 .0% 58 31.7% 89 48.6% 35 19.1% 1 .5% 116 63.4% 58 31.7% 9 4.9% 0 .0% Sci Tech Soc Studies Health/PE

Modern & Class Lang Visual Performing Arts Career Prep

n %

No action taken on this yet n % Planning in progress n % Partially complete n % Work complete 0 .0% 3 2.6% 62 53.9% 50 43.5% 0 .0% 4 3.5% 73 63.5% 38 33.0% 0 .0% 3 2.6% 65 56.5% 47 40.9% 36 31.9% 50 44.2% 23 20.4% 4 3.5% 25 21.9% 61 53.5% 28 24.6% 0 .0% 55 48.2% 37 32.5% 21 18.4% 1 .9% Sci Tech Soc Studies Health/PE

n %

No action taken on this yet n % Planning in progress n % Partially complete n % Work complete 2 1.1% 15 8.4% 108 60.3% 54 30.2% 7 3.9% 30 16.7% 96 53.3% 47 26.1% 2 1.1% 32 17.8% 95 52.8% 51 28.3% 83 46.6% 73 41.0% 21 11.8% 1 .6% 54 30.2% 91 50.8% 33 18.4% 1 .6% 104 58.1% 60 33.5% 15 8.4% 0 .0% Sci Tech Soc Studies Health/PE

n %

No action taken on this yet n % Planning in progress n % Partially complete n % Work complete

(31)

Number of Certification Assessments Planned

Respondents were asked to indicate the total number of certification assessments that were planned for each grade span in English Language Arts and Mathematics. If more than 12 assessments were planned for any grade span, respondents were asked to provide a rationale. Tables 17 and 18 indicate the number and percent of SAUs that planned fewer than 8, 8 to 12, and more than 12 assessments in English Language Arts and Mathematics respectively. Table 17. Number of Assessments in English Language Arts By Grade Span

Table 18. Number of Assessments in Mathematics by Grade Span

The majority of respondents indicated that they planned to administer 8 to 12 assessments in both English Language Arts and Mathematics, a range suggested by MDOE in the LAS Guide, Principles and Criteria for the Adoption of Local Assessment Systems. Respondents were also asked to provide a narrative rationale for planning more than 12 assessments. Although responses varied, most SAUs indicated that additional assessments were needed to fully assess all standards and ensure fairness and sufficiency. Some SAUs indicated they needed additional assessments for their early literacy initiatives. Other SAUs reported a decision to assess

performance indicators singly rather than multiply, which increased the total number of assessments. It was also common for SAUs to indicate that they are now in the process of refining their assessment plans to reduce the number of assessments given.

Distribution of Assessments by Type

Respondents were asked to indicate the type and number of certification assessment tasks planned for each grade span in five content areas. These data are represented in Table 19.

27 15.3% 19 10.8% 9 8.2% 126 71.6% 134 76.1% 93 84.5% 23 13.1% 23 13.1% 8 7.3% Fewer than 8 8 - 12 More than 12 Number of Assessments: English/Language Arts n % K - 4 n % 5 - 8 n % 9 - 12 24 14.5% 18 10.8% 7 6.4% 108 65.1% 126 75.9% 71 64.5% 34 20.5% 22 13.3% 32 29.1% Fewer than 8 8 - 12 More than 12 Number of Assessments: Math n % K - 4 n % 5 - 8 n % 9 - 12