ExCEL After School Programs Evaluation Plan

(1)

ExCEL After School Programs Evaluation Plan 2011-12

Section One: Vision for the 2011-12 Evaluation

In 2011-12, San Francisco ExCEL is trying to accomplish two goals simultaneously:

• First, ExCEL is striving for increased compliance and uniformity in the wake of several years of audit findings and compliance issues.

• At the same time, ExCEL seeks to preserve the unique aspects of its model, including the unique approach of its various CBO partners who oversee the day-to-day

operations of individual ExCEL sites.

Toward that end, ExCEL seeks to gather information to establish the nature and success of

its current programs in order to tell this year’s story of San Francisco ExCEL. This story needs

to be told both to the district, state and federal parties invested in program compliance, and to the CBO partners, parents of enrolled students, district staff and other community

stakeholders invested in program quality and a diversity of opportunities for San Francisco young people. In future years, this 2011-12 story will also serve as a starting point for the professional development and support the ExCEL office offers sites and will suggest evaluation questions ExCEL will pursue.

As an additional process goal, ExCEL seeks to build upon its relationship with its many CBO

partners by ensuring that information collected by the evaluation process is returned to sites

quickly. This transparency is key both in light of recent compliance concerns and as sound program practice; sites who receive their information back quickly will trust that ExCEL evaluation dollars are being put to good use and will be able to use these data to inform on-going program improvement.

San Francisco ExCEL has contracted with Public Profit, an Oakland-based firm with extensive experience in out-of-school time, youth development, program administration and working with large districts with multiple CBO partners, to develop and execute an evaluation plan in line with these goals.

(2)

Section Two: A Theory of Action for ExCEL Programs

We are mindful that the Public Profit team is entering a complex system with many established program quality expectations and reporting requirements. Specifically:

• ExCEL programs must answer to a variety of program implementation and outcome requirements. ExCEL programs must reach certain program performance targets such as number of kids served, average daily attendance, and total attendance (total kid-days) to comply with federal and state grant guidelines. They must also support San Francisco Unified School District’s (SFUSD) current strategic goals around access and equity among students, student achievement, and district accountability to students and families.

• ExCEL programs are also accountable to the community as a whole. For several years, ExCEL has been a leader in the citywide Afterschool for All Group (AFA), which seeks to create afterschool opportunities for all children and families who want them and has established standards for program quality. Finally, ExCEL uses the California After School Program Quality Self-Assessment Tool (QSA) to guide program reflection and program improvement. Both AFA’s standards and the QSA tool therefore establish standards for program quality for ExCEL programs.

Since this is a year to establish the nature and success of ExCEL’s current programs, Public Profit proposes that ExCEL use a research-based Theory of Action developed in Public Profit’s work with a variety of out-of-school time programs.

We hypothesize that students who participate regularly in high quality programs demonstrate the greatest improvements in key outcomes, including safety, school engagement, social skills development, and academic skills and academic behaviors.

Theory of Action for San Francisco ExCEL Programs 2011-12

Regular Participation Program Attendance

In High Quality Programs Safe

Supportive Interactive Engaging

Diverse, accessible, inclusive School-day aligned

Skill-Building Physically active

Has Direct Benefits for Youth

Improved safety New skills and experiences Strong relationships with peers and adults Improved academic behaviors School engagement Contributes to Other Positive Outcomes English fluency Academic performance Higher test scores

(3)

The above Theory of Action addresses many of these program implementation and outcomes requirements. It also will direct the writing of the 2011-12 story about the uniqueness and effectiveness of ExCEL and its many programs. The writing of this story, through data collection and reporting, is guided by a set of four evaluation questions:

1. What progress have ExCEL programs made toward target enrollment and daily attendance rates?

2. In what ways are ExCEL programs providing high quality services?

3. What changes in behavior, attitude, experiences and skills are observed among out of school time participants (i.e., direct outcomes)?

4. What changes in academic performance are observed among out of school time participants (i.e., contributory outcomes)?

This story, or the results of this year’s evaluation, will then be used to modify the above Theory of Action for future years.

(4)

Section Three: Evaluation Activities and Data Sources

Public Profit proposes a mixed methods study design that combines observations from site visits, surveys, and student-level data to explore the evaluation questions.

Key evaluation activities for the 2011-12 ExCEL evaluation include:

• Create data collection tools to reflect ExCEL priorities, including program quality and student outcomes.

• Coordinate with attendance management system (Cityspan) to capture attendance information.

• Observe program activities at 40 sites, a sample of ExCEL’s programs, using an observation tool aligned with AFA and ExCEL-specific program quality goals (site visits).

• Create and maintain evaluation dashboard addressing attendance and program quality. • Field online participant and staff/principal surveys in Spring 2012. Available in English,

Spanish and Chinese.

• Prepare a final evaluation findings report that includes an analysis of participation, program quality and participant outcomes. Provide site-level data reports that summarize key program metrics.

• Prepare drafts of mandated federal and state reporting requirements for review and completion by ExCEL staff.

Evaluation Data Sources

EVALUATION QUESTION INDICATORS DATA SOURCES

What progress have ExCEL programs made toward target enrollment and daily

attendance rates?

Attendance • Program Grant _{Amounts (ExCEL)} • Program

Attendance (Cityspan) Average daily attendance

In what ways are ExCEL programs providing high quality services? Safe environment • Site Visits • Stakeholder surveys Supportive environment Interaction Engagement School-day alignment Physical activity

(5)

EVALUATION QUESTION INDICATORS DATA SOURCES

Skill-building activities

What changes in behavior, attitude, and skills are

observed among out of school time participants (i.e., direct outcomes)? Increased safety • Stakeholder surveys New skills New experiences

Relationships with adults Relationships with peers Academic behaviors School engagement

What changes in academic performance are observed among out of school time participants (i.e.,

contributory outcomes)?

English Fluency

Data from RPA: • English Learner Status • State Assessments • CAHSEE • A to G Status • Credits Earned • High School Graduation Academic Improvement: CST ELA Academic Improvement: CST Math

CAHSEE Pass Rates Credits Earned

High School Graduation A to G Completion

Note: Our ability to successfully complete the evaluation is directly related to the quality and timeliness of information received from individual sites and from the SF ExCEL program office. Substantial errors or inconsistencies with the data provided to Public Profit will affect the deliverable dates noted in the following section.

(6)

Section Four: Evaluation Timeline

Month Activities Purpose

August 2011

Evaluation Overview

Presentation to Site Coordinators and CBO Partners at August Institute

Learn about how programs will be assessed as part of the annual citywide evaluation; understand the timeline for key evaluation activities.

Convene Working Group Determine guiding priorities for 2011-12 Evaluation, especially defining program quality.

September 2011 Develop Site Visit Protocol Determine definition of program _{quality to be used in 2011-12.}

October 2011

Present Site Visit Protocol

Presentation to Site Coordinators.

Ensure transparency of the 2011-12 evaluation.

Mandated Reporting

Federal APR and CDE Elementary and Middle School annual reports

Ensure ExCEL’s compliance with federal and state grant requirements.

November 2011

Site Visit Pilot

Test run of Site Visit Protocol at 5 ExCEL sites.

Determine reasonable (observable) definition of program quality to be used in 2011-12.

Presentation on Evaluation

Update to CBOs on Evaluation activities and progress.

Ensure transparency of the 2011-12 evaluation.

Draft Stakeholder Surveys

Create draft student and other surveys based on Evaluation Plan and prior year surveys.

Determine additional measures of both program quality and student direct outcomes.

2010-2011 Program Profiles

Summaries of prior year based on data from the district.

Demonstrate commitment to transparency.

Provide information for on-going program improvement.

December 2011

Site Visit Protocol

Finalize items and structure.

Determine reasonable (observable) definition of program quality to be used in 2011-12.

Monthly Dashboard

Internal dashboard highlighting Cityspan and site visit information goes live.

Tell the story of 2011-12 ExCEL as it unfolds.

(7)

January 2012

Present Final Site Visit Protocol

Update CBO Partners and Site Coordinators on Evaluation progress and present site visit protocol.

Ensure transparency of the evaluation.

Prepare selected sites for a site visit. Familiarize all sites with the

definition of program quality as determined for 2011-12.

Quality Assurance Check (Cityspan)

Check data entry and activity categorization in light of APR and CDE reporting requirements.

January – April

2012 Conduct Site Visits

Conduct 40 site visits to observe program quality.

Support on-going program

improvement at those selected sites. Ensure transparency by returning site visit report within 2 weeks of site visit.

Support grant compliance through site visits of FPM-selected sites.

February 2012 Finalize Stakeholder Surveys Translate into Spanish and Chinese. Load into online platform.

Gather measures of program quality and student direct outcomes.

March 2011

Present Stakeholder Surveys

Update CBO Partners and Site Coordinators on Evaluation progress and present stakeholder surveys.

Ensure transparency of the evaluation.

Familiarize all sites with the definition of program quality and direct outcomes to be measured by surveys.

Field Online Stakeholder Surveys Gather measures of program quality _{and student direct outcomes.}

April 2012

Draft Annual Report

Determine structure and priorities. Begin to outline the story about 2011-12 ExCEL programs.

June 2012

Ensure ExCEL’s Summer Programs compliance with federal and state grant requirements.

(8)

July 2012 Obtain Demographic & School Day Attendance Data from district

All but CST scores.

August 2012

Obtain Test Score Data from district

Deliver 2011-12 Program Profiles

Demonstrate commitment to transparency.

September 2012

Mandated Reporting

Federal APR annual report draft submitted to ExCEL

CDE Elementary/Middle School and ASSETS annual reports prepared

Annual Report

Report to ExCEL and sites.

Tell the 2011-12 ExCEL story. Support on-going program improvement.