Baltimore s After School Strategy. Assessing After-School Program Quality in Baltimore: Interim Findings of An In-Depth Study of Eight Programs

(1)

Baltimore’s After School Strategy

Assessing After-School Program Quality in Baltimore:

Interim Findings of An In-Depth Study of Eight Programs

Prepared by Carolyn Marzke

with Lee Pearson

September 2003

Policy Studies Associates 1718 Connecticut Avenue, NW

Washington, DC 20009

Under contract to

The Family League of Baltimore City Baltimore, Maryland

(2)

Executive Summary

Baltimore’s After School Strategy is a collaborative, city wide initiative led by the Safe and Sound Campaign (SAS), the Family League of Baltimore City (FLBC), and The After-School Institute (TASI) to expand and improve the quality of after-school programs for children and youth in Baltimore City. Launched in early 1998, the After School Strategy has directed more than $29 million from a combination of private and public sources to support (1) the development of an after-school “system” and infrastructure in Baltimore, and (2) expansion and quality improvement in more than forty after-school programs operating in more than one hundred sites around the city (Evaluation Overview, 2003). To date, the Strategy has funded over 40 provider organizations operating more than 80 after-school program sites.

To guide after-school program quality improvement efforts, the After-School Strategy

developed the Standards for Baltimore After-School/Out-of-School Opportunities in YouthPlaces. The Standards are divided into two parts: (1) Organizational Standards, which address the quality of human relationships and the overall program infrastructure; and (2) Program Standards, which address

program content and opportunities for skill-building and mastery.

The goal of improving quality is integral to the Strategy’s theory of change as both a means of drawing more youth to participate in after-school programs and for increasing the likelihood that programs will affect outcomes for participating youth. The In-Depth Study on After-School Program Quality was designed to inform the After School Strategy’s efforts to assess and improve program quality by examining quality in a sample of eight programs against the Standards and refining a systematic approach to quality assessment based on structured observations of program activities. The study was guided by the following research questions:

y What is the quality of these eight programs assessed against the Standards for Baltimore After-School/Out-of-School Opportunities in YouthPlaces?

¾ What is the quality of interactions among youth and staff, as defined in the Organizational Standards?

¾ What is the quality of program content and opportunities for skill-building and mastery, as defined in the Program Standards?

y What are the links between quality and site features that may affect quality, such as cost or staffing strategies?

y What is the relationship between quality and program utilization? Are youth more likely to attend higher-quality programs?

y What is the relationship between program quality, other program features, and youths’ self-reported experiences and perceptions of the benefits of participating in after-school programs?

The findings in this interim report are based on data collected in the eight study sites during the 2002-03 program year, including:

(4)

y Structured observations of 189 after-school program activities in the fall of 2002 and the spring of 2003, during which observers rated the prevalence of indicators of the quality of youth interactions, staff-youth interactions, mastery orientation, opportunities for higher-order skill-building, and youth engagement.

y Surveys of 241 youth participants the fall of 2002, which included items about youth characteristics; participation levels and reasons for attending; other after-school destinations; and perceptions of youth interactions, staff-youth interactions, and the benefits of participation.

y Data about program enrollment, attendance, utilization, and cost from the Family League of Baltimore City Management Information System for the 2002-03 program year

Program Features Likely to be Associated With Quality

The eight programs included in the study illustrate the diversity of after-school programs funded by the strategy. They are sponsored by large and small agencies, some public and some private, and three of the eight programs are school-based. Three programs serve elementary grades only, one serves middle school students only, two serve elementary and middle school grades, and two are open to youth at all grade levels.

On Average, Programs Served Fewer Students Than They Were Contracted to Serve

While the range of enrollment target levels was wide (30 to 125), six of the eight programs included in the study were contracted to serve between 40 and 60 youth. The average daily attendance (ADA) for study sites between September 2002 and January 2003 was 49 youth. Compared to the total number of youth the study sites were contracted to serve, these attendance levels translated into a mean utilization rate of 0.78 (i.e., 78 percent of contracted “slots” were filled on any given day). At the extremes, the utilization rate in one site was 0.26, and another site served more than the contracted number of youth for a utilization rate of 1.3.

Program Budgets Vary Widely Among the Eight Sites, As Do Estimates of Programs’ Cost per Youth

The average FY03 budget among the study sites was $132,463. The average cost per youth, based on average daily attendance (ADA) for September 2002 through January 2003, was $4,648, with a low of $583 and a high of $10,348.

Program Staff Vary in Their Size and Qualifications

The study sites employ an average of nine full- and part-time staff, although staff sizes range from four to 16. Five of the eight sites employ certified teachers as after-school program staff. In the

(5)

absence of staff survey data for the 2002-03 program year, the presence of certified teachers on staff was the sole measure available for staff qualifications.

Program Schedules, Mix of Activities Offered, and Extent of Enrichment

Programming

The after-school components of the programs included in the study operate on varying

schedules for an average of 11 hours per week, with a range of six to 15 hours of operation per week. Five sites operate five days a week, one is open four days a week, and two meet twice a week. For the most part, youth in these programs have relatively little choice about the activities in which they participate. Five programs maintain group-based assigned schedules in which all activities are required, and another three have menu-based schedules.

Programs offered a mix of the types of activities specified in the Standards: homework help, other academic/cognitive enrichment, arts, sports, workforce development, civic development, and open time. Many programs also offered activities that were difficult to place in one of the program areas defined in the Standards, such as cooking and modeling. The study team computed both the number of different activities offered and an average number of hours per week to describe not only the “richness” of programming but also the level of exposure youth receive to a particular type of activity. The study sites offered an average of seven different kinds of activities per week. Overall, sites offered more hours per week of academic/cognitive development activities than any other type of activity, including homework help. Arts activities were also prevalent, with the remaining categories less well-represented across programs. All sites but one offered both academic and non-academic activities as required by the Standards.

In addition to estimating the hours per week of activities offered in each program area, we also estimated the average total hours of enrichment activities offered across program areas, excluding homework time and open time. The amount of enrichment available varied tremendously among sites, from a low of one hour per week (rounding up) to a high of 16 hours per week. Sites fell into two categories based on this estimate—those offering 11 or more hours per week and those offering 4 ½ or fewer hours per week of enrichment. School-based programs offered substantially more enrichment hours per week on average—13 hours per week compared to 3 hours per week in community-based programs.

Quality of Youth Interactions

Observed Interactions Among Youth Were Generally Positive, While Youth Survey Respondents Offered More Mixed Assessments of Peer Interactions

During the activity observations conducted for the study, observers recorded the number of youth present and rated youths’ interactions with one another, documenting both positive and negative indicators of the quality of these interactions. These ratings are summarized in an Observed Youth Interactions Index. In addition, the youth survey included items about peer interactions, which are summarized in a Youth Survey Peer Interactions Scale.

(6)

The average number of youth present in observed activities was 12, and the median group size was 10. These are well within the guidelines established by the Standards, and very few activities exceeded those recommendations.

Youth interactions in observed activities were generally positive and typically characterized by warm and friendly exchanges in the course of socializing. Other types of interactions demonstrating specific interpersonal skills—such as sharing materials or listening respectfully to peers during discussions or when they are talking in front of the group—were less prevalent. While observers recorded very few overtly negative interactions among youth, youth themselves indicated some ambivalence about the extent to which their peers cared about each other, got along well, and treated each other with respect. The majority of youth offered positive responses on these survey items, but were more likely to agree “a little” than to agree “a lot”, and a substantial minority offered negative responses to these items.

The Quality of Observed Youth Interactions Was Positively Associated with Youth Engagement Levels

Youth interactions were somewhat more positive in activities marked by high levels of youth engagement and in activities observed in the spring. Youth interactions were consistent across different types of activities, however, and youth engagement and time of year explained relatively little of the variation among sites in youth interactions observation index scores.

Youths’ Assessments of Peer Interactions Were Positively Associated with the Degree of Mastery Orientation in Observed Activities

Youth in programs whose activities were notably more mastery-oriented offered more positive assessments of peer interactions. Older youth and boys also reported more positive views about peer interactions.

Quality of Staff-Youth Interactions

Observers and Youth Offered Positive Assessments of Staff-Youth Interactions Overall, Although the Extent to Which Staff Provide Individualized Attention to Youth Was Less Clear

During the activity observations conducted for the study, observers also recorded the number of staff present during the activity and rated staff-youth interactions, documenting both positive and negative indicators of the quality of these interactions. These ratings are summarized in an Observed Staff-Youth Interactions Index. The youth survey also included items about staff-youth interactions, which are summarized in a Youth Survey Staff-Youth Interactions Scale.

The average staff:youth ratio in the activities observed for the study was 1:7—one staff member for every seven youth—well below the guidelines of between 1:10 and 1:15 established by the

Standards. Staff-youth ratios were at or below 1:10 in 80 percent of observed activities, and at or below 1:15 in 90 percent of activities.

(7)

The quality of staff-youth interactions across observed activities was generally high, reflecting the prevalence of positive interactions and the relative absence of overtly negative interactions among staff and youth. Staff consistently used positive language and tone of voice, and in many activities their interactions with youth were warm and friendly. Informal conversations among staff and youth and individualized attention and encouragement were observed less often than group-based, task-focused interactions. Likewise, youth offered relatively positive overall assessments of their own interactions with staff, and positive perceptions of how staff treat youth. There was more variation, however, in the extent to which youth reported individualized, personal interactions with adults in these programs. In particular, forty percent of youth survey respondents reported that they had never talked one-on-one with a staff member in their after-school programs.

There Were No Clear Patterns of Association Between the Quality of Observed Staff-Youth Interactions, Youths’ Assessments of the Quality of Staff-Youth Interactions, and Site Features

Observed staff-youth interactions were higher in spring than in the fall, but this was the only significant factor associated with variations in this measure across sites. The quality of staff-youth interactions was relatively consistent across different types of activities, and was not associated with site features such as location, cost, or the presence of certified teachers on staff.

Similarly, youths’ views about the quality of staff-youth interactions did not differ based on observation measures of quality, nor were there any associations between site features and youths’ assessments of staff-youth interactions, whether they would talk to staff about a serious personal concern, or whether they had ever talked one-on-one with a staff member.

The Qualities That Influence Youths’ Attitudes Toward Staff May Not Be Well-Captured in Snapshot Observations

The results of observation and survey measures of staff-youth interactions did not correspond well with one another. In particular, there was more variation in observers’ assessments of staff-youth interactions than in youths’ responses to survey items about how staff treat youth. Moreover, youths’ responses to questions about more individualized or personal interactions with staff were negatively associated with observation measures. The survey results are based on non-representative samples of youth at the site level, which may also explain the low correspondence between the survey and

observation measures. Interviews with youth are likely a better way to assess the qualities of programs and staff that promote positive staff-youth relationships, and may generate more sensitive survey items and observation indicators in the future.

Extent of Opportunities for Skill-Building and Mastery

The Extent of Opportunities for Skill-Building and Mastery Was Low Overall

The degree to which observed activities were designed to help youth master skills was low overall, but varied considerably across sites and types of activities. In general, when staff were focused on helping youth to learn something, staff-centered instructional strategies were more prevalent than other more youth-centered and individualized strategies, such as providing feedback to individual youth

(8)

or offering opportunities for youth to produce something or to perform. Higher-order skill-building opportunities such as cooperative activities and activities that require youth to think strategically or solve complex problems, answer thought-provoking questions, or engage in extended discussions with their peers were observed infrequently during observed activities.

Measures of Mastery Orientation and Higher-Order Skill-Building Were Higher in Enrichment Activities and in Sites with Certified Teachers on Staff

The extent of opportunities for skill-building and mastery varied substantially across sites, and the type of activity and the presence of certified teachers on staff were both strongly associated with ratings of the mastery orientation and higher-order skill-building in observed activities. Intentional skill-building opportunities were most pronounced in arts, academic enrichment activities, and the collection of activities that were difficult to categorize by program area, such as cooking, gardening, and chess club. Academic and “other” enrichment activities also incorporated more opportunities for higher-order skill development. Not surprisingly, these types of activities were more mastery-focused than sports or open time activities.

Efforts to build cross-cutting skills such as literacy, critical thinking, or cooperation in the context of arts, sports, or other apparently recreational activities were the exception rather than the rule in observed activities. In most sites, arts activities targeted artistic skills and products were individual rather than collective, and academic activities generally sought to develop individual youths’ basic reading and math skills. Open time and sports activities were unstructured opportunities for youth to relax, play, and socialize informally with one another in a supervised setting. While these activities provided opportunities for youth to build some skills on their own, the skill-building was not facilitated or enhanced much by the adults in the room (i.e., these were not intentional learning environments).

Other Findings Related to Quality

Observed Quality Was Higher in the Spring Than in the Fall

Interactions among youth and staff were slightly more positive at the end of the program year than near the beginning. In addition, observers saw more evidence of higher-order skill-building, regardless of the mix of activities observed, although the extent of such opportunities remained quite limited overall. The small improvement in observed interactions may reflect the fact that youth and staff had had time to get to know each other better, and staff may have relaxed somewhat in their approach to leading activities, leading to warmer interactions overall. The difference in interactions may also be explained by the departure of youth who did not get along well with peers or staff between the fall and spring rounds of observations. To the extent that embedding opportunities for higher-order skill-building into activities can be challenging, staff may have gained confidence over the year and stepped up their efforts to make mastery-oriented activities more challenging.

Program Cost Was Inversely Associated With Measures of Program Quality

Relationships between cost and the various measures of quality developed for the study were consistently negative. These findings run counter to expectations that programs spending more than the

(9)

typical per-youth cost estimates circulating in the after-school literature will be of higher quality. They are particularly surprising given the relationship between certified teachers and measures of skill-building and mastery and the fact that it is often more expensive to hire certified teachers than other types of staff. In fact, the presence of certified teachers on staff was itself inversely associated with program costs per youth. One hypothesis based on this finding is that the following trade-off exists between location, cost, and the quality of program content: school-based programs have lower facilities costs and a ready pool of enrollees, and can therefore afford to hire teachers at a relatively low cost per youth, while community-based programs’ overhead costs and higher recruiting costs and challenges make such staffing strategies less affordable.

There Were Differences Among Sites in Overall Levels of Quality

Analyses of differences in the observation measures for youth interactions, staff-youth interactions, mastery orientation, and higher-order skill-building indicate that some sites are consistently high and others consistently low in quality across these domains. Two sites received notably high index scores on three of the four observation measures compared to other sites, with both receiving particularly high scores for youth interactions and mastery orientation in addition to one other area. Two other sites received notably low scores in at least three domains compared to other sites. Both of these sites received particularly low scores for staff-youth interactions, mastery orientation, and higher-order skill-building, and one scored lower than other sites on all four measures.

Youth Engagement and Satisfaction

A central hypothesis in the After School Strategy’s theory of change is that youth will find higher-quality programs more interesting, fun, and engaging, and consequently will attend more often. The study assessed youth engagement during activity observations, and asked youth survey respondents to report their overall satisfaction with their programs.

Youth Were On-Task and Appeared To Be Enjoying Themselves in Most Observed Activities, Particularly in Arts Activities and Activities with Smaller Group Sizes

Overall, youth were fairly engaged in most observed activities, and rarely were disruptive to the point that such behavior affected the overall climate of the activity. Youth were somewhat more engaged in arts activities and less engaged (although on-task) during homework time and other academically-focused activities. In addition, activities with smaller group sizes had higher youth engagement index scores.

Most Youth Liked Their Programs, and Youth Who Offered More Positive Assessments of Peer and Staff-Youth Interactions Also Reported Higher Levels of Satisfaction

Survey responses indicate that nearly two-thirds of youth “really” like their programs, and nearly one-third “sort of like” their programs. Few youth survey respondents indicated that they did not like attending their after-school programs. Younger youth reported somewhat higher levels of satisfaction than older youth. In addition, there was a strong positive relationship between youths’

(10)

perceptions of the quality of peer and staff-youth interactions and their overall satisfaction with their programs. Youth satisfaction was not associated with site features such as the presence of certified teachers on staff, cost, or location. Likewise, none of the five measures of observed quality was a significant predictor of youths’ self-reported satisfaction.

Associations Between Quality and Utilization

School-Based Sites and Sites Offering Fewer Than Five Hours Per Week of Enrichment Had Higher Utilization Rates Than More Enrichment-Oriented Programs in Other Settings

Sites offering more hours of enrichment per week seemed to have slightly lower utilization rates than sites offering fewer enrichment hours, controlling for program location (school- or community-based). This suggests that youth may prefer more “down time” after school, and are more likely to attend recreation-oriented programs. School-based sites had higher utilization rates, suggesting that youth are more likely to attend programs that are easier to get to after school. There were no clear relationships between utilization and other site features or measures of quality based either on observations or youth survey responses. The analysis of relationships between site features,

observation measures, and utilization was constrained by the small number of sites in the sample, and the association between location and utilization was very sensitive to the treatment of two sites with very different utilization rates from the other six.

Associations Between Quality and Youths’ Perceptions of the Benefits of

Participation

There Were Few Clear Relationships Between Site Features, Measures of Observed Quality, and Youths’ Perceptions of the Benefits of Participation

The youth survey included 18 items asking youth whether they had learned to do something really well in their after-school programs and whether their programs had helped them in school, socially, with decision-making skills, or with career or college exploration. The average Youth Survey Program Impact Scale score was 8.9 out of 18 total possible points, which means that, on average, youth identified between eight and nine areas in which the after-school program had helped them to learn or improve in some way.

Nearly all respondents reported that their after-school program had helped them learn to do something other than academics really well, and almost all youth survey respondents indicated that they had learned to work together with others at the program. Nearly two-thirds of youth indicated that they had learned to do fine arts and to speak or perform in front of a group at their programs, and almost half reported learning to act, dance, or sing. Most youth reported that the program had helped them to get their homework done and to do better in school, to make new friends, and to learn resistance skills. Many youth reported that the program had helped them to enjoy school more or to enjoy reading more, and to avoid getting into fights. Despite the dearth of programming in this area, the majority of youth also reported learning about jobs or careers or about college in their programs.

(11)

There were no significant variations in youths’ overall self-reported program impact based on respondents’ age gender, or length of participation, nor were there any significant differences in these scores based on site features or measures of observed quality. The lack of association between measures of observed quality and youths’ assessment of the benefits of participation persisted in analyses of individual survey items. While site features were associated with a few items, no clear patterns of association between site features, observed quality, and youths’ views about how their programs have helped them emerged from these analyses.

The lack of association between measures of observed quality and youths’ assessment of how the program has helped them suggests that outside views about program quality have little bearing on what youth believe (or at least report) about the benefits of participation. This study did not collect data to explore whether observed quality is associated with third-party assessments of youth outcomes. The findings based on youths’ reports therefore should not be interpreted as evidence that quality is

irrelevant to program impact.

(12)

1. Introduction

This report describes findings from the first year of a two-year study of program quality in eight after-school programs supported by Baltimore’s After School Strategy. The findings are based on activity observations conducted in fall 2002 and spring 2003, as well as youth surveys administered in fall 2002.

Baltimore’s After School Strategy

Baltimore’s After School Strategy is a collaborative, citywide initiative led by the Safe and Sound Campaign (SAS), the Family League of Baltimore City (FLBC), and The After-School Institute (TASI) to expand and improve the quality of after-school programs for children and youth in Baltimore City. The Strategy is one element of the city’s Urban Health Initiative, funded by the Robert Wood Johnson Foundation and spearheaded by the Safe and Sound Campaign to improve the health and safety of children in Baltimore City.

Launched in early 1998, the After School Strategy has directed more than $29 million from a combination of private and public sources to support (1) the development of an after-school “system” and infrastructure in Baltimore, and (2) expansion and quality improvement in more than forty after-school programs operating in more than one hundred sites around the city (Evaluation Overview, 2003). To date, the Strategy has funded over 40 provider organizations operating more than 80 after-school program sites. Figure 1 summarizes the core elements of the After School Strategy (Halpern & Horton, 2003).

The goal of improving program quality is integral to the Strategy’s theory of change as both a means of drawing more youth to participate in after-school programs and for increasing the likelihood that programs will affect outcomes for participating youth. As Figure 1 illustrates, many of the Strategy’s core components—the development of standards and quality control mechanisms, the formation of a training and technical assistance organization, linking funding and accountability to quality improvement—relate to quality. The In-Depth Study of After-School Program Quality was designed to inform the After School Strategy’s efforts to assess and improve program quality by

examining quality in a subset of grantees and refining a systematic approach to quality assessment based on structured observations of program activities.

(13)

Baltimore’s Program Quality Standards

One of the first activities undertaken by the After-School Strategy team in 1998 was the development of the Standards for Baltimore After-School/Out-of-School Opportunities in YouthPlaces. The Standards are based on the National School-Age Care Alliance’s (NSACA) Standards for Quality School-Age Care, with additions designed to (1) ensure that the standards reflect the full age range of children and youth served by after-school programs, and (2) address the quality of program content. The Standards are divided into two parts—Organizational Standards and Program Standards (a full list of the Standards is included in Appendix A). The Organizational Standards address the quality of YouthPlaces' overall program infrastructure and socioemotional climate in five sections, all of which apply to all programs:

Figure 1

Principal Elements of Baltimore’s After-School Strategy

Adapted from The Design and Implementation of Baltimore’s After-School Strategy: An In-Process Look, by Robert Halpern and Carol Horton (May 2003) ➢Identifying and seeking out new funding sources

➢Developing and promoting quality standards

➢Linking program funding to commitment to move toward and ultimately achieve standards

➢Using a fiscal intermediary (The Family League of Baltimore City)

➢Establishing a training, technical assistance and network-building intermediary (The After-School Institute)

➢Implementing an initiative to provide skilled instruction and apprenticeship in arts, sports, and academics (A-Teams)

➢Building capacity to collect and use information for planning, decision-making, quality control, program improvement and other purposes

➢Developing a committee in which stakeholders work together on system-building tasks ➢Deepening the public’s, government and elected officials’ understanding of the value of

(14)

y Human Relationships y Indoor Environment y Outdoor Environment y Health, Safety, and Nutrition y Administration

These five categories contain a total of 34 standards, each of which is defined by one or more indicators for a total of 172 indicators.

The Program Standards address the quality and richness of program activities and content. They include standards that apply across all program activities, as well as standards specific to each of six program content areas. There are 17 Program Standards, which are defined by a total of 66 indicators. The six program areas articulated in the Program Standards are:

y Cognitive development y Recreation y Workforce development y Artistic development y Civic development y Open time

Recent reviews of research in child and youth development emphasize the importance of opportunities for skill-building and mastery for positive development (see, for example, Eccles & Gootman, 2002). The Program Standards represented a unique contribution to the emerging field of quality standards for school age care by expanding the focus on content as a dimension of program quality. As written, they provide for somewhat more flexibility of application than do the

Organizational Standards, allowing for the kind of diversity in content across programs that is desirable within a community if youth are to have a variety of choices. For example, the Standards do not require programs to offer activities in all six program areas. Instead, they require that all programs provide skill-building and mastery opportunities in at least one non-academic program area (e.g., arts, recreation, civic, employment). All programs are expected to promote cognitive and academic development, however, directly through academic activities as well as indirectly through “an infusion of opportunities for cognitive development” throughout the program.

(15)

Assessing Program Quality Against the Standards

As the fiscal intermediary for the Strategy’s grants to providers, the Family League of Baltimore City (FLBC) has promoted use of the Standards through its after-school grant application, contracting, and contract monitoring procedures. For example, applicants are required to conduct self-assessments against the Standards, report the extent to which they meet each standard, and describe priorities and plans for addressing unmet standards. However, the Strategy did not provide guidelines for how programs should conduct these self-assessments. As Halpern and Horton observed in their recent evaluation of the Strategy’s implementation, “…there was no single, specified process that grantees had to adhere to when conducting their initial self-assessments…Consequently, there is no way of knowing precisely how each of the grantees determined where it initially stood with regard to the Standards without going back and carefully re-reading each individual set of proposals.”1_{Indeed, a} review of the proposals submitted by the eight programs included in the In-Depth Study of Program Quality yielded little or no information about the self-assessment processes in most of these programs.

FLBC Contract Managers responsible for monitoring each funded program assess “Standards adherence” during periodic site visits. Again, however, the process for assessing whether programs meet the Standards is not defined clearly enough to enable Contract Managers to be confident in the event that their conclusions differ from programs’ self-assessment results (Halpern & Horton, 2003). Reflecting on the subjectivity of the self-assessment and monitoring processes, one site leader

interviewed for this study noted, “If they said you didn’t meet this [standard], I’d say you should have been here yesterday.” Another commented, “Standards are subjective. I think I’ve met the standard, but someone else might think I’m not meeting the standard.”

Program directors and site leaders also commented on the tension between honoring programs’ individual missions and priorities and satisfying what they interpret individually as the requirements represented in the Standards. As one program director observed, “The influx of training and money are great, but we have a mission, history, and culture of our own. Certain things are more important to us.” Another commented, “There are [lots of] centers under my jurisdiction. They can’t be mirror images of each other, they are not all sponsored by Safe and Sound. I have to think about the standards as they apply to a particular situation.” Finally, another noted, “We need to validate the issue of variety in programming. The problem with the Standards was that there was an expectation that we would all do the same thing. Requiring that programs meet a certain number of standards is okay, but not all standards should be required.” These comments suggest that some questions remain among program leaders not only about how to judge whether programs do meet the standards, but also about whether all programs are or should be expected to meet all standards.

1_{Halpern & Horton, p.15.}

(16)

Finally, any assessment of program quality must consider external factors likely to affect quality. The unstable funding environment for after-school programs and the resulting instability in staffing often constrain programs’ quality improvement efforts. One program director interviewed for the quality study commented directly on the influence of funding on priorities, noting that “certain program elements [related to the Standards] had their day but are not focused on right now, and some are funding-contingent.”

(17)

(18)

2. Study Goals and Methods

The In-Depth Study of After-School Program Quality is designed to assess quality and explore factors affecting quality in eight after-school programs funded by Baltimore’s After School Strategy. The study design reflects the Strategy’s interest in developing a more systematic approach to assessing program quality against the Standards.

An overview of the theory of change for the study is presented in Figure 2. The theory guiding the study expands on the After School Strategy’s theory of change to identify site characteristics and inputs, elements of program and activity quality as outlined in the Standards, and relationships between these program features and utilization and youth outcomes. Specifically, the goals of the study are to:

y Develop a systematic and valid approach to assessing after-school program quality across the range of program features included in the Standards for After-School Opportunities in Baltimore

y Assess the quality of eight after-school programs funded by Baltimore’s After School Strategy against the Standards for Baltimore After-School/Out-of-School Opportunities in YouthPlaces

y Explore links between quality and program attributes and resources that may affect quality in these eight programs

y Describe the relationship between quality, program utilization, and youths’ self-reported experiences and perceptions of the benefits of participating in after-school programs

Given the small number of sites included in the sample, this is a case study rather than a comprehensive evaluation of the quality of after-school programs supported by the After School Strategy. Moreover, the study is intended to describe patterns in quality across these eight sites, and does not report on the quality of individual programs. To protect the confidentiality of individual sites, this report identifies sites by letters or numbers, randomly assigned to prevent linking sites to their identifiers based on descriptive data.

(19)

Figure 2

Theory of Change for the In-Depth Study of After-School Program Quality

Total budget Cost per youth

Location

Hours of operation per week Enrollment size Ages served Type of activity Organizational Standards Number of adults Number of youth Child-adult ratio Organizational Standards Youth Interactions Staff-Youth Interactions Program Standards Mastery Orientation Higher-Order Skill-Building

Site Characteristics Activity Characteristics

Attendance/Utilization

Outcomes

Program Standards

Number and types of activities offered

Use of certified teachers

(20)

Site Selection

The study sample includes eight of the more than 40 organizations and 80 after-school program sites funded by the Strategy in fiscal year 2003. It is a purposive sample of sites selected to include (1) a mix of large and small, public and private sponsoring organizations; (2) school-based and

community-based sites; (3) sites in a variety of neighborhoods targeted by the Strategy; (4) sites serving a mix of age ranges; and (5) sites with enrollment levels of 30 or higher to increase the chances of obtaining a large enough survey sample to allow for site-level analyses and comparisons.2_A

comparison of the study sample and all Strategy-funded sites according to enrollment size, utilization rates, and budget and cost figures is provided in Figure 3.

Figure 3 Site Capacity, Utilization,

Budget, and Cost

Study Sites All Funded Sites Total Sites 8 83

Average Number of Youth

Contracted by FLBC to Serve3 57 58

ADA Sept ’02 – Jan ‘03 49 46

Average Utilization Rate4_0.78_0.79

Average Cost per Contracted Slot $3,109 $2,573

Average Cost per Youth $4,648 $4,027

Average Total Program Budget $132,463 $125,017 Average Total Strategy Funds $83,728 $66,690 Average Percent of Budget

Represented by Strategy Funds 65% 60%

Source: FLBC MIS for September 2002 through April 2003

The sites in this study sample are contracted to serve nearly the same number of youth on average as funded sites overall, reported similar ADA between September 2002 and January 2003, and therefore have the same average utilization rate (just shy of 80 percent). Average costs and budget figures are higher for sample sites than for Strategy-funded sites as a whole, however. The average

2_{Sites serving older youth were disproportionately represented in the group of sites excluded for small}

enrollment levels. As a result, the study sample does not include any sites that focus on high-school age youth. PAL centers participating in a separate study also were excluded to prevent imposing additional burden on these sites.

3_{Total contracted to serve represents the total number of youth FLBC expects programs to enroll.} 4_{ADA as a percent of number contracted to serve.}

(21)

total budget for study sites is nearly $7500 higher than the total pool of sites. Study sites received more Strategy funds on average, and those funds represent a slightly larger proportion of their budgets than funded sites as a whole. Likewise, the average cost per slot is a little more than $500 higher for sample sites, and the average cost per youth is more than $600 higher. This reflects the fact that one study site whose budget is particularly large reported very low average daily attendance levels between September 2002 and April 2003.

Study Timeline and Data Collection Activities

The study began in the summer of 2002 and will conclude in the summer of 2004. During the summer and early fall of 2002, PSA staff interviewed program directors and site leaders and refined and tested the observation instrument. In late fall 2002, teams of observers from PSA conducted two-day visits to each program. PSA also administered surveys to participating youth in the third grade or higher in separate visits; survey administration typically spanned two consecutive afternoons. In spring 2003, observers conducted a second round of visits over two afternoons in each site. This report draws from the observations and surveys conducted between October 2002 and May 2003, describing the results of the fall and spring observations and the initial round of youth surveys.

Observation Instrument and Methods

A team of observers visited each site twice during the 2002-2003 program year, interviewing site leaders and conducting structured activity observations over the course of two consecutive

afternoons. To structure the observation visits, PSA developed a set of data collection instruments and a method for conducting activity observations. The observation instruments include data collection forms designed to capture both program-level information and data about the quality of individual activities. A copy of the observation instrument can be found in Appendix B.

This section of the report provides considerable detail about the observation instrument, including an overview of the content, description of the methods, and summary of scoring procedures. While such details generally are relegated to a report appendix, the observation instrument has become a source of considerable interest to the After School Strategy as a potential tool for contract managers, TASI staff, and programs themselves to use for a variety of purposes beyond research and evaluation. Moreover, the observations were central to the study’s efforts to assess program quality, and the concept of structured observations is less familiar than other research methods such as surveys and interviews. For all of these reasons, this section of the report is designed to help readers understand the observation instrument and our approach to conducting the observations.

(22)

Overview of the observation instrument. The observation instrument includes a site leader interview guide, program overview form, and activity observation instrument. The Site Leader Interview Guide includes questions designed to capture basic descriptive information about the program’s goals, activity schedules, and staffing, as well as site leaders’ perspectives about the quality of their program and implementation issues affecting quality. Site visitors record information about program facilities and activity schedules on the Program Overview Form.

The heart of the site visit is a set of activity observations using an After-School Activity Observation Instrument (AOI). The AOI was structured to provide site visitors with a framework for thinking about and recording what they see in ways that yield quantitative as well as qualitative data about the quality of program activities. The AOI builds on the Standards for After-School

Opportunities in Baltimore and is adapted from existing observation instruments developed for assessing school age care programs and school classrooms and instruction. The instruments consulted during the development process include the NSACA Program Observation Tool, a draft of the instrument

developed by Public/Private Ventures for the evaluation of the Extended-Service Schools Initiative, the School Observation Measure (Smith & Ross, 1999), and the School-Age Care Environment Rating Scale (Harms et al, 1996).

The quality of each observed activity is recorded using an Activity Observation Coding Form, which asks observers to mark whether specific activities, behaviors, and opportunities occurred consistently during each observation. These activities, behaviors, and opportunities are grouped into six categories, each of which contains a list of quality indicators (see Figure 4):

y Youth Interactions y Staff-Youth Interactions

y Opportunities for Skill-Building and Mastery y Youth Engagement

y Activity Organization

y Activity Setting and Resources

Most indicators in the Activity Observation Coding Form describe positive behaviors or activities, but as Figure 3 illustrates the instrument also includes negative indicators. Observers mark positive and negative indicators if they see consistent evidence of them during the course of

observation. Some indicators do not apply unless a particular event occurs (these are in italics in Figure 4). For example, observers may not have an opportunity to assess how staff handle serious conflicts among youth if such conflicts do not occur. Likewise, an activity may not include materials, or there may not be more than one staff member present, so ratings of materials or staff interactions are irrelevant. Observers mark indicators of this type “NA” if the triggering event does not occur or the precondition does not exist.

(23)

The AOI also includes an Activity Context Coding Formfor capturing basic descriptive

information about each activity observed. This form includes space to briefly describe the activity and a set of checklists of activity types, types of spaces, materials used, number of staff/adults, grade levels served, number of youth, group size(s), and grouping strategies.

The AOI is a “living” document that has been refined over the course of pilot tests and between rounds of data collection. The instrument was piloted in early fall of 2002, and the observation team made revisions based on this initial pilot. In addition, site visitors conducted an extra day of

observations in the first three site visits conducted for the study during fall of 2002 for purposes of reliability testing. Rather than overwhelming activities with a team of five observers, we conducted paired reliability tests in which every observer was paired with every other observer for two activity observations over the course of these three site visits. Some indicators were dropped after the fall 2002 round of observations and a few others added to capture information that observers recorded in their notes as missing from the form and important to their overall assessments of quality.

Additional reliability checks occurred prior to and during the spring 2003 round of

observations, although weather-related program closures precluded a full round of tests. Inter-observer agreement based on the fall and spring tests was 86 percent overall across all indicators in the

instrument. This means that pairs of observers marked indicators in the same ways 86 percent of the time. Agreement varied slightly across the rating categories, from 83 percent for Youth Interactions to 87 percent for Skill-Building and Mastery and for Youth Engagement.

Observation methods. Each observation visit occurred over two consecutive afternoons, and

included an interview with the site leader, a tour of the program facilities, and a set of activity observations. Site visitors stayed in each activity for about 25 minutes—long enough to conduct two ten-minute observation segments using the Activity Observation Coding Form and to record additional descriptive information about the activity using the Activity Context Coding Form. Site visitors observed as many different program activities as possible given programs’ hours of operation and activity schedules

(24)

Figure 4

Activity Observation Form Categories and Indicators Youth Interactions

Youth:

+ Generally have warm, friendly interactions with each other

+ Cooperate/share with each other

+ Listen to each other/treat each other with respect + Discuss/resolve disagreements/disputes constructively

- Tease each other in a clearly unfriendly way - Threaten/bully/intimidate each other Staff-Youth Interactions

Staff:

+ Are somewhat engaged with youth + Are highly engaged with youth

+ Have warm, friendly interactions with youth + Engage informally in conversations with youth + Use positive language and tone of voice with youth + Listen actively and attentively to youth

+Praise/encourage youth

+ Use positive behavior management techniques + Help youth to resolve disagreements constructively

- Have few or no interactions with youth - Appear bored or distant

- Use harsh disciplinary methods

- Belittle youth or embarrass them in front of peers - Leave youth unsupervised

- Are unaware of teasing/bullying or other serious conflicts among youth

Youth:

+ Interact constructively/positively with staff + Seek out positive contact/interactions with staff - Ignore directions/tune staff out

- Are rude/actively negative toward staff Youth Engagement

+ Youth are actively engaged, focused, interested, enjoying the activity

- Youth are disruptive and/or out of control

Opportunities for Skill-Building and Mastery

Staff:

+ Are clearly focused on instruction/helping youth to master a skill or learn something new

+ Provide direct instruction/lecture/demonstrate/give directions

+ Engage youth in brief question/answer exchanges + Ask youth “why”, “how”, and “what if” questions that require complex answers

+ Facilitate discussion among youth

+ Critique/offer constructive feedback to individual youth - Criticize youths’ work without offering guidance - Discourage appropriate/reasonable questions/discussion/ requests

Youth:

+ Write (not for homework)

+ Read or are read to (not for homework)

+ Work cooperatively with each other to achieve a goal + Work on projects with culminating products or events + Choose what or how they do something or help determine the direction of an activity

+ Lead activities or groups of peers + Tutor/mentor other youth

+ Think strategically/analyze/solve complex problems + Use information to accomplish a goal/make a decision + Participate in structured discussions

Activity Organization

+ Instructions are clear/youth know what to do + Activity is appropriate for youths’ skill levels + Staff have prepared/assembled needed materials

+ Staff work well together

Activity Setting and Resources

+ Materials/equipment are functional/in reasonably good condition

+ There are enough materials

+ There is enough room for the activity

+ External interruptions/distractions are minimal and/or manageable

- Space is unsafe for the activity - Other safety concerns

(25)

Observers selected activities based on both the breadth and depth of programming across the program areas defined in the Standards (e.g., academics, arts, sports, open time, etc.). In programs offering a wide variety of activities across multiple program areas, observers selected as many different types of activities as possible. In programs offering many activities in a particular program area and a few in others, observers accordingly selected a larger proportion of activities from the program area most heavily represented in the schedule, while making an effort to observe the “outliers” as well. Observers also attempted to visit activities for all age groups served, although it was not always possible to see every type of activity for every age group.

Although the observation team attempted to conduct the same number of activity observations in each site, the number of observations conducted in each site ultimately varied from 20 to 27. Programs’ hours of operation vary even from their written schedules, with some open 2 hours and others 2 ½ or 3 hours each afternoon, which affected the amount of time available for observations. The fluid nature of program schedules also affected the actual number of observations conducted each day. It was not uncommon to encounter activities cancelled due to staff absence or poor attendance. To the extent possible within the study budget, the observation schedule was adjusted in an effort to increase the number of activity observations in sites with shorter or more unpredictable schedules. The total number of activities observed in each site during each round of observations is provided in

Appendix C.

Activity scoring and analysis. Each activity was rated twice over 20 minutes, in two 10-minute

observation segments. At the end of each ten-minute segment, observers checked all indicators of quality that were consistently observed during that time period in each of the six rating categories on the Activity Observation Coding Form. These forms were then scored according to the following rules: for each segment, a positive indicator received a score of 0 if it was not marked, and a score of 1 if it was marked or rated NA. Negative indicators were reverse scored—a negative indicator received a score of 1 if it was not marked or if it was rated NA, and a score of 0 if it was marked. Reverse scoring therefore credits activities for the absence of negative behaviors. The indicator scores for each of the two ten-minute segments were then added together to generate a total score for the activity, so that each indicator received a total score of either 0 (not marked in either segment), 1 (marked in one segment only), or 2 (marked in both segments).

The sum of the indicator scores within each category represents the total quality index score for that category. The mean of the indicator scores within each category represents the mean quality score for that category. The total possible score for each category, based on the total number of positive and negative indicators, is provided in Figure 5. Total index scores vary according to the number of indicators in each category. Mean indicator scores can only vary between 0 and 2, since each indicator received a score of either 0, 1, or 2.

(26)

Figure 5

Range of Possible Scores Quality Index

Total Item Score Mean Item Score

Youth Interactions Index (5 items) 0 – 10 0 - 2

Staff-Youth Interactions Index (11 items) 0 – 22 0 - 2

Opportunities for Skill-Building and Mastery Mastery Orientation Index (4 items)

Higher-Order Skill-Building Index (7 items)

0 – 8 0 – 14

0 – 2 0 – 2

Youth Engagement Index 0 – 4 0 - 2

Activity Organization Index (4 items) 0 – 8 0 - 2

Activity Setting and Resources Index (6 items) 0 – 8 0 - 2

Youth Survey

The study used a revised version of the youth survey instrument developed and administered for the Strategy by PSA in 2001. The survey included questions about the following topic areas:

y Youth demographics

y Risk status of participating youth

y Program participation (enrollment duration, frequency of attendance) y Youths’ sense of safety in the program

y Youths’ perceptions of how program staff treat youth participants y Youths’ perceptions of how program participants treat each other

Activity participation patterns

Extent of opportunities for skill-building and mastery

Extent of opportunities for youth leadership, influence, and service

The Standards served as a framework for the survey instrument. To the extent possible, PSA adapted items from surveys developed and used in other major after-school evaluations. PSA

developed new survey items to capture information about areas of the Standards not addressed in existing instruments.5_{A copy of the survey and information about survey scales is included in} Appendix D.

5_{The instruments from which PSA adapted items include surveys from PSA’s evaluation of the TASC} After-School Program in New York, the Academy for Educational Development’s evaluation of the Beacons After-Schools in New York, and Public/Private Ventures’ study of the contributions of voluntary youth organizations to healthy adolescent development.

(27)

PSA administered the survey over the course of two to three afternoons in each site to youth age 9 or older who had parental permission to participate in the study, based on lists generated by the Family League MIS. A total of 241 youth completed surveys, which represents approximately 73 percent of the youth of any age who had consent to participate in evaluation data collection, and 53 percent of the number of youth these sites were contracted to serve.6_{Age-specific enrollment levels} were not available for these sites, so it is not possible to calculate a true survey response rate for enrolled youth age 9 or older.

Other Data Sources

In addition to data collected specifically for the study, PSA obtained data from the Family League of Baltimore City’s management information system (FLBC MIS) about program budgets, enrollment levels, and average daily attendance. All FLBC MIS data used in the analyses for this report are for the 2002-03 program year; attendance and utilization figures were available for September 2002 through January 2003.

(28)

3. Overview of Participating Programs:

Program Structure and Activity Schedules

The eight programs included in the study illustrate the diversity of after-school programs funded by the Strategy. They are sponsored by large and small agencies, some public and some private, and offer a wide variety of activities. This chapter provides an overview of key characteristics of participating programs sometimes referred to as “structural” program features. Structural program features establish the setting and context for positive relationships and high-quality activities, and include staff qualifications and support, program size and group configuration, financial and physical resources, and a program’s external affiliations.7_{The structural program features described in this} chapter include sponsorship, location, and affiliation with schools; enrollment levels; budgets and costs per youth; and staffing.

This chapter also describes the types of activities offered and activity schedules across the eight study sites. These data enable a rough assessment of the mix of academic and non-academic

opportunities represented in these programs’ schedules, and the breadth and depth of opportunities available across program areas.

Sponsorship, Location, Facilities, and School Affiliation

Figure 6 provides a summary of programs’ structural features, including the type of

organization that sponsors the after-school site, location of services, and school affiliation. Two of the study sites are sponsored by large public agencies, four are sponsored by large private organizations, and two are sponsored by small private organizations, as defined by the Family League. Five operate in community-based settings, and three are located in schools, including one elementary school, one middle school, and one private school. One community-based site is strongly affiliated with a school, offering support to students at school during the school day, although the after-school program itself operates in another location.

These programs operate in facilities used for a variety of other purposes in addition to after-school activities. The three after-school-based programs have access to all available facilities in their buildings. The five community-based programs operate in a variety of facilities, including two

recreation centers, a community center, a facility owned by a private foundation, and an arts building.

7_{Marzke et al, 2002.}

(29)

Figure 6

Summary of Structural Program Features

Study Site Sponsorship Location and School Affiliation Total Budget Use of certified teachers Number of Youth Contracted to Serve Contracted Cost Per Youth ADA Actual Cost Per Youth (Based on ADA) Grade Levels Served Number of staff

A _{Large private Community} _$258,704 _No ₃₀ _$8,623 ₂₅ _$10,348 _1-8 ₁₆ B _{Large private Community} _$217,530 _Yes ₄₀ _$5,438 ₃₂ _$6,798 _3-5 ₆ C _{Small private Community} _$149,983 _No ₄₅ _$3,333 ₄₅ _$3,333 _1-8 _N/A D _{Large public} _Community _$127,435 _No ₅₅ _$2,317 ₃₇ _$3,444 _1-8 ₈ E _{Small private Community} _$108,500 _Yes ₄₀ _$2,713 ₁₁ _$9,864 _1-12 ₆ F _{Large private} _School _$97,438 _Yes ₁₂₅ _$780 ₁₆₇ _$583 _1-8 ₁₃ G _{Large public} _School _$59,416 _Yes ₆₀ _$990 ₃₀ _$1,981 _1-5 ₁₂ H _{Large private} _School _$40,701 _Yes ₆₀ _$678 ₄₉ _$831 _6-8 ₄

(30)

Most programs have adequate indoor space that includes classrooms or meeting rooms as well as other specialized space. Six of the eight programs have access to computer labs, and six have some form of indoor recreation area or gym. One program located in an arts building has access to a dance studio, art room, and several rehearsal and performance spaces. As is typical for urban after-school programs, study sites generally lacked access to safe, inviting outdoor spaces for youth to play games or sports. In an effort to give youth some time outdoors, one program allows participants to play on the sidewalk outside the building, although it is located on a very busy road; older youth play on a vacant lot across the street that is strewn with broken glass and sloped downhill so that balls continually roll into the road. One program must cope with limited indoor space as well as lack of access to outdoor space—all activities are confined to one of two rooms.

Among the activities observed for the study, 60 percent occurred in a classroom or general activity room, 9 percent in a computer lab, 7 percent in a gym, 4 percent in a school cafeteria, 3 percent in some form of outdoor space such as a playground or playing field, 3 percent in a school auditorium, 1 percent in a school library, and 10 percent were in other types of rooms or spaces (e.g., dance studio, teachers’ lounge/kitchen).

Program Budgets and Cost Per Youth Estimates

To inform resource allocation decisions, after-school program funders and policymakers need information about the relationship between funding levels and program quality. To what extent do “you get what you pay for” in after-school programs? To enable exploration of this question, the study obtained data about program budgets and costs from the FLBC MIS for the 2002-03 program year.

Program budgets vary widely among the eight study sites, as do the most recent available estimates of programs’ costs per youth.8_{The average FY03 budget among the study sites was} $132,463; budgets ranged from $40,701 to $258,704. The average cost per youth, based on average daily attendance (ADA) for September 2002 through January 2003, was $4,648, with a low of $583 and a high of $10,348. The cost per youth reflects both variations in program budgets and in program ADA. The site with an estimated cost of $583 per youth reported attendance levels well in excess of the total number of youth it was contracted to serve, while the highest-cost site reported an

exceptionally low ADA during the fall of 2002.

8_{The cost figures come from estimates generated by FLBC based on the total number of youth programs were} contracted to serve and reported average daily attendance figures for September 2002 through January 2003.

(31)

Staffing

Site leaders interviewed for the study attributed the quality of their programs in large measure to the quality of the staff they are able to hire. As one commented, “If you get good staff, the rest is not important. Staff is the absolute key…The better staff you have, the better the program.”

According to programs’ grant proposals, the study sites employ an average of nine full- and part-time staff, although staff sizes range from four to 16. Two sites assign group leaders who stay with a particular group of youth and organize most activities for them. In four sites, staff serve instead as content specialists, and groups of youth rotate through their rooms or spaces for activities in a particular content area. Two sites have both group leaders and activity specialists—group leaders stay with one group of youth, and typically supervise homework help, snack, and attendance, while youth also “travel” to content specialists for certain activities.

Five sites employ certified teachers, including all four of the based or strongly school-linked programs as well as one community-based program. Other information about staff

characteristics and qualifications is not available for the staff employed in the 2002-03 program year; the last round of staff surveys were administered by FLBC in spring 2002, at the close of the 2001-02 program year. Given the level of staff turnover reported in these programs, it is unlikely that these data serve as an accurate estimate of staff characteristics across program years.

Enrollment Size and Ages Served

The number and ages of program participants may affect program content, participation

patterns, and the quality of interactions among youth and staff. For example, research suggests that the quality of interactions between staff and children is lower in programs with larger enrollment

(Rosenthal & Vandell, 1996). Evaluations of after-school programs in New York and other cities have documented consistently lower attendance rates for older youth than for elementary school age children (e.g., Reisner et al, 2002). To explore the relationships between enrollment size, quality, and

attendance, PSA obtained data from the FLBC MIS about ages served, enrollment size, and attendance rates among the eight programs included in the study.

Overall, the study sites were contracted to serve a total of 455 youth for the 2002-03 program year. The average number of youth that programs were contracted to serve was 57. While the range of enrollment target levels was wide (30 to 125), six of the eight programs included in the study were contracted to serve between 40 and 60 youth. The study sites serve a mix of elementary, middle school, and high school grades in various configurations. Three programs serve elementary grades

(32)

only, one serves middle school students only, two serve elementary and middle school grades, and two are open to youth at all grade levels.

Attendance and Utilization Rates

The average daily attendance (ADA) for study sites between September 2002 and January 2003 was 49 youth (Figure 7). On the extremes, one site reported that an average of only 11 youth attended the program each day during this time period, while another reported an ADA of 167. Compared to the total number of youth the study sites were contracted to serve, these attendance levels translated into a mean utilization rate of 0.78 (i.e., 78 percent of contracted “slots” were filled on any given day), with a low of 0.26 and a high of 1.3.

Figure 7 Participation Levels

Study Site

Number of Youth

Contracted to Serve ADA Utilization Rate

Grade Levels Served A _{30 25}_{0.83 1-8} B _{40 32}_{0.81 3-5} C _{45 45}_{1.00 1-8} D _{55 37}_{0.66 1-8} E _{40 11}_{0.26 1-12} F _{125 167}_{1.33 1-8} G _{60 30}_{0.51 1-5} H _{60 49}_{0.81 6-8} Average 57 49 0.78 ---

(33)

Hours of Operation and Activity Schedules

The after-school components of the programs included in the study operate on varying

schedules for an average of 11 hours per week, with a range of six to 15 hours of operation per week. Five programs are open five days a week for 2-3 hours each day. One program operates four days a week (M-Th) for two hours per day, although its activity schedule is organized into modules of activities offered twice a week (M/W or T/Th). The remaining two programs are open three days a week, for 2 to 2 ½ hours each day.

For the most part, youth in these programs have little choice about the activities in wh

Baltimore s After School Strategy. Assessing After-School Program Quality in Baltimore: Interim Findings of An In-Depth Study of Eight Programs