Roca s Performance Benchmark And Outcomes Report

Roca’s Performance Benchmark

And Outcomes Report

Fiscal Year 2012

July 2011 – June 2012

Organizational Overview ...1

Effort to Become High Performing... 1

FY12 Implementation Changes ... 2

Organizational Participant Enrollment ...4

Enrollment and Eligibility ... 4

FY12 Enrollments ... 5

Attrition Reasons ... 5

Demographics ... 5

Primary Target Population Performance Benchmarks ...6

Total Served ... 6

Attrition Reasons ... 7

Demographics ... 7

Risk Factors ... 8

Relentless Outreach ... 8

Transformational Relationships ... 9

Phase 1 Contact Standards ... 9

Phase 2 Contact Standards ... 10

Stage Based Programming ... 10

Programming Engagement ... 10

Programming Standards ... 11

Educational and Prevocational Programming ... 11

Work Force Readiness ... 12

Transitional Employment ... 13

Job Development and Placement ... 15

Reducing Arrests and Technical Violations ... 15

Phase 3 Intermediate Outcomes ...16

Lessons Learned and Next Steps ...17

Table of Contents

Efforts to Become High Performing

Roca’s journey to becoming a high performing, effective non-profit started many years ago when it began asking a critical question— Are we helping young people change their lives and how do we know? After realizing that, in spite of great dedication and hard work, Roca was not helping young people change behaviors to improve their lives to a significant degree; Roca took stock and rethought what it was doing. Roca understood that just creating a place for young people to belong or be engaged in activities wasn’t good enough. Rather, Roca realized that these young people deserved and needed the organization to get better at its mission to move them out of harm’s way and toward economic independence. Roca had to learn how to help them change and learn new behaviors.

In order to do this, Roca had to:

 Understand how proven theories, lessons, and skills could be applied and practiced with a group of young people most give up on;

 Study, adapt and implement elements of evidenced based programs and approaches based on foundational theories of behavior change;

 Research and study evidence-based practices and programs in an effort to more effectively serve our communities highest risk young men who without an intervention face a reality of jail, violence, and poverty;

 Work to align the Intervention Model and Roca’s organizational practice with the Eight Evidenced Based Principles of Effective Intervention1 identified forcommunity corrections;

 Build on its own direct experience of what worked and what did not to refine and develop new approaches to improve the likelihood of impacting the lives of our young people;

 Deepen its capacity to examine the ways in which it collected and used data regularly to continuously inform its strategic and operational goals; and,

 Be accountable to its young people and with full faith and transparency, figure out how to help them change their lives or shut down.

Roca engaged in a systemic cycle of research, design, action, tracking data, and use of data for continuous improvement to deliver an intervention worthy of the young people we serve. As a result of this learning and commitment to performance management and continuous improvement, Roca has made significant progress with the design and coaching of the model and the development of a performance management system.

Since 2005, when Roca went through its first Theory of Change Process, the following areas have been refined and clarified every year resulting in significant impacts on Roca’s learning and evaluation agenda.

 Clarity of the Model

 Moved from a multi-service youth development model to a single service intervention model designed to intentionally move a targeted group of very high risk young people to outcomes.

 Developed and operationalized a model based on the Transtheoretical Model/Stages of Change, Cognitive Behavioral Theory and Motivational Interviewing to create a framework for an entire experience, with multiple opportunities for young people to work on long term behavior change.

 Started looking at and using evidence based principles from the criminal justice system to help young people learn how to change their behavior.

 Performance Based Management

 Built a comprehensive system to track data and to use the data to inform the on-going continuous improvement of: the model, staff coaching and training, programmatic operations, and data collection and quality assurance practices.

1

Crime and Justice Institute (2004). Implementing evidence-based practice in community corrections: the principles of effective intervention.

Boston, MA: Author. Available online at: http://crjustice.org/cji/evidencebased.pdf.

Organizational Overview

 Developed an extensive training and supervision agenda to support and enhance the ongoing skills, knowledge, and capacity of staff to deliver the model as intended.

 Stage Based Programming

 Developed capacity to deliver programming that meets young people where they are in their readiness to engage and learn. Roca’s Stage-Based Programming was designed to engage young people where they are in the stages of change cognitively and behaviorally. The Stages of Change include: Pre-contemplation, Contemplation, Planning, Action, and Sustaining. The Stages of Change Process also recognizes Relapse as a fundamental part of the process of behavior change. Roca incorporated this fundamental principle of the Transtheoretical Model of Behavior Changes (Prochaska and DiClemente, 1984) into its programming – a “stage-matched” approach. This means that the activities or approaches are designed to match where each young adult is in the progression in order move them progressively through the stages of change toward successful outcomes.

 Trained Youth Workers to identify where the young person is in terms of readiness for change – not just once or at the beginning of the intervention but continuously throughout. This is done through the consistent youth worker participant interactions to observe language, attitudes, participation and other behaviors. This assessment of readiness informs the intensity (e.g., drop-in versus a 10-session class) of life skills, education and employment programming in which young people are engaged.

 Worked rigorously on building and evolving its workforce readiness and pre-vocational programming to create meaningful curriculum, skill building, and pathways that lead to real employment opportunities for the target population.

FY12 Implementation Changes

Model Timeline

In July 2011, Roca changed the length of the Intervention Model to a 2 year intensive model with 2 years of follow up. This change was made as a result of research on effective models in corrections and reviewing our data and discovering that young people who were in the model too long were beginning to do worse. We had to move them out of the model earlier for them to maintain their momentum for long term behavior change. Through further examination of the participant data, we studied the participants who were doing the best with employment and reduction in criminal involvement outcomes. On average, these participants had spent 24-28 months in Roca’s Intervention Model. Based on the data, we reduced the intensive portion of the model from 3 years to 2 years to increase the success rate of our young people.

Supervision Tools

In July 2011, Roca implemented new standardized supervision tools in Efforts to Outcomes. The new tools significantly increased our capacity to monitor the fidelity of implementation of the transformational relationships and to identify staff coaching and training needs

Primary Target Population

In November 2011, following our 3rd Theory of Change Process, Roca refined and narrowed the primary target population for the Intervention Model to focus on serving:

 Young Men;

 17-24 years old;

 Justice System Involved or Juvenile Justice Systems involved w/risk indicators predictive of adult criminal justice system involvement;

 No employment history; and,

 Live in contextually determined radius that promotes participation in center based services.

Roca’s choice to work with this primary population was based in part on the risk principle. According to the Crime and Justice Institute, “The risk principle identifies who to target for the most intensive of services and programming. Specifically, limited resources should be directed to those at highest risk for involvement in the criminal justice system. Further, higher risk clients should receive the greatest dosage of treatment and intervention. This principle is of critical importance given the serious

constraints of limited budgets and working with growing community supervision and prison and jail populations.”2 Additionally, in examining and learning about our work through the first replication in Springfield, which only serves young men, it became even clearer that focusing on this population increased the capacity to implement the Intervention Model and drive toward outcomes.

2

Bechtel, Kristin and Barbara A. Pierce, MPPM. “An Overview of What Works in Correctional Interventions.” Crime and Justice Institute (January 2011).

Roca knows that it can help these young men create the behavior changes critical to keeping them out of prison and engaged as members of our communities.

This change in target population required a re-structure of the Roca Chelsea Site to align with newly defined target population and accommodate its secondary population of young mothers. The change also initiated a process of completing young people out of Roca who had been engaged in Roca prior to the new model and who did not meet the target population. Roca also clarified its enrollment, intake, and eligibility practices to ensure that the young people to be served in the model meet the appropriate risk level. Finally, Roca clarified its benchmarks and outcomes for the target population. As indicated in the Participant Timeline below, Roca has become very clear on how to move its young people through the model in order to support behavior change, skill

acquisition, and the attainment of long term outcomes.

Secondary Target Population

In FY12 Roca also committed to continued service of a secondary population of high risk young mothers. While traditional services for young mothers exist within our communities, there are few, if any, programs designed to target very high risk young mothers who are involved in high risk behaviors and refuse voluntary services. Many young mothers are: street involved; gang involved; abusing drugs/alcohol; involved in domestic violence situations; involved in the juvenile justice system; involved with the foster care system; have dropped out of school; are unemployed; have been victims of abuse and neglect; struggle with trauma/mental health problems; have unstable living conditions; are isolated immigrants/refugees; are engaged in risky sexual behavior; and, are at high risk of secondary/tertiary pregnancy. This is sad for the young women themselves and often tragic for their children.

In Chelsea, Revere, and East Boston, Roca works with 200 young mothers a year. Roca operates the Harbor Area Healthy Families Program for 100 first time pregnant and parenting teens. Roca also works with a higher risk group of young mothers combining its Intervention Model with parenting education. Healthy Families, based on the national model of Healthy Families America, is a statewide program that is funded and administered by the Children's Trust Fund (CTF). The goals of the program are to: Reduce child abuse and neglect by supporting nurturing parenting; Achieve optimal health, growth, and development in infancy and early childhood; Promote educational attainment, job readiness, and life skills; Reduce repeat teen pregnancies; and Promote optimal parental health and wellness. Funded by the Massachusetts Pregnant and Parenting Teen Initiative (MPPTI), the Mass Department of Children and Families Young Parent Support Program, and several private foundation, Roca serves an additional 100 young mothers with a combination of its high risk intervention model, parenting education, and nursing support. Using effective engagement efforts, Roca successfully identifies, recruits, and retains very high risk young mothers for up to two years in the programming.

Page | 3

Page | 3

Enrollment and Eligibility Process

Young people are initially connected to Roca through various means. Youth Workers identify young people as potential participants through their intensive outreach and street work. Many young people are referred through partners including: local Police Departments, District and Superior Probation Departments, Courts, the Department of Youth Services, schools, health clinics, and other community based organizations.

When a young person is identified as a potential Roca participant, the young person is assigned to a Youth Worker who begins making contacts with the young person and gathering information to assess eligibility. From the first day of enrollment, Youth Workers have 60 days to determine the young person’s eligibility for Roca’s Intervention Model. Eligibility requirements are based on meeting the target population criteria. Additionally, Roca seeks to serve young people who are not able to engage in other programs, who are resistant to engagement and change, and who are in need of an intervention strategy. Young people are considered ineligible for the following reasons:

 they are not in need of an intensive intervention;

 they do not meet the target population criteria;

 they cannot participate in a cognitive behavioral model;

 they are incarcerated and cannot be accessed for contacts/services;3

 they moved out of the geographic area and cannot be accessed for contacts/services; and/or

 they are unable to be contacted/located.4

If it is determined that a young person is ineligible with in the first 60 days, they are removed from the Youth Worker’s caseload. Young people can only be removed from a caseload by a senior administrator. Therefore, a removal request has to be submitted with the ineligibility reason and it must be approved by the supervisor in order for the dismissal to take place. If a participant remains on a caseload after 60 days from enrollment, Roca considers itself accountable for driving to outcomes with the young person.

Given the change in Roca’s primary target population, eligibility criteria were aligned with the newly defined primary target population criteria beginning in November 2011. Any young person who did not meet the new criteria was considered ineligible for the Model. Additionally, Roca worked to identify participants who had been enrolled for a number of years (prior to the clear articulation of desired outcomes, benchmarks and timelines) and participants who no longer met the target population criteria. For this group of participants, completion plans were developed in order to set achievable goals and dismiss these

participants over a designated period of time. Some young people were able to be completed from the intervention as a result of attaining outcomes; however, a majority were dismissed as unsuccessful and were provided with support and referrals to other services as appropriate.

3 _{These are young people are typically referred while incarcerated who do not have release dates or are serving a long}

term sentence. They are eligible for services when they are released.

4 _{These are young people for whom we are unable to obtain correct addresses, telephone numbers or other identifying}

information that we are able to use to locate the young person

Highlights

In FY12, participants included 4 categories of young people:

 Young men who met

Roca’s newly defined primary target population criteria

 Young mothers

 Young women who met

Roca’s previous high risk target population definition

 Young men who met

Roca’s previous high risk target population definition

In FY13, Roca will only be working with:

 Young men who met

Roca’s newly defined primary target population criteria

 Young Mothers

Given Roca’s focus on

demonstrating impact with the newly defined primary target population, the later part of this report is focused on studying the benchmarks and progress for young men who met Roca’s newly defined primary target population criteria.

Page | 4

Page | 4

Organizational Participant Enrollment

FY12 Enrollments

In FY12, Roca enrolled a total of 1120 young people. This includes both young people already enrolled at the start of the fiscal year, as well as those who enrolled during FY12.

 888 young people were enrolled in Roca Chelsea

 232 young people were enrolled in Roca Springfield

The chart below displays the participants’ enrollment status on June 30, 2012, the last day of the fiscal year. In FY12 Roca actively worked to dismiss young people who no longer fit the newly defined primary and secondary target populations throughout the year. Those that were active remained engaged in Roca programs while those who had been completed, dismissed, or determined as ineligible were no longer engaged in Roca.

Participant Attrition Reasons (Ineligible and Dismissed)

Participant Demographics

Roca Roca

Gender City

Male 62% Chelsea 46%

Female 38% East Boston/Boston 17%

Age Range Revere 9%

16 Years Old and Younger 14% Springfield 21%

17 - 18 Years Old 25% Other 7%

19 - 21 Years Old 38% Race

22 - 24 Years Old 19% African American 15% 25 Years Old and Older 1% Asian 1%

Unknown 2% Caucasian 8% Hispanic 63% Bi/Multi-Racial 4% Other 10% 17% 18% 8% 56% 0% 10% 20% 30% 40% 50% 60% Ineligible Dismissed Completed Active Enrollment Status 6% 9% 10% 14% 25% 36% 0% 5% 10% 15% 20% 25% 30% 35% 40%

Cannot participate in cognitive intervention model Incarcerated for extended time period Does not need Transformational Relationships Not in geographice service area Unable to find/locate Not in target population

Reasons for Ineligiblity

9% 9% 14% 17% 22% 28% 0% 5% 10% 15% 20% 25% 30%

Unable to move to Phase 2 Moved out of geographic service area Incarcerated for extended time period Unable to contact / locate Failed to achieve completion plan requirments Unable to move to Phase 3

Reasons for Dismissal

Highlights

Of the 1120 young people enrolled:  631 remain actively engaged  202 were dismissed  194 were assessed as ineligible

 93 completed goals and/or benchmarks related to Pre-FY12 programming and follow up benchmarks

(but no longer being continued in the model or tracked for outcomes)

Program Retention

Roca’s overall retention rate is 65% of all participants enrolled

(this includes those who were assessed as ineligible for the model within the first 60 days of enrollment).

Of those who were

considered eligible for the

model and services at

Roca, Roca’s retention

rate is 78%.

With Ineligibles Without Ineligibles

Page | 5

Page | 5

As Roca is committed to demonstrating impact with its newly defined primary target population, the following information expands on Roca’s work with young men, ages 17-24 who are street/gang involved and/or who have been involved with the criminal justice system. This data represents Roca’s first effort at ONLY benchmarking this target population of young men to the performance benchmarks and outcomes

In order to provide a clearer picture of how the changes made to the model in FY12 have impacted delivery and to begin looking at how to benchmark annual progress of participants through the model, this year’s data breaks out participants who were enrolled prior to the start of FY12 and those enrolled during FY12 at each of the sites, Chelsea and Springfield. As mentioned earlier, those enrolled prior to the start of FY12 had originally been on a 3year timeline for intensive services. Given the change in the timeline to a 2 year intensive at the beginning of FY12, participants were worked to the new timeline benchmarks despite having started under a different initial model.

Additionally, this year, new enrollments were particularly high. The change to the 2 year timeline and the refinement of the target population also required a process of completing young people out of Roca who had been engaged in Roca for more than 3 years and who did not meet the newly defined target population. As a result, Roca was able to accommodate new enrollments for the newly defined primary target population.

In FY12, Roca served a total of 409 young men who met these target population criteria.

(Please Note: this number does not include those young men who were enrolled but determined ineligible for not meeting target population risk level within the first 60 days of enrollment)

Of the 409 young men served:

 178 had been enrolled prior to the start of FY12

 231 were new enrollments in FY12

Highlights

Roca Chelsea served 273 young men

 129 had been enrolled prior to the start of FY12

 144 were new

enrollments in FY12

Roca Springfield served 136 young men

 49 had been enrolled prior to the start of FY12

 87 were new enrollments in FY12

Of the 409 young men:

 298 remain actively engaged

 111 were dismissed

Of the young men in

Roca’s primary target

population who were

considered eligible for

Roca’s Model, Roca’s

retention rate is 73%.

Page | 6

Page | 6

Primary Target Population Performance Benchmarks

Participant Attrition Reasons

As indicated, the primary reason for dismissal is “Incarceration for an Extended Period of Time”. Often the young men Roca serves have pending trials and or suspended sentences when they are initially enrolled.

If a young person is convicted and/or has a suspended sentence imposed for a violation that exceeds 18-24months, Roca removes them from the caseload as the participant is no longer accessible for the intervention. However, these young men are still considered eligible and may be re-enrolled when they return to the community.

Participant Demographics

Roca Chelsea Roca Springfield Roca

Gender

Male 100% 100% 100%

Female N/A N/A N/A

Age Range 17 – 18 Years Old 32% 18% 27% 19 – 21 Years Old 47% 46% 46% 22 – 24 Years Old 22% 37% 27% City Chelsea 53% N/A 36%

East Boston/Boston 22% N/A 15%

Revere 10% N/A 7% Springfield N/A 100% 33% Other 14% 0% 10% Race African American 11% 46% 22% Asians 2% 0% 1% Caucasians 9% 2% 7% Hispanics 68% 40% 59% Bi/Multi-Racial 6% 4% 5% Other 4% 8% 6% 6% 11% 23% 23% 38% 0% 5% 10% 15% 20% 25% 30% 35% 40%

Unable to contact / locate Unable to move to Phase 3 Moved out of geographic service area Unable to move to Phase 2 Incarcerated for extended time period

Reasons for Dismissal

Worth Knowing 84% _Percent of New Enrollments Retained.

Page | 7

Page | 7

Highlights

Roca Chelsea Race

Roca Springfield Race

Chelsea, 36% Springfield, 33% E. Boston/ Boston, 15% Other, 10% Revere, 7% Hispanic, 68% Caucasian, 68% Asian, 2% African American, 11% Other, 4% Bi/Multi- Racial, 6% African American, 46% Hispanic, 40% Caucasian, 2% Other, 8% Bi/Multi- Racial, 4%

Page | 7

Page | 7

Risk Factors at Enrollment

Launched in July 2010, the Springfield Replication had a very focused criteria for enrollment specifically, young men who were involved in the criminal justice system.

Roca Chelsea Roca Springfield Roca

Risky/Anti-Social Behavior

Street/Gang Involved 86% 96% 89%

Criminal Justice Systems Involvement

Prior Arrests 83% 96% 88%

Prior Convictions 59% 88% 69%

Prior Adult Incarcerations 28% 56% 37%

Prior Adult Supervisions 44% 55% 48%

Prior Juvenile Supervisions 34% 32% 33%

Education Status

High School Drop Out – Has GED 5% 10% 7%

High School Drop Out – Needs GED 69% 81% 73%

Employment History

No Employment – Past 6 Months 84% 95% 88%

Unemployed 84% 96% 88%

Young Parent

Young Father 16% 42% 25%

Given the focus of the primary target population and enrollment criteria for Chelsea, we expect to see similar demographic patterns for Chelsea in the coming year as new participants must meet a certain level of criminal justice system involvement.

Relentless Outreach

The young men Roca targets for its Intervention Model do not easily show up and are often unwilling to engage. They are young men who are still actively involved in criminal and street behaviors and as such are often evasive and try to make themselves hard to find. Relentless outreach is Roca’s strategy to find them, engage them, and to stay in it with them for the long run. In the very beginning, when a youth worker is trying to build a relationship with a young person, outreach efforts far exceed actual contacts. Youth workers go to their homes, talk with parents, friends, associates, go to known hot spots, parks, and corners all in the effort of finding the young person and demonstrating the consistency of their effort and commitment to engage with him.

Once the relationship has begun to develop, outreach is still the critical tool for youth workers to build trust, reliability and consistency. Young People need to see that the youth workers are coming to find them when they don’t show up and are going to look for them when something is going on in the street in order to make sure they are safe. Outreach is not something that happens in the beginning of the relationship, but all throughout their engagement with Roca.

The most critical time for outreach is actually further into the relationship and after the young person has been engaged in his process of change. It is in the moments of relapse, when the young person blows up, gives up, and doesn’t want anything to do with Roca anymore that outreach becomes the most primary and crucial strategy employed by youth workers. Youth Workers have to do whatever it takes to find our young people and help them realize that even in the hardest of moments, we are not going anywhere.

Highlights

As seen in the enrollment demographics, Springfield enrollments had higher rates of involvement in the criminal justice system.

Criminal Justice Systems Involvement 32% 55% 56% 88% 96% 34% 44% 28% 59% 83% 0% 50% 100%

Roca Chelsea Roca Springfield

Prior Arrests

Prior Convictions

Prior Adult Incarcerations

Prior Adult Supervisions

Prior Juvenile Supervisions

Page | 8

Page | 8

PE = Pre-FY12 Enrollments NE = New FY12 Enrollments

AE = All Enrollments

Transformational Relationships

Roca understands relationships as the foundational component to supporting young people’s change. The development of the transformational relationship component and the emphasis on how Roca staff members engage in building trust aligns with the research on the importance of relationships related to behavior change. The young people in Roca’s target population have formed norms and values that have mostly been influenced through unhealthy and often harmful experiences in and of relationships. The young adults have had years of negative reinforcements. Roca attempts to counter the years of negative

influences, to build positive norm and value influencing relationships, and to engage young people in the process of transformation through the strategic and intentional

implementation of unremitting outreach and transformational relationships.

Because high-risk young people have had little or no experience with adults who are both non-judgmental and have high expectations of healthy behavior, they are prone to frequent disengagement and rejection of constructive relationships. Therefore, a youth worker must ceaselessly reconnect with a young person who periodically rejects him or her and refuses to engage. This pattern is crucial to the process and must happen continuously all throughout the relationship. While a young person may show up at Roca three times in one week, he or she may just as likely refuse to come back the following week. It is the youth worker’s responsibility to track down that young person and reengage. Because the relationships is the critical tool used for change, the youth worker’s need to effectively know how to use himself or herself to engage young people and build trust is essential.

Through evaluation studies conducted on Roca’s Model, the single most predictive indicator of retention and attainment of long term outcomes is young people receiving 2-3 contacts each week. Youth Workers are expected to make 2-3 actual contacts each week with all 25 of the young people on their caseloads. These contacts are intentional interactions with the participant by the youth worker to build and deepen relationships and to push, encourage, challenge, and support behavior change. Actual Contacts do not include contacts with any third parties, i.e. parents, girlfriends, probation officers, etc… Additionally, actual contact percentages do not include efforts made by the youth worker that did not result in a face to face or a verbal conversation with the participant.

Of the 409 participants served in the Primary Population:

 241 are Phase 1 Participants

 129 are Phase 2 Participants

 39 are Phase 3 Participants

Phase 1 Contact Standards

Due to the very nature the target population recruited, new Phase 1 participants are expected to be harder to contact. Initial engagement in the Phase 1 requires a great deal of effort on the part of the youth worker which doesn’t always result in an actual contact with the young person. This year, with the high volume of brand new enrollments in the Phase 1, youth workers faced additional challenges making 2-3 actual contacts each week with new participants. Quite often, when a young person was initially enrolled on a youth workers caseload, there were missing or incorrect addresses and phone numbers. Roca relied heavily on its referring partners to help gather information that could be used by youth workers to track these newly enrolled young people down.

While new enrollments in FY12 were managed more efficiently to the new program

timelines, there were challenges related to moving those previously enrolled to a Phase 2 in

Worth Knowing

Page | 9

Page | 9

Highlights

Roca categorizes where a young person is in the model by his Phase status.

Phase 1: Engagement

Phase 2: Behavior Change

Phase 3: Sustaining

benchmarks and outcomes and are being supported through less intensive services on sustaining and advancing those outcomes.

49% of Phase 1

Participants were seen

2-3 times weekly

Page | 9

Page | 9

PE = Pre-FY12 Enrollments NE = New FY12 Enrollments

AE = All Enrollments

a timely manner. The goal is to move Phase 1 participants to a Phase 2 between 4-6months. For participants previously enrolled, while some had not yet hit their 6 moth benchmark, others were not moved to a Phase 2 as a result of not managing the length enrolled and administrative actions required to move to a Phase 2. Compliance reports were created at the end of FY12 to track and identify potential management issues related to the movement of young people through the model.

Phase 2 Contact Standards

With the establishment of a relationship, the percentage of young people who youth workers are able to make actual contacts with increases. Phase 2 participants were more likely to meet the standard of being seen 2-3 times each week.

New enrollments had higher rates of contacts than those enrolled prior to FY12

Stage Based Programming

Roca’s stage based programming approach aligns well with what the literature suggests is the most effective practice in creating opportunities for individuals to learn and practice new behaviors. When young people are attached to programming that “meets them where they are” in their readiness, willingness, and ability to engage, and that allows for them to practice and experience their own challenges and successes, the level of participation and potential success increases. By applying this evidence-based framework in the program delivery, young people move along a pathway of education, pre-vocational training, life skills, transitional employment and unsubsidized employment opportunities.

Programming Engagement

Getting this group of young people into programming, while challenging, is critical to supporting young people in their change process and their skill attainment. Youth workers try to drag young people into programming early on in their enrollment in the model to expose young people to the learning possibilities and to help them begin the practice of showing up.

Percentage Engaged in Stage Based Programming

PE = Pre-FY12 Enrollments NE = New FY12 Enrollments AE = All Enrollments

Highlights

Phase 1 and 2 Contact Standards Comparison

67% of all Phase 2

Participants were seen

2-3 times weekly

80% of all participants

were engaged in stage

based programming

Programming Engagement

PE = Pre-FY12 Enrollments NE = New FY12 Enrollments AE = All Enrollments 49% 67% 0% 10% 20% 30% 40% 50% 60% 70% 80% Phase 1 Phase 2 83% 78% 80% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% PE NE AE

Page | 10

Page | 10

Programming Standards

Ideally, regular attendance to programming builds as young people become more engaged in their own change and learning processes. Roca is studying the patterns of participation for Phase 1 and Phase 2 participants to understand both what it takes to get young people into programming and what the expected level of attendance should be going forward for this target population. As we can already see from the data below, new enrollments are attending at a higher rate than those who were enrolled previously to FY12. With the clarity of the model and the new timeline initiated in FY12, youth workers were more intentional about getting new enrollments into programming at a faster pace earlier in their enrollment.

Education and Pre-Vocational Programming

Our educational programming helps young people re-engage and advance in education, and either re-enter school or obtain their GED. Roca offers instruction on basic literacy, pre-GED, GED, Microsoft applications, writing, and tutoring/homework help. Roca helps young people whose first language is not English to improve proficiency in literacy and communication skills through multi-level ESL classes.

Pre-Vocational Training provides young people with the basic job skills and certifications needed to enter the workforce and sustain employment. Young people participate in on-site training courses to earn other informal/formal industry certifications including: OSHA, CPR, Heartsaver AED, Lead Safe Removal, Carpet Care, Floor Care, ServSafe, forklift operation, green cleaning, housekeeping and more. We also offer apprenticeships, such an

apprenticeship with the Carpenters' Union. Soft skills are developed in resume and interview training, job readiness and career exploration.

PE = Pre-FY12 Enrollments NE = New FY12 Enrollments AE = All Enrollments

PE = Pre-FY12 Enrollments NE = New FY12 Enrollments AE = All Enrollments

Worth Knowing

Page | 11

Page | 11

Highlights

More than half of all the

Phase 2 participants

attended programming at

least 1-2 times weekly

67% of all participants

were engaged in

educational and

pre-vocational programming

52% of those engaged in

educational and

prevocational

programming made gains

Page | 11

Page | 11

Industry Recognized Certification Breakdown Roca Chelsea Roca Springfield Roca GED 4 5 9 Basic Cleaning 0 24 24 Carpet Care 0 14 14 CPR 17 10 27 First Aid 0 21 21 Floor Care 0 13 13

Fork Lift Training 45 0 45

Green Cleaning 0 7 7

Lead Safe Removal 16 0 16

OSHA 10 - Construction 16 0 16

OSHA 10 - General Industry 16 0 16

OSHA 30 4 0 4

OSHA Asbestos 16 0 16

OSHA Blood Bourne Pathogen 13 0 13

OSHA Hazard Communication 0 20 20

Restroom Sanitation 0 18 18

ServSafe 2 0 2

Slip and Fall Prevention 0 14 14

Wood Floor Care 0 9 9

To improve the quality and impact of our pre-vocational training, Roca developed four new 30-hour pre-vocational curriculums in construction, hospitality, retail and culinary arts. These courses will be launched in FY13 , and require full attendance. We customized our hospitality training based on Marriott's employment guidelines, and Whole Foods assisted with the design of our culinary arts curriculum. These new courses will be delivered over 10 week periods and will help young people prepare to enter transitional employment, build a resume and increase their employability

Work Force Readiness

Getting this group of young men ready for work is not as easy as just placing them in a job. In order to be successful at work, there are steps that have to happen, documents that have to be obtained, skills that have to be learned, and behaviors that have to be practiced and mastered. In an effort to more appropriately assess young people’s workforce readiness, Roca developed a Workforce Readiness Criteria that includes four domains young people have to complete to be considered workforce ready.

 Domain 1 is based on a checklist of basic needs that must be completed including: Photo ID, Social Security Card, Birth Certificates, Resumes, Transportation Plans and Childcare Plans.

 Domain 2 requires participants to complete 12 workforce readiness workshops on topics that include: Sexual Harassment, Workplace Communications, Mock Interviews, Understanding CORI, and Conflict Management.

 Domain 3 covers demonstrated skills and behaviors that are observed and assessed by crew supervisors, youth workers, and vocational instructors. Skills and behaviors being assessed include: being on time, appropriate language, team work, following directions, positive attitude, and personal hygiene. Participants need a minimum of 6 Assessments and to consistently receive a rating of always or often to be

considered for job placement.

Highlights

121 young people attained at least one industry recognized certification

148 young men were engaged in Workforce Readiness Workshops

102 participants had a minimum of 6 Demonstrated Skills and Behaviors

Assessments 55% 25% 20% 0% 10% 20% 30% 40% 50% 60% 70% 80% 1-3 Workshops 4-7 Workshops 8-12 Workshops 11% 16% 54% 0% 10% 20% 30% 40% 50% 60% 70% 80%

Page | 12

Page | 12

Highlights

Roca served 172

unduplicated participants

in Transitional

Employment.



161 were served in

Basic Transitional

Employment



31 were served in

Advance

Transitional

Employment

Page | 13

Page | 13

Page | 13

Page | 13

 Domain 4 tracks concrete benchmarks of achievement including: successfully completing transitional employment, meeting program attendance standards and completing a minimum of 2 industry recognized vocational certifications

Transitional Employment

Roca operates basic and advanced transitional employment slots designed to support the development and practice of workforce behaviors necessary for job placement and retention.

The flow chart below illustrates the progression of a young person through transitional employment and job placement services at Roca.

Basic Transitional Employment

Basic Transitional Employment was designed to help young people learn and practice how to work by working. This includes the development of basic soft skills necessary for successfully retaining employment, including: showing up on time, completing an entire work day, following directions, workplace communications, etc… In the process of learning how to go to work, young people may lose their transitional work slot multiple times. Each time a young person loses their job, they must undergo a rehire process that moves them toward positive behavioral change. Based on the new timeline of the model, young people can be in basic transitional employment for a maximum of 18 months from the day they first begin on a work slot.

Successful completion of basic transitional employment for participants is defined by reaching 60 days of consecutive employment without losing their work slot or missing work without cause. Once participants meet this successful completion benchmark, they will either be referred to Advanced Transitional Employment or to job placement. This referral is based on their progression not only through the work experience but also through other parts of the model.

Transitional Employment Flow Chart

Unsubsidized Job Placement

Job Retention ATEP Slot Work Readiness

Terminated

Terminated

Orientation

BTEP Bench

BTEP Active Slot Work Readiness Re-Hire

Meet WR Benchmarks

No Yes

BTEP = Basic Transitional Employment ATEP = Advanced Transitional Employment WR = Work Readiness

The following benchmark information is based on the 161 young men who were served in Basic Transitional Employment in FY12 only (these benchmarks do not include placement numbers for those young people who completed basic transitional employment prior to FY12 who were being served in advanced transitional employment or job placement only in FY12).

Advanced Transitional Employment

Advanced transitional employment was designed to support young people who have completed basic transitional employment but need more time to build overall workforce readiness skills before being placed in unsubsidized employment.

Successful completion of advanced transitional employment for participants is defined by continuous retention of their advanced transitional employment slot and the completion of all workforce readiness criteria.

The following benchmark information is based on the 31 young men who were served in Advanced Transitional Employment in FY12 only (these benchmarks do not include placement numbers for those young people who completed advanced transitional employment prior to FY12 who were being job placement only in FY12).

Dismissed from Roca Model

 Removed from Youth Worker caseloads and no longer engaged in Roca’s Model 25%

Could Not Complete

 Met maximum allowed length enrolled in advanced transitional employment 75% 8 Unsuccessfully Completed 8 Actively Enrolled 31 P arti ci pant s S e rv e d i n A dv ance T ra ns it ional Em pl o ym e nt

Placed in Unsubsidized Employment 80% Pending Placement 20% 15 Successfully Completed

Dismissed from Roca Model

 Removed from Youth Worker caseloads and no longer engaged in Roca’s Model 57%

Could Not Complete

 Met maximum allowed length enrolled in basic transitional employment 43%

42 Unsuccessfully

Completed

Placed in Advanced Transitional Employment 64%

Placed in Unsubsidized Employment 24% Pending Placement 12% 25 Successfully Completed Active Slot:

 Currently filling a slot on a work crew 31%

Bench Slot

 Eligible to fill a slot on a work crew 21%

Re-Hire Slot

 Working towards becoming eligible for slot on a work crew after losing a slot 48% 94 Actively Enrolled 161 P arti ci pant s S e rv e d i n B asi c T ran si ti onal Em pl o ym e nt

Highlights

Basic Transitional

Employment

Of those who successfully

completed Basic Transitional

Employment:

86% have retained

employment

Advanced Transitional

Employment

Of those who successfully

completed Advanced

Transitional Employment:

75% have retained

employment

Page | 14

Job Development and Placement

Given the changes in the model this year, the following placement and retention benchmarks for this report are only for the primary target population young men who were placed in unsubsidized employment in FY12.

There a total of 63 job placements made. Of those job placements, 51 were unduplicated participants.

 67% were still employed at the end of the reporting period

 33% are working towards replacement

Reducing Arrests and Technical Violations

As one of the long term outcomes for participants is no incarcerations, Roca tracks

performance indicators for arrests and technical violations for throughout their engagement in the model. While it is expected that young people may have new arrests and technical violations while in the model, Roca hopes to see a reduction over time in both areas for individual participants as they move through the model. The impact and benefits of the new model rolled out in FY12 can also be seen in the positive progress of new enrollments who have less arrests and violations than those enrolled prior to FY12

PE = Pre-FY12 Enrollments NE = New FY12 Enrollments AE = All Enrollments

PE = Pre-FY12 Enrollments NE = New FY12 Enrollments AE = All Enrollments

Highlights

Overall Job Placement

67% of young people

placed in unsubsidized

employment in FY12 were

still working at the end of

the fiscal year.

Reducing New Arrests and

Technical Violations

Page | 15

In FY12, 39 primary population participants graduated to a Phase 3. Phase 3 is the 2 year less intensive follow up phase of Roca’s Intervention Model. Young people are graduated to a Phase 3 when they have completed the intensive 2 years of the model and they have meet the intermediate outcomes of: Unsubsidized Employment Retention (min of 3 months); Educational Gains; No New Arrests (min 6 mo); and, No Technical Violations (min of 6 mo). Young people are tracked in a Phase 3 for 2 years. They are provided with less intensive support through contacts with a TR3 Youth Worker, are able to continue participating in educational and pre-vocational programming, and are assisted with job development and advancement support.

By definition, those who graduated to Phase 3 came from the population already in progress at the start of FY2012, as they had to have completed 2 years of programming. All of the Phase 3 participants were served in the Roca Chelsea Site as Roca Springfield Replication just completed 2 years and will not have TR3 graduates until FY13.

Of those who graduated to a Phase 3 in FY12:

79% 70% 90% 100% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Employment Retention Education Gains No New Arrests No New Technical Violations Year 1 Intermediate Outcomes

Page | 16

Page | 16

Phase 3 Intermediate Outcomes

It has been an intensive year of transition as a result of implementing changes to the model, target population, and program operations. These changes have provided clarity and structure in driving young people toward outcomes, in coaching and holding staff accountable to performance indicators and standards, and in developing improved operations, systems, and measurements to improve our capacity to deliver a high quality, replicable model of intervention with this target population.

As evidenced in the data, overall, new enrollments in FY12 are doing better than those enrolled prior to FY12. These new

enrollments have been managed more efficiently to the timeline and benchmarks of the model. Also, from the beginning of their engagement, they have been able to participate in programming that is now strategically designed to move our young people through the model, incrementally working on behavior change and building their capacity to participate in more structured opportunities for skill development directly linked to the attainment of short, intermediate, and long term outcomes.

Becoming worthy enough to serve this group of young men and to help them make real changes in their behaviors is intricately connected to our on-going engagement in a process of continuous quality improvement. By working systemically and strategically to improve our programming and organizational structures we have been able to increase our ability to have a meaningful impact and to be of value to those we serve. Completing participants who do not meet the target population, improving the ways in which we measure ourselves, making adjustments to track performance, using data to manage performance, and focusing on staff training and development has allowed Roca to get to more clear on the outcomes we care about with the group that is most in need of this level of intervention.

With all of the changes implemented in FY12, Roca is now poised for the next level of work to fully align itself with Eight Evidenced Based Principles of Effective Intervention and to move toward demonstrating the impact of its model through more rigorous evaluation efforts. To do this, Roca will continue to look at critical look at current research and will specifically be looking at: a deeper understanding and potential implementation of risk/needs assessment; specific Cognitive Behavioral Programs that may enhance the model, and; research and consideration of sanctions as informed by the current thinking in evidenced based practices. Roca will also continue to use its data to ensure fidelity and to improve programmatic and participant benchmark targets.

Roca’s 3 main Implementation Goals include:

 Increasing contact standards to 80%;

 Increasing programming standards to 80%; and,

 Improving overall job placement and retention rates.

FY13 Targets and Benchmarks:

 Roca Chelsea will serve 470 (350 TR1 & TR2; 120 TR3) high risk young men

Actual Starting point as of 7/1/12- Roca Chelsea will serve 420 (300 TR1 & TR2; 120 TR3)

 Roca Springfield will serve 180 (150 TR1 & TR2; 30 TR3) high risk young men

Actual Starting point as of 7/1/12- Roca Springfield will serve 130 (100 TR1 & TR2; 30 TR3)  80% will be seen a minimum of 2-3x a week by their youth worker

 80% will meet programming standard of 2x per week in any programming

 80% will be actively engaged in alternative education and workforce readiness programming  40% will be engaged in Transitional Employment Programming

Of those engaged:

 80% will complete workforce readiness criteria (within 18 months from enrollment)

 80% will attain a minimum of 2 industry recognized vocational certifications (within18 months from enrollment)  80% will demonstrate academic gains (within 24 months from enrollment)

 80% will be placed in unsubsidized employment (within 24 months from enrollment)  80% of those placed will retain employment for a minimum of 3 months

Lessons Learned and Next Steps