A PRACTICAL GUIDE FOR PROGRAM ASSESSMENT AND ACTION PLANNING

(1)

YOUTH AND ADULT LEADERS FOR

PROGRAM EXCELLENCE:

A PRACTICAL GUIDE FOR

PROGRAM ASSESSMENT

AND

ACTION PLANNING

• C

AMINO

& A

SSOCIATES

•

(2)

Youth and Adult Leaders for Program Excellence:

A Practical Guide for Program Assessment and Action Planning

By Linda Camino, Shepherd Zeldin, Carrie Mook, and Cailin O’Connor

Collaborators

The lead funder for this project is the Surdna Foundation, Inc. We graciously thank Robert Sherman for his critical perspective and collegial support throughout all aspects of this work.

Generous support was also received from the Upstate Center of Excellence and ACT for Youth, both located at Cornell University. We are appreciative of Jane Powers, Director of ACT for Youth, for her ongoing support and for feedback on the many drafts of this resource kit.

We thank our colleagues in Milwaukee. The United Way of Greater Milwaukee provided initial funding for this project. We are thankful to Beverly Njuguna for bringing together groups of youth and practitioners to discuss youth participation, and for believing that organizational assessment could help enhance partnerships. Quinn Wilder, from the Youth Care Learning Center, was a skilled and collegial collaborator throughout the pilot test phase of this project.

A special thanks to Wisconsin Extension, especially Greg Hutchins, Program Leader of 4-H Youth Development, for being an early adopter of our program assessment and reflection methods, and Matthew Calvert, 4-H Youth Development Specialist, for his contributions to initial drafts of the YET and ORG-YET tools.

We are grateful to the many youth and program staff who participated in the field tests in New York and Wisconsin. Their observations, critiques, and tips have strengthened this guide in substantial ways.

• C

AMINO

& A

SSOCIATES

•

(3)

Youth and Adult Leaders for Program Excellence

Permission required to reproduce or distribute this material: [email protected] or [email protected]

INTRO:

Overview,

Frequently Asked

Questions, and

Outline of Steps

(5)

Overview

Youth and Adult Leaders for Program Excellence is a resource kit designed to help youth-serving organizations enhance the quality of their programs. Specifically, the resource kit is designed to:

Strengthen the quality of youth-adult partnerships and youth voice in program decision- making, and

Strengthen the quality of developmental opportunities, relationships, and emotional supports offered to program participants.

Organizations can focus on some or all of these aspects of program quality, depending on their interest and need. Staff and youth can also adapt the assessment process to explore additional issues.

Youth and Adult Leaders for Program Excellence provides a structured, easy-to-use strategy for

organizations to conduct a rigorous assessment of their programs. Equally important, the resource kit guides organizations through the process of using the results of the assessment for program planning and improvement. And finally, the resource kit details ways that organizations can use the findings to help gain support from key constituencies, such as funders and community leaders.

This resource kit guides staff and youth through five phases of assessment and program improvement:

Phase 1: Planning and preparing to conduct a program assessment Phase 2: Collecting and compiling data

Phase 3: Analyzing and understanding the data Phase 4: Sharing results with the group

Phase 5: Action planning and finalizing the report

The resource kit is designed to allow organizations to choose their own assessment. The four assessment tools included in the resource kit focus on:

the extent to which youth are involved in the design and implementation of programming, the extent to which youth have a voice in organizational governance, and

the provision of developmental opportunities and supports in program activities.

Programs can focus on one or more of these aspects of program quality, depending on their interest and need. Staff and youth are also encouraged to add their own questions to the assessment

instruments.

(6)

The methods in this resource kit are designed for staff and youth across a full range of contexts, from after-school programs, to community-based organizations, to residential settings. It is most appropriate for programs that are actively seeking to improve their services, and which perceive youth as being key actors in that process.

Benefits to youth and adults

who lead the assessment and action planning process

Conducting a rigorous assessment and engaging in program improvement can be time-consuming.

However, the benefits are substantial when the assessment is done well. Following the steps in this resource kit will not only help to strengthen youth programming within an organization – the process of assessment and action planning is also a developmental opportunity itself. Both youth and adults who serve as members of the Assessment Leadership Team will benefit from their involvement.

Youth will develop skills in the practical tasks of research and evaluation, such as choosing and customizing an assessment tool, entering data into a database, analyzing survey results, and writing an assessment report. They will also practice public speaking and group facilitation, and develop supportive relationships as they work with a group of their peers and adults. Youth who play active roles in this type of process tend to show increases in self-confidence, social competencies, and community engagement. Young people’s identity development is bolstered as they try out new roles and gain a sense of their own abilities.

While the practical tasks of research and evaluation may not be as new to participating adults as they are to youth, adult members of the Assessment Leadership Team will also benefit from their

involvement in the assessment. Adults often report feeling an increased sense of efficacy in their work and connection to their organizations after working closely with youth on an evaluation project. The partnership model used in the Youth and Adult Leaders for Program Excellence process improves adults’ confidence in youth and attitudes about young people’s abilities – even among adults who work with youth every day.

This resource kit provides the tools to reap these and other benefits – to improve youth

programming, strengthen organizations, and provide developmental opportunities for the young people and adults who work together on the Assessment Leadership Team. The time and energy it takes are worth it.

(7)

Frequently Asked Questions

We have worked with a variety of youth-serving organizations on issues of program assessment and improvement using the tools and methods included in this resource kit. Listed below are some questions we’ve been asked by organization leaders, program staff, and youth – and our answers to them.

Organization Leaders

Why is it important to engage in organizational self-assessment?

Youth programs are almost always evaluated and judged by people outside the organization – funders, community leaders, politicians. There are advantages to this, of course. But the downside is that the people who operate and attend the programs are rarely involved in program assessments.

Their voices are not heard and their expertise is not utilized. This resource kit is designed so that staff and participants can assess the quality of their own programming, and equally important, take steps to improve it. It puts ownership of the process and the action steps in the hands of the people who have the greatest stake in the outcome.

Does this organizational assessment and action planning process really work?

Youth and Adult Leaders for Program Excellence is based on extensive field tests with a variety of youth serving organizations; for example, a drop-in center, a shelter for homeless and runaway youth, a youth activism program, a youth philanthropy program, a community coalition, a large high school, and a 4H community club.

Each time that we worked with a program, we learned something new about how to improve the process and how to work out the bugs. During the development phrase of this resource kit, we constantly asked staff and youth what worked for them and what did not. Over time, we have deleted that which did not work well, and have strengthened those aspects of the process that work the best. We are now confident that this resource kit reflects “best practice” in the eyes of youth workers and youth, and is consistent with the lessons of research.

Those organizations that were able to take the time to go through each of the phases and steps of Youth and Adult Leaders for Program Excellence were the ones that benefited the most. It is hard for organizations to find the time to devote to assessment and action planning, but those organizations that allocated the time and kept at it were the ones that gained the most tangible results. Throughout this resource kit, we identify proven strategies and tips for success. If you use these tips, and your own creativity, we think that you will be pleased with the results.

What types of programs and ages of youth is this resource kit designed for?

We generally think of “youth” as people between the ages of 12 and 18, although this resource kit has been used successfully with groups serving both younger and older participants. Any program that aims to promote positive youth development would be able to conduct a Youth and Adult Leaders for Program Excellence assessment, regardless of the setting or their program activities.

(8)

Can you provide other materials to support the process?

Using youth-adult partnerships for program assessment and change is a relatively new innovation in the United States. Many stakeholders – funders, boards of directors, local politicians – are therefore hesitant to endorse the strategy. We are preparing research briefs to provide research-based

information to these constituencies on the benefits of this strategy. These briefs, available in early 2005, will also list a range of useful web-sites.

It is, of course, important to involve parents to the extent possible. When young people are involved in collaborating with staff and are also involved in making decisions with their families, positive developmental outcomes are likely. Therefore, we have written a series of four newsletters for parents who wish to engage youth in family decision-making. The newsletters offer practical advice and tips for parents that have been gleaned from interviews with youth and their parents and from the research literature. These newsletters, called “Involving your teen in decision-making,” are available on-line at http://www.uwex.edu/ces/4h/staff.html or

http://www.actforyouth.net/publications.asp?type=Newsletters.

Program Staff

My program has never done an assessment before. For example, we’ve never collected and interpreted data, and don’t really know how to write assessment reports. How can we do this?

This is not a problem. This resource kit is user-friendly. We do not assume that either the adults or the youth on the Assessment Leadership Team have prior experience with conducting an

assessment. We provide you with step-by-step directions for collecting the data. If you are

comfortable using Excel for analysis, we have templates that you can use. If you don’t want to use a computer, we have directions for analyzing the data by hand. Similarly, we provide a template to guide you through the process of writing a short report. In several of the more challenging steps, we include “assessment stories” – real-life experiences of other groups that have gone through this process – which should help your Assessment Leadership Team tackle those tasks.

We are not saying that this is easy all of the time. It will be challenging at times because it is new. At the same time, the challenge will often be fun and will always be engaging. We hope it will be a positive learning experience for all who participate.

What do you mean when you say “youth are full partners with adult staff” in program assessment and action planning?

Youth are rarely involved in designing organizational assessments, interpreting the data, or in creating action plans for program improvement. These tasks have traditionally been seen as the responsibility of adults.

When we say “full partners,” we mean that youth and adults will share power, decision-making, and responsibility throughout all aspects of this process. This resource kit is not youth-led and it is not adult-led. The assessment and action planning process is designed so that all persons – youth or adults – can participate in ways consistent with their own interests and expertise. It takes everybody to run a successful program.

(9)

What do you mean when you say “program excellence?”

Program excellence refers to the actions that organizations take to continuously improve the quality of their programming. Youth and Adult Leaders for Program Excellence is based on three assertions:

Program excellence is achieved when staff are involved in program assessment and

reflection. Staff learn and are more motivated to strengthen their own practice when they are actively engaged in the processes of assessment and change.

Program excellence is achieved when youth participate as partners with staff throughout the assessment and action planning phases of improvement.

Program excellence is a gradual process. Strategies of shared learning and continuous improvement are needed to sustain motivation and engagement over the long-term.

These assertions are grounded in the experience of practitioners, research on organizational innovation, and the “best practice” literature in the field of youth development

How much time does it take to do this assessment and program improvement process?

All told, this process takes about 24 hours. Most of those hours are for meetings of the Assessment Leadership Team, but there are also three meetings of your entire group, and some hours set aside for a few members of the Assessment Leadership Team to work together to write a report based on the assessment. You can move through the five phases of the process in as little as a month. We recommend that you finish the process within four months.

The important thing is continuity. Once you start, it is critical that you keep the momentum going.

This resource kit guides you through a series of five phases, each of which provides a clear sequence of steps and concrete directions that will help you sustain momentum.

Some programs may not have the time to go through the whole process, or they want to conduct a program assessment for a specific constituency. We are, therefore, developing user guides that will help programs use the assessment as a foundation for (a) convening or reporting to political and other local leaders, (b) establishing accountability and strengthening communications with external audiences, and (c) orienting and training volunteers who are new to youth participation and youth development. These user guides will be available in early 2005.

Does one staff member need to coordinate the assessment process?

One member of the Assessment Leadership Team serves as the Assessment Coordinator. This person is responsible for copying pages out of the resource kit for each Team meeting, taking care of practical details like meeting arrangements, and keeping momentum going. Often, this role will fall to a staff member. However, it could also be an adult or youth volunteer. It depends on the make-up of your Assessment Leadership Team. Whoever the Assessment Coordinator is, he or she will need access to a photocopy machine and other resources to support the assessment process.

I notice that there are four different assessment instruments that our program can use. Could you provide some background on these assessment tools?

There are four assessment tools in Youth and Adult Leaders for Program Excellence:

YET, which stands for “Youth Engagement Tool,” assesses the degree to which youth have legitimate chances to be heard, respected and taken seriously during program design and implementation. It also examines the outcomes that youth derive from their participation.

(10)

ORG-YET, which stands for “Organizational-Youth Engagement Tool,” assesses the extent to which youth are involved in decision-making forums and the degree to which the organization creates structures to support youth engagement over the long term. It also examines the outcomes that the program derives from youth engagement.

PAAT, which stands for “Program Assessment and Activity Tool,” assesses the developmental opportunities and supports that are provided to youth in the program.

Opportunities include the changes to explore new things, be creative, gain membership in a group, and contribute to others. Supports include nurturance, emotional safety, high expectations, standards and boundaries, and strategic support through transitions.

kidPAAT, which stands for “Program Assessment and Activity Tool (for Younger

Participants),” assesses similar opportunities and supports as PAAT, but in a more simplified format. We recommend using the kidPAAT for children under the age of 12; 12-year-olds may need assistance to complete the full PAAT, but may feel they are too old for a smiley- face survey.

These assessment tools (except kidPAAT) are designed to be completed by youth and adults.

Having youth and adult staff complete the same surveys makes the analysis much easier. It also allows organizations to quickly see the commonalities and differences in how youth and staff view their programs.

These assessment tools are research and practice-based. That is, they reflect the lessons of research and the wisdom of exemplary practitioners about what is most important to measure when assessing program quality. It is important to note that we have been using the PAAT for over five years and the YET for close to three years. They have been field tested with diverse youth in diverse settings.

Our organization hires youth on a part-time basis to help run the program. When analyzing the data, do you consider these “youth staff” to be youth or staff?

It is ultimately your choice. When your group members fill out their surveys (except on the kidPAAT), they will be asked to indicate their age range and whether they consider themselves program participants or staff members. This gives the Assessment Leadership Team the flexibility to pinpoint those staff members who are under 18 or under 25, and choose whether to consider their responses together with other youth or with adult staff.

Our rule of thumb is this: If the “youth staff” have significant management or implementation roles in the program, we consider them to be staff. If the “youth staff” attend staff meetings, we consider them to be staff. If these criteria are not met, we would recommend that you analyze the “youth staff” as youth.

How can I get assistance?

If you have questions or concerns about using Youth and Adult Leaders for Program Excellence, we recommend that you first visit the Act for Youth website (www.actforyouth.net) that is operated by Cornell University. This website provides detailed information on the resource kit, as well as

guidance on obtaining additional materials and assistance. You may also contact Shepherd Zeldin at [email protected] or call 608.263.2383. A member of our team will be happy to talk with you.

(11)

Youth

I have a pretty busy schedule, and “program excellence” isn’t really one of my priorities for how I use my time. Why should I get involved in this program assessment?

It can be hard to see the point of using your time on a program assessment when there are so many other things you could be doing. But there are plenty of good reasons to care about program assessment and action planning, and you could get a lot out of being part of the Assessment Leadership Team. For example:

Even great programs need to take time to assess what they are doing well and where they have room to improve, so that they can continue to be great. If you’ve had a good

experience with your program, this assessment is a way for you to be sure that other kids will have good experiences with it in the future. If you can see ways your program could

improve, then this is a great way to find out whether other people agree with you and start to work toward those changes.

Members of the Assessment Leadership Team learn how to do research, like how to get from a number showing the average score on a series of questions to a concrete

recommendation for program change. They also learn how to facilitate a group discussion and write a report on the assessment. These are valuable leadership skills for school, work, and life in general!

Youth and adults work side by side on the Assessment Leadership Team in a true

partnership. We’ve heard from both young people and adults that their experiences on the Assessment Leadership Team resulted in deeper friendships and better understanding of people in the other age group. For young people, developing those relationships with adults can also be very valuable when they look for jobs, apply for college, or need help with a problem in their community.

Our program is entirely youth-run and does not have any adult staff members. Can we still use this resource kit for a self-assessment?

We’ve never tested Youth and Adult Leaders for Program Excellence with an all-youth program. But we would still encourage you to use the resource kit for program assessment and action planning. If there are no adults involved in your program, you should just make sure that your Assessment Leadership Team includes youth from a broad range of ages, and try to include some participants who are not part of the program’s usual leadership group. If you are affiliated with a larger

organization that employs adult staff, or if you sometimes work with adults in your community, you might want to consider involving adults in your Assessment Leadership Team. Adults may have a different perspective that could be helpful in the assessment process.

Can we write our own survey questions?

This resource kit includes four assessment tools (YET, ORG-YET, PAAT, and kidPAAT – all described above) which were written based on research and practical experience. Each of the assessment tools leaves room for your group to add questions. Your group might choose to add more specific questions that are similar to the ones already in the tool. Or, you might choose to ask about something else that’s important to your program. It’s up to you.

(12)

PHASE 1

Outline of Steps

P ^{H A S E} 1: Planning and preparing for a program assessment

Time required:

A one-hour meeting with the Executive Director and program participants to select Assessment Leadership Team members

One or two meetings (for a total of 2–4 hours) to get to know each other, learn about the program assessment process, and select a survey to use

Responsible Parties:

Executive Director, Youth and adult leaders Assessment Leadership Team

STEP 1 Identify the Assessment Leadership Team

An Assessment Leadership Team is selected, consisting of youth and staff. The Assessment Leadership Team has up to six members, including both youth and adults.

STEP 2 Hold orientation meeting

The Assessment Leadership Team meets for an orientation meeting.

That meeting includes an overview of the process, discussion about which survey to use, and methods of analysis and interpretation. This is also a chance for members of the Assessment Leadership Team to get to know each other and prepare to work together.

STEP 3 Select a survey

The Assessment Leadership Team reviews the purposes of the four surveys and selects the one that will provide the kind of assessment information they want. The Assessment Leadership Team can add questions to the survey at this meeting. The team members practice reading the scripts to introduce the survey and make a plan for administering it.

(13)

PHASE 2

Outline of Steps

P ^HASE 2: Collecting and compiling data

Time required:

One hour of program time to explain and administer the survey

Up to two hours to enter survey responses in the database and print charts and graphs

Responsible Parties:

Assessment Leadership Team Members of the Assessment Leadership Team

STEP 4 Administer the survey

The Assessment Leadership Team distributes the surveys to youth and staff, helps people fill them out (as needed), and then collects the surveys.

STEP 5 Enter data and print output

Members of the Assessment Leadership Team (and any other group members who want to participate) enter the surveys into a database, and then print out tables and figures created by the template. (This step can be done by hand, without using a computer, if necessary.)

(14)

PHASE 3

Outline of Steps

P ^HASE 3: Analyzing and understanding the data

Time required:

Two hours to look at the data and identify key results Up to three hours to write the first draft of the report

Responsible Parties:

STEP 6 Look at and discuss data

The Assessment Leadership Team (and any other group members who want to participate) look at the tables and figures printed out to understand what is important in the data. They decide what they key results of the survey are.

STEP 7 Write first draft of report

The Assessment Leadership Team prepares a brief report on the key results and recommendations (using a template). After it’s drafted, the report is given to the larger group to read.

(15)

PHASE 4

Outline of Steps

P ^HASE 4: Sharing results with the group

Time required:

Two hours to prepare for the large group meeting Two hours of program time to present and review the

findings of the assessment

Responsible Parties:

Assessment Leadership Team Assessment Leadership Team, Program participants

STEP 8 Prepare for group meeting

The Assessment Leadership Team meets to figure out how to present results and recommendations to the larger group and facilitate a discussion.

STEP 9 Facilitate group meeting: Review data and

finalize recommendations

A group meeting is facilitated by the Assessment Leadership Team.

The purposes of the meeting are (1) to discuss the data analysis, (2) to discuss the recommendations in the draft report, (3) to get additional recommendations from the large group, and (4) to agree upon the most important recommendations.

(16)

PHASE 5

Outline of Steps

P ^HASE 5: Action planning and finalizing the report

Time required:

Two hours to complete action step worksheets Two hours to finalize the report

One hour to reflect on the assessment process and get the final report approved

Two-hour meeting to review follow-up on the action steps

Responsible Parties:

Assessment Leadership Team, Executive Director

STEP 10 Complete action step worksheets

and finalize report

The Assessment Leadership Team and others complete the action steps worksheets. Then, they finalize recommendations and add action steps to the final report.

STEP 11 Reflect on the process and get Executive

Director’s approval

The Assessment Leadership Team and representatives of the whole group meet with the Executive Director to reflect on the process and to discuss how things went and what they learned from it. The Executive Director signs off on the report.

STEP 12 Follow up on action steps

About three to four months after the report is finalized, the Assessment Leadership Team and Project Guides meet with the Executive Director to see how things are going, and to discuss how best to continue to implement the report recommendations.

(17)

PHASE 1:

Planning and Preparing for a

Program Assessment

Step 1: Identify the Assessment Leadership Team

Step 2: Hold orientation meeting

Step 3: Select a survey

(18)

9.;;6;7.;1

?2=.?6;7

Planning and preparing might not sound like the most exciting part of your assessment project. But no Leadership, Assessment, or Research Team just plunges in and wings it. It just doesn’t work very well to do things that way. If you want to do assessment and research that is useful—that people can really use—you have to do some up-front planning. Planning and preparation can also be a lot of fun if you use it as an opportunity to get to know your Team at the same time as you get organized for your assessment.

This section will take you through choosing a Team, thinking about why you and your program want to do the assessment, assigning responsibilities, and team building..

Keys to Success

•

Take the time to work through this planning section in the beginning. Don’t skip it!

Other teams have found that the activities included in the section have helped to make the process a success by doing the groundwork.

• Make sure the Program Director and/or Executive Director know what you’re doing.

Communicate regularly with him/her. Communicate where you are in the process.

These individuals are key allies who can help others in the program understand what you’re doing and why. Besides, if they know about the progress of the Assessment, you’re in good position to ask for additional help or resources that you might need.

• Consult with the Program or Executive Director to determine the group you’re going to present the assessment results to, and a date to do it. You might be presenting the results to a Board of Directors or Advisors, a group of Program Participants, or another group. Get a date for the presentation as soon as possible!

• The Assessment Leadership Team will meet several times together as you lead the assessment process. In this section, there are some fun team building activities you can do at meetings.

• Have refreshments at team meetings! Snacks are very helpful. Somehow, assessment, analysis, and writing get everyone hungry!

(19)

1 2 3 4 5 6 7 8 9 ^{10 11 12}

Identify the Assessment Leadership Team

A. Choosing the Assessment Leadership Team – The group or person who initiated the program assessment process should use this as a guide to selecting youth and adults to guide the assessment process.

B. Purposes and Benefits – Use this as a handout for your whole group when you introduce the idea of doing a program assessment. It may also help you to recruit members of the Assessment Leadership Team.

C. Leaders’ Roles and Responsibilities – You can choose whether to use this handout with your

whole group or with just the Assessment Leadership Team. It helps to clarify what roles people

will play on the Team. Write down everyone who is interested in each of the roles (except

Coordinator) to be sure that you have enough people in each role later on.

(20)

Choosing the Assessment Leadership Team

Youth and Adult Leaders for Program Excellence is designed to be carried out by teams of youth and adults working in partnership.

However, partnership does not necessarily mean that equal numbers of youth and adults have to be on the team. Working in partnership means that adults and youth honor and respect each other’s contributions.

The youth and adults in the programs and organizations that have used this assessment process have found that it works best if the Assessment Leadership Team is composed of about 4-5 youth, with 2-3 adults. They have found that a few more youth to adults works best because it is youth programs that the tools assess. However, they have found that having only one adult doesn’t work very well, so you might want to avoid that situation! Another thing that other teams have learned is that both the youth and adults should be interested in learning from one another.

How to choose members for the Team, is of course, up to you and the circumstances of your program. But, it’s always good to know some options before you decide how exactly you’re going to do this. Team members can be chosen according to the following, for example:

Choose individuals who want, and have the commitment, to maintain involvement on the team. Make sure people have the time to do the process. For example, a person who is really interested, but who has many other things to do may not be the best candidate for the team.

Remember, doing the assessment process can take several weeks, sometimes a couple of months, depending on how often the Team can meet, or if there are other things that take higher priority in the program or organization.

Choose individuals with different levels of experience in some of the skills you need for this assessment: computer skills in Microsoft Excel and/or Word, writing, analysis involving averages and reading graphs, leading and facilitating discussion groups, etc.

Choose individuals who want to learn some of the skills involved: computer skills, writing, analysis, leading and facilitating discussion groups. Wanting to learn can be a powerful motivator as well as reward!

(21)

ASSESSMENT STORIES Better understanding

of each other

Youth and adults in an after-school program did the “Youth and Adults for Program Excellence” assessment, and presented the results to their Board of Directors. Board members were excited to hear about the results and read the report. They said that it was a new way for them to think about the program.

The said that it was a good way to understand what program participants were thinking and feeling. The Board members also said that it was very helpful in giving them a concrete way to talk about the program specifics to other community members and funders.

Did someone say “resumé”?

Two young people who were on the Assessment Leadership Team for their program's self-assessment reported that they had learned valuable skills — which also helped other people see their value!

Terrence got an internship over the summer working for city government. His supervisor said that Terrence's

experience working with adults and his skills in Excel made him stand out from the other applicants. And Maria wrote her college application essay about her experience as a leader for her program’s self-assessment. She was accepted at three colleges!

Program Assessment:

Thinking about Purposes and Benefits

We are about to start a program assessment process called “Youth and Adult Leaders for Program Excellence.” There are many reasons to carry out a program assessment.

It helps the program to become better.

It gives all the people in the program a chance to reflect on what they like about the program and what the program could do better.

It is a chance for a small group of youth and adults (called the “Assessment Leadership Team”) to work together and learn how to do research.

Purposes and benefits for programs

Programs and organizations need to assess how well they are doing their job and how well they are meeting the needs of their participants. Our program decided to use this assessment because other programs have learned a lot from it. Some purposes and benefits of this self-assessment process for other programs have been:

Improving processes

Helping the program make constructive change Better understanding of goals and processes Learning to be more effective

Telling the organization or program’s story better to funders and others

Purposes and benefits for individuals on the Assessment Leadership Team

Each person who serves on the Assessment Leadership Team will learn something by doing it. A lot of them may also get a chance to teach something they know to other members of the team. Each member of the Assessment Leadership Team might have different reasons for being there, such as:

Learning skills related to assessment and group change Learning to use research for practical purposes

Being a leader for positive change Increasing skills in team work

Learning how to use surveys, in-put and analyze data, and present the results to others

(22)

Leaders’ Roles & Responsibilities

A group of youth and adults will work together to carry out the Youth and Adult Leaders for Program Excellence self-assessment. Who will this be?

members of our Assessment Leadership Team:

One person should take the role of Assessment Coordinator. This is not the “boss” of the

assessment process, but the person who makes sure that everyone knows about the meetings, makes copies out of the resource kit for the rest of the Team, arranges for snacks and materials to be available at meetings, and generally makes sure the assessment process is on-track.

our Assessment Coordinator:

Members of the Assessment Leadership Team will need to work together to accomplish several types of tasks. Different types of leadership are needed for each of these tasks. It is helpful to think in advance about who will step up to take the lead for each type of task.

Data analyzers:

People who enjoy looking at numbers and survey responses to identify areas where the program is doing well and areas where it could improve. The data analyzers will provide leadership in the Data Analysis meeting.

Report writers:

People who like to write, and who are good at explaining things in simple language.

The writers will have to spend some time outside of Team meetings to put together a first draft of the report, and then modify it later to reflect the results of the assessment and action planning process.

Facilitators:

People who are comfortable being in front of a group, and who can keep a group on track through sometimes difficult discussions. Facilitators work together to make sure that goals are met in a meeting or discussion, and make sure that everyone gets a chance to speak.

(23)

1 2 3 4 5 6 7 8 9 ^{10 11 12}

Hold orientation meeting

A. Purposes and Benefits for Us – You reviewed examples of “Purposes and Benefits” at the large group meeting. Now it’s time to brainstorm the purposes and benefits for your program and for the Assessment Leadership Team as a whole. Be sure to leave a little quiet time so that everyone can write down their own personal purposes and benefits of being on the Team.

B. Team Building – This is a good time to have some fun and let everyone get to know each other before you get further into the process. We suggest a few activities to get you started. For more fun team building activities, check out www.wilderdom.com/games/gamesspecific.html.

C. Planning the Process – Go through the “Outline of Steps” at the beginning of this guide with the entire Assessment Leadership Team. This is a good opportunity to talk about what each of these steps will require. The Team should set target dates for the steps in the process.

Note : The Assessment Leadership Team needs to select a survey to use in the program

assessment. (This is Step 3 in this guide.) You may choose to do that now or at a later meeting,

depending on how much time you have.

(24)

Purposes and Benefits for Us

Your team needs to have clear ideas about the reasons for doing this program assessment. You, as an individual, should also have some clear ideas about why you’re participating in the assessment.

This will help you keep it clear in your own mind, and help you answer questions from people who might ask you, “Why are we doing this assessment?” and, “Why are you on the team?”

Purposes and benefits for our program

List the ways you think doing the self-assessment process can benefit your program, organization, or group:

Purposes and benefits for me personally

It can be hard to think about the benefits to ourselves as we do teamwork. We sometimes think that doing service to our own programs and organizations should not benefit us personally. But it does!

Knowing what you are getting out of a process can help you get through times of challenge, as well as celebrate the victories. Write the ways that you see your participation in this process benefiting you personally:

(25)

Team Building

This Assessment Leadership Team will be meeting several times together. You’ve already worked on purposes and benefits, and roles and responsibilities for shared leadership in the Team. If

members of the Team haven’t worked together very much before, you might want to do some team building activities at your meetings, like the ones listed below. These can help team members understand each other better, and work better together. Besides, they’re fun!

Sound Game

When participants come in, give each person a note card with a sound written on it. (For example,

“Oink” or “Ding Dong.”) Ask them not to share their sound. When they are instructed, they are to make the sound and find those that are making the same sound. When they have found those that are making the sound, they will be divided into groups to work on whatever tasks need to be done at the meeting.

Zip Zap Zoom

Set up a circle of chairs so that there is one chair for each person except one. Have everyone take a seat, except for one person, who stands in the middle of the circle. The person in the middle should go up to any person sitting in the circle, point at him or her, and shout one of three words: Zip, Zap, or Zoom. If the pointer says “Zip,” the person in the chair must call out the name of the person seated to her right. If the pointer says “Zap,” the person in the chair must call out the name of the person seated to her left. The seated person has only five seconds to call out the correct name. If she takes longer than five seconds or calls the wrong name, she goes to the center of the circle and becomes the pointer. (The previous pointer takes his seat.) If the pointer says “Zoom,”

than everyone seated in the circle must get up and find a new seat (at least two chairs away). The pointer does not get to claim a chair, but this does help mix up the seating arrangement and make it more challenging. The pointer stays in the middle until he can get someone to mess up.

Chit Chat

Divide people into small groups and ask them to share their answers to these questions.

What was the happiest moment of your life?

What is the hardest thing you have ever done?

What is the greatest compliment you ever received?

What room in your house do you like best?

What is the one thing you want to accomplish next week?

If you had a time machine that you could use only once, what point in the future or history would you visit?

What is your favorite time of day?

If you were to go to the moon and could carry only one thing, what would it be?

(26)

Scheduling the Process

The Assessment Leadership Team has many leadership tasks to carry out in this process. Drafting out target dates for each task and jotting some notes and plans for tasks is helpful in letting the Team know the big picture, and when the Team is going to be working.

At first, you don’t have to have everything perfectly planned, but it helps to have a draft that you can keep working on as you progress. The one exception to this is in getting a date and an

audience for the presentation of your results. You want to do this as early as possible because everyone is busy. If you get a date early, it is more likely that your audience will not have other competing activities on their calendars, so they can reserve the time to hear your results. Also, getting a presentation date can be very helpful to the Assessment Leadership Team. Somehow, having a deadline to work toward helps everyone get the tasks done.

Target Date Notes and Plans

Step 1 Identify Assessment Leadership Team

DONE!

Step 2 Hold orientation

meeting

DONE!

Phase 1: Planning and preparing ^{Step 3}Select a survey

Step 4 Administer the survey

Phase 2: Collecting and compiling data

Step 5 Enter data and

print output

(27)

Target Date Notes and Plans

Step 6 Look at and discuss data

Phase 3: Analyzing and understanding the data

Step 7 Write first draft of report

Step 8 Prepare for group meeting

Phase 4: Sharing results with the group

Step 9 Facilitate group meeting Step 10

Complete action step worksheets and finalize report Step 11 Reflect on process

and get Executive Director’s approval Phase 5: Action planning and finalizing the report

Step 12 Follow-up on action steps

(28)

1 2 3 4 5 6 7 8 9 ^{10 11 12}

Select a survey

A. Choosing the Right Survey – The Assessment Leadership Team should review the purposes of each survey instrument and decide which one they want to use. Bring at least one copy of each survey (available in the Appendixes of this resource kit) to the meeting, so that Team members can look at the surveys themselves and choose the best one for your group.

B. Adding Questions to a Survey – Your program may want to take this opportunity to gather more information about how participants feel about the program, or ask questions about a particular aspect of the program that needs to be assessed. Adding questions to the survey can also help members of the Assessment Leadership Team feel more ownership of the assessment process – so take the time to talk through this step.

Note : Look ahead! Before your Assessment Leadership Team disperses, be sure you have

planned ahead for administering the survey. See Step 4 to be sure you’ve thought of

everything…

(29)

Choosing the Right Survey

You can choose from four survey tools for your assessment. Each one serves a different purpose.

Youth Engagement Tool

The YET is designed for groups that use youth-adult partnerships to accomplish their goals, or where youth and adults share decision-making responsibilities for a program. It is designed to help you assess how much and how well your program is involving members. Specifically, the YET is about:

Group effectiveness

How well the group functions – establishing trust, meeting goals, setting high standards, and providing enjoyment – and whether it accomplishes what it sets out to do

Youth voice and opportunities

Chances to be heard, respected, and taken seriously by adults and by other youth Impacts

What members and the program gain from involvement by youth and adults – improvements in knowledge, skills, and abilities

Organizational Support For Youth Engagement Tool

Like the YET, this tool is designed for programs where young people work with adults to make decisions for the group. It assesses the level at which your organization or program actively supports youth engagement in decision making. Specifically, the tool asks about:

Degree of youth engagement in decision making Numbers and diversity of youth in decision-making roles Leadership and support for youth in decision making

How much adults and youth in the organization/program support youth engagement; how much the organization/program has structures to support it

Outcomes of youth engagement in decision making

How much youth in decision-making is helping the organization/program do its work Functions where youth are engaged in decision making

Areas of the organization/program where youth are engaged in decision-making; how much youth are involved in decision-making in those areas

(30)

Choosing the Right Survey

(continued)

Program & Activity Assessment Tool

The PAAT is designed to help you assess how much and how well your program is providing its members with a learning experience to promote positive youth development. Specifically, the PAAT asks about:

Opportunities

Do participants feel that they have enough chances in this program to explore new things, be creative, participate in the group, contribute to others, have fun, and engage in part-time employment?

Supports

How well does the program provide its participants with nurturance and friendship,

emotional safety, high expectations, standards and boundaries, and strategic support through transitions to independence or adulthood?

Program & Activity Assessment Tool

For younger participants

The kidPAAT measures Opportunities and Supports that promote positive youth development, like the PAAT, but is written for younger participants. Usually, if you have participants under 12 years old who need to fill out the survey, the kidPAAT will be your best choice.

You may want to use both the PAAT and the kidPAAT if you have a wide range of ages in your program. Youth ages 11–13 can choose which survey they want to fill out.

As a group, decide which survey is best for what you want to find out.

We will use:

(31)

Adding Questions to a Survey

Each survey has space for your group to add in additional questions that you would like answered.

After you choose which survey is the right tool for your assessment, all the Team members should read through it. (You may need to take a break while someone makes copies of the survey for everyone.) What else do you think you would like to learn from this assessment?

If your organization recently changed its programming or is considering making changes, this is a good chance to find out how program participants feel about the changes. For example:

Recent changes to our program have made it a more 1 2 3 4 5 6 fun place to spend time.

Or you could ask an open-ended question:

How would you feel if the center changed its closing time to 9:00?

You might want to use the additional questions to ask more specific questions, about a particular program or a recent experience in the program. If your program is looking for ways to get more youth involved in community work, you could ask,

What do you think is the most important issue for youth in our neighborhood?

Providing opportunities to work on those issues might get more youth involved.

In general, think about what information will be helpful for your Assessment Leadership Team and the organization to have. What do you need to know to make the best recommendations for program improvement? What does the organization need to know about its participants to make it the best possible program?

Write in your extra questions in the blank spaces before you photocopy the survey and distribute it for data collection. The spaces to write in extra questions are:

Part 5, items A, B, C, and D Part 5, items A, B, C, and D

Items 14, 20, and 30 Items 11 and 20

If you do not need to add any questions, draw a line in the blank spaces on the survey before photocopying it. This will keep people from getting confused when they fill in the survey.

(32)

PHASE 2:

Collecting and Compiling Data

Step 4: Administer the survey

Step 5: Enter data and print output

(33)

Collecting and Compiling Data

Data collection is how you’ll get information from youth and adults in the program. Since getting information is a main purpose of the self-assessment, good information collection is important!

Compiling your data accurately is also important – this is when you get all the surveys together so that you can look at things like averages and differences between groups.

This section helps you prepare to administer the survey you chose to use for the assessment, and outlines how you should introduce the survey to the rest of your group. It also gives some general tips for entering your data into an Excel workbook or compiling data by hand. (Specific instructions for the survey you are using are found in the Appendix for that survey tool.)

Keys to Success

COLLECTING ENOUGH RESPONSES

• The more completed surveys you get, the better your information will be. As a rule of thumb, you want to collect surveys from at least 25% (or about one quarter) of program participants (youth).

• Often, there are fewer adults than program participants. You might want to have every adult associated with the program fill out a survey.

• Sometimes people need assistance in reading the survey. If someone would like to have the survey read out loud to him/her, be prepared to do this.

GETTING QUALITY DATA

• Make sure to have someone introduce the survey to the people who will fill it out.

It is very important that people understand why you’re doing the survey, and to understand how they should fill it out.

• Remind people that you need their honest answers. Be sure not to let yourself

influence their ratings, or people might just give you the answers they think you want!

Your program won’t be able to improve if everyone just says everything is fine.

• Allow enough time so that people can think through each question.

SHOWING ACCURATE RESULTS

• Take your time to enter the survey responses accurately into the Excel workbook.

You will be using the charts and graphs from Excel for your data analysis, so you want to be sure they reflect people’s real answers on the survey!

(34)

1 2 3 4 5 6 7 8 9 ^{10 11 12}

Administer the survey

A. Logistics – Plan in advance so that, when it’s time to administer the survey, it’s smooth sailing.

B. Administering the Survey – These are some basic tips for administering the survey to your larger group.

C. Confidentiality– Researchers always need to be careful about preserving the confidentiality of responses to a survey. Youth researchers, in particular, will benefit from talking about this issue before administering the survey.

D. Script for Introducing the Survey – You will find this script in the Appendix for the survey

instrument chosen by your Assessment Leadership Team. One or two member(s) of the Team

should be responsible for introducing and explaining the survey, with this script as a guide.

(35)

Logistics

(lo-jis’-tiks)

The management of the details of an operation

When you assemble your group to administer the survey, there are a few things you will need to have prepared in advance. You don’t want to stand there in front of the group and not know what to say, or not have enough surveys for everyone. You also don’t want to run around looking for pens and pencils for ten minutes while the group sits and waits to fill out their surveys. So, it’s a good idea to make sure that someone is responsible for each item on the checklist below.

Task or materials needed Person responsible

Make copies of the survey and bring to the meeting space

• Number of copies needed: _____

• Additional questions should be written in before copying, if needed

Bring pens and pencils to the meeting space

• Number needed: _____

• Be sure to return the pens and pencils to where they came from when you are done, too

Read the script to introduce the survey

• Customized for your group

• Rehearsed

Be prepared to answer questions from the group while they fill out the survey

Put the completed surveys in a safe place until the

Assessment Leadership Team meets again for data

entry

(36)

Administering the Survey

When it’s time to have the members of your program fill out the survey, someone from the

Assessment Leadership Team will need to introduce the survey, answer questions that people have, and collect the completed surveys.

Who

will take this role? __________________________________________________

It’s important that the members of your program understand why they are being asked to fill out the survey. You want them to be honest and to give the survey all their attention for as long as it takes to fill it out. If they rush through it or don’t really pay attention, your results probably won’t be very accurate. Remind people that even if they really like the program, they should think carefully about how it could improve to become even better or to meet the needs of even more young people.

Circling the highest number for every question won’t help the program to improve.

This kit provides a script for introducing the survey(s) you are using in your group. You can find the survey script in the Appendix for the survey you are using.

You don’t need to read the script word for word, but many people find it helpful to have something written down before they speak in front of a group. It will help you to avoid leaving anything out.

Be sure to practice what you are going to say at least once before your group meeting!

Finally, remember to make sure that members of the Assessment Leadership Team get a chance to fill out their surveys, too! You might want to take turns answering questions from the rest of the group so that everyone on the Team gets a chance to sit down and complete the survey.

good luck !

(37)

a few words about…

Confidentiality

Confidentiality is a big concern whenever you do research or evaluation. People need to feel comfortable saying how they really feel, which means that they need to trust the Assessment Leadership Team not to gossip about who said what or put them on the spot later because of something they wrote on their survey.

There are two aspects of confidentiality that are important. The main one is that everyone on the Assessment Leadership Team respects people’s right to express their opinions and understands that survey responses are confidential. The second aspect of confidentiality is that the people who fill out the survey feel safe expressing their opinions and don’t doubt that the Assessment Leadership Team will keep their responses confidential.

Here are some things members of the Assessment Leadership Team can do to address both aspects of confidentiality:

*

Remind everyone that every program has room for improvement and that they should answer honestly to help the program become better. (This is part of the “Script for introducing the survey” included in each Appendix.)

*

Don’t look over anyone’s shoulder while they are filling out the survey, and don’t stand somewhere where they might think you can see what they’re writing.

*

Remind people not to write their names on the survey.

*

Don’t try to figure out who said what or whose survey you are looking at during data entry.

Even if you recognize someone’s handwriting or think you know who might have written something, don’t say anything to the rest of the Assessment Leadership Team. People’s responses to the survey should be kept confidential.

*

When your whole group meets to discuss the findings of your assessment, don’t single out anyone because you think they responded in a certain way. Steer clear of asking people to say how they responded to a particular question. Instead, ask something like “What are some of the reasons people would have responded negatively to this question?” and “What can we do to improve people's experiences in this program?”

A PRACTICAL GUIDE FOR PROGRAM ASSESSMENT AND ACTION PLANNING

YOUTH AND ADULT LEADERS FOR

PROGRAM EXCELLENCE:

A PRACTICAL GUIDE FOR

PROGRAM ASSESSMENT

AND

ACTION PLANNING

• C

& A

•

Youth and Adult Leaders for Program Excellence:

A Practical Guide for Program Assessment and Action Planning

By Linda Camino, Shepherd Zeldin, Carrie Mook, and Cailin O’Connor

Collaborators

• C

& A

•

CONTENTS

Intro: Overview, FAQ’ s , and Outline of Step s

Phase 1: Planning and preparing to

conduct a program assessment Phase 2: Collecting and compiling data Phase 3: Analyzing and understanding

the data

Phase 4: Sharing results with the group Phase 5: Action planning and finalizing the

report

Appendixes: Survey Tools and templates

for data analysis and reporting

INTRO:

Overview,

Frequently Asked

Questions, and

Outline of Steps

Overview

Phase 1: Planning and preparing to conduct a program assessment Phase 2: Collecting and compiling data

Phase 3: Analyzing and understanding the data Phase 4: Sharing results with the group

Phase 5: Action planning and finalizing the report

Benefits to youth and adults

who lead the assessment and action planning process

Frequently Asked Questions

Organization Leaders

Why is it important to engage in organizational self-assessment?

Does this organizational assessment and action planning process really work?

What types of programs and ages of youth is this resource kit designed for?

Can you provide other materials to support the process?

Program Staff

My program has never done an assessment before. For example, we’ve never collected and interpreted data, and don’t really know how to write assessment reports. How can we do this?

What do you mean when you say “youth are full partners with adult staff” in program assessment and action planning?

What do you mean when you say “program excellence?”

How much time does it take to do this assessment and program improvement process?

Does one staff member need to coordinate the assessment process?

I notice that there are four different assessment instruments that our program can use. Could you provide some background on these assessment tools?

Our organization hires youth on a part-time basis to help run the program. When analyzing the data, do you consider these “youth staff” to be youth or staff?

How can I get assistance?

Youth

I have a pretty busy schedule, and “program excellence” isn’t really one of my priorities for how I use my time. Why should I get involved in this program assessment?

Our program is entirely youth-run and does not have any adult staff members. Can we still use this resource kit for a self-assessment?

Can we write our own survey questions?

PHASE 1

Outline of Steps

P H A S E 1: Planning and preparing for a program assessment

Time required:

Responsible Parties:

STEP 1 Identify the Assessment Leadership Team

STEP 2 Hold orientation meeting

STEP 3 Select a survey

PHASE 2

Outline of Steps

P HASE 2: Collecting and compiling data

Time required:

Responsible Parties:

STEP 4 Administer the survey

STEP 5 Enter data and print output

PHASE 3

Outline of Steps

P HASE 3: Analyzing and understanding the data

Time required:

Responsible Parties:

STEP 6 Look at and discuss data

STEP 7 Write first draft of report

PHASE 4

Outline of Steps

Intro: Overview, FAQ’ ^s , and Outline of Step s

P ^{H A S E} 1: Planning and preparing for a program assessment

P ^HASE 2: Collecting and compiling data

P ^HASE 3: Analyzing and understanding the data

P ^HASE 4: Sharing results with the group

P ^HASE 5: Action planning and finalizing the report

9.;;6;7.;1

?2=.?6;7

1 2 3 4 5 6 7 8 9 ^{10 11 12}

1 2 3 4 5 6 7 8 9 ^{10 11 12}

1 2 3 4 5 6 7 8 9 ^{10 11 12}