Specify Task Responsibilities and Timelines

Once the evaluation questions, design, and methods have been specified, one must determine who will carry out the various tasks and when the tasks need to be accomplished. For example, line staff may perform some data collection in the routine course of their activities. Including line staff in the stakeholders’ group should generate the cooperation necessary to collect the data and

19–7

should also ensure that data collection requirements do not unduly burden staff. Clerical staff may perform other tasks (e.g., extracting information from agency records, collating information about a case from multiple sources, performing computer data entry). The evaluation may require outside consultants for some of the more technical tasks, such as designing or tailoring the management information system, modifying or critiquing the evaluation design, locating or developing measurement tools, and analyzing and interpreting data. Although the program administrator must retain overall responsibility for the evaluation, assigning someone else to handle day-to-day management of the evaluation is a good idea. The program may decide that hiring an outside consultant as evaluation manager enhances the credibility of the evaluation.

IV. Two Types of Evaluation

The two general types of evaluation—implementation and outcome—are both essential.

Implementation and outcome evaluations address different kinds of questions but can

complement each other. This section describes each type of evaluation in detail, together with a model that links the two.

A human services program, such as an IAP, intends to deliver intervention activities, using staff and other program resources (inputs), with or on behalf of specified clients (inputs) to achieve certain outcomes. In an IAP program, relevant program activities might include assessment and case planning, case management, surveillance, and service brokerage. “Outcomes” refer to observable changes in program participants or their situations after they receive services.

Outcomes may include changes in the following:

• Knowledge (e.g., participants now know where to find resources in their communities).

• Attitudes (e.g., participants now think that education is a good thing to acquire).

• Beliefs (e.g., participants now believe that using illegal drugs can impair their health).

• Behavior (e.g., participants use verbal problem-solving skills instead of fighting to resolve conflict).

• Environment (e.g., the program has helped create new job opportunities for youth).

The following sidebar outlines this process.

IAP-Topic19

19–8 Program Logic and Evaluation*

Inputs: resources dedicated to or consumed by the program (e.g., money, staff time, facilities, equipment, and supplies); and clients (e.g., for an IAP program, serious juvenile offenders released from institutional placement).

Activities: what the program does with the inputs (e.g. strategies, techniques, and types of treatment or services offered). IAP activities might include assessment, individual case planning, the surveillance/service mix, graduated incentives/consequences, and service brokerage.

Outputs: volume of work accomplished (e.g., number of units of service delivered, number of clients or participants served, quantity of materials distributed).

Outcomes: benefits or changes for individuals or populations during or after participation in program activities. Outcomes are usually expressed in terms of the KABBE model:

Knowledge—e.g., cognitive gains.

Attitudes—e.g., attitudes regarding school, work, peers.

Beliefs—e.g., beliefs in laws.

Behavior—e.g., reduced recidivism or improved social skills, school, or work performance, etc.

Environment—e.g., improvements in the juvenile justice system.

Program logic (or theory of action): logical link between inputs, activities, outputs, and outcomes. Can be articulated in two directions

• From intended outcomes backward—by asking how do we expect participants to attain the intended outcomes.

• From inputs or activities forward—by asking why we are gathering these inputs or providing these activities.

Outcome chain: logical or temporal chain from immediate (proximal) outcomes to intermediate outcomes to long-term outcomes. For example, an IAP program component focusing on parent education could have the following outcome chain: by increasing parents’ knowledge of child development and parenting techniques (immediate outcome), we expect their interactions with their children to become more positive (intermediate outcome), strengthening the bond between youth and parents (intermediate outcome) in a way that enhances positive youth development (long-term outcome) and decreases the probability of delinquent behavior (long-term outcome).

*Adapted from the United Way of America model (Hatry et al., 1996).

19–9

For a successful evaluation, all aspects of this intervention system must be measured and linked.

Often, evaluations fail because they neglect to measure the interventions themselves (type, intensity, frequency, duration, and/or quality of program activities actually delivered to clients);

as a result, any outcomes, even if measured carefully, cannot be attributed to the program

interventions. In other words, a good evaluation must be able to show what actually happened (or did not happen) to whom and with what results.

For example, if some clients did not show reduced recidivism after 12 months, one might be tempted to conclude that the IAP model was ineffective. However, perhaps the clients never actually received intensive surveillance and services. Their poor outcomes may reflect not a failure of the IAP model but rather a failure to actually implement the model with those clients.

Suppose that a program had a goal of reducing recidivism for 80 percent of its clients, but an outcome evaluation showed that only half showed reduced recidivism after 12 months. Again, one might be tempted to conclude that the IAP model was ineffective. Perhaps, however, many of the clients referred to the IAP were inappropriate for such a program, or perhaps the

successful 50 percent were systematically different from the unsuccessful 50 percent. In this example, rather than abandon the IAP model, one could instead refine the referral process or adjust the interventions for some subgroups of clients. Only thorough measurement of client characteristics and careful analysis of relationships among client characteristics, interventions, and outcomes can produce understandable and truly useful results.

V. Evaluating Program Implementation

Implementation evaluation focuses on a program’s ongoing activities and thus can be

incorporated into a program’s routine supervision functions. This type of evaluation asks who is doing what for whom, when, and how? Are these activities being conducted as intended? If there are discrepancies, what do they mean? What can be altered to bring the reality more into line with program intentions, or, alternatively, what can be altered to modify the intentions in the face of reality?

Although implementation evaluation may use quantitative information (e.g., number of

staff/client contacts, number of clients enrolled in job training, average length of time for clients to progress through various steps of the program), qualitative information may be especially useful in developing an understanding of how and why certain program components actually function as they do. Open-ended interviews with staff about their perceived frustrations and satisfactions may reveal a great deal about unanticipated obstacles to program implementation.

Observation of a few intake sessions may clarify why an assessment process is not working as intended. In addition to client records, other program documents, such as mission statements, contracts, press releases, internal memos, and meeting minutes, can all contribute valuable information.

Implementation evaluation is most useful as a frequently repeated process of testing, reporting, modifying, and retesting. Several potential questions for an implementation evaluation of IAP programs are discussed below. Although a given program will develop its own questions through the planning process described above, the following suggestions should be helpful.

IAP-Topic19

19–10 A. Are the Participants Appropriate?

The IAP model is designed to reintegrate into the community youth with serious delinquency histories. The intense support IAP provides should permit the earlier release of such youth from institutional placement than would otherwise be possible. It is conceivable, however, that some judges and probation officers may view IAP so favorably that they will direct more youth (not just the most seriously delinquent ones) into IAP programs. Committing more youth to

institutions, perhaps for short periods, may strain aftercare programming resources that are available for seriously delinquent youth. IAP programs must carefully monitor the characteristics of incoming clients as well as the referral and intake process to detect discrepancies from the intended target population.