Q2: Evaluation Use

Evaluation question 2 asks: How are program stakeholders within the organization using evaluation? As a branch of the utilization-focused strand of the evaluation roots tree described by Alkin (2004), collaborative evaluation, similar to other responsive stakeholder approaches, seeks

to employ strategies that make it more likely to be used and relevant throughout the project. Specifically, researchers have found that the interaction and involvement by stakeholders and evaluators maximizes the use of evaluation long term (Johnson et al., 2009). Moreover, one of its purposes or roles, per se, is to be used by the program and its staff for program improvement (Chelminsky & Shadish, 1997; Fetterman, 2001; O’Sullivan, 2004). In general, evaluation use refers to the level of traction gained by the evaluation to shape the program. For example, was the evaluation used to make programmatic, organizational, and/or staffing decisions? This section primarily shows results of two survey questions asked YMCA survey respondents directly about the use of evaluation across 13 program activities. However, within the survey, one question series item asked specifically about the influence of collaborative evaluation on the organization’s use of evaluation, and 71% of respondents said they “are utilizing evaluation more than before.” Additionally, 83% of 40 respondents reported their organization was more likely to integrate evaluation activities into their existing programs.

Table 18 shows respondents’ reported rates of evaluation use by item and cohort. Survey participants were asked to indicate how often, if at all, they use evaluation for the following activities listed a through m. The top reason respondents reported used evaluation was to better understand the extent to which their program had been successful. In contrast, the least reported reason to use evaluation was to make a program staffing decision. Of the 13, 10 items averaged a majority agreement response of over 90 percent for both cohorts. Four items on the survey had the most variation of YMCA members by cohort group: j. Analyze my students’ performance compared to their peers, k. Make a funding decision, l. Change one of my program’s priorities, and m. Make a program staffing decision.

Table 18.

Number of Respondents in Agreement about Frequency of Evaluation Use by Item and Cohort

Item All Cohort I Cohort II

1 a. Better understand the extent to which my

program has been successful 36/36 15/15 16/16

2 b. Make decisions about program activities 35/36 15/16 15/15 3 c. Reflect on the weaknesses of my program 35/36 15/16 15/15 4 d. Reflect on the strengths of my program 35/36 15/16 15/15 5 e. Make decisions about program content 34/35 14/15 15/15 6 f. Communicate with program stakeholders 34/35 14/15 15/15

7 g. Monitor program services 35/37 14/15 17/17

8 h. Better understand the student perceptions of the

program 33/35 14/15 14/15

9 i. Adhere to funding guidelines 31/34 13/15 13/14

10 j. Analyze my students' performance compared to

their peers 29/32 14/14 11/14

11 k. Make a funding decision 29/35 13/15 11/15

12 l. Change one of my program's priorities 28/34 10/14 13/15 13 m. Make a program staffing decision 25/31 11/14 11/12

The use of evaluation to make staffing and funding decisions received the highest level of variation by cohort and lowest frequency overall. However, a chi-square analysis showed no significant differences by cohort group or initial evaluation knowledge.

Table 19 shows results from another series of survey questions asking respondents to determine how they used the logic model in program development. These findings are similar to the previous ones but with an even stronger endorsement for the use of logic models in program elements. In fact the majority of respondents indicated a frequent to almost always use of the logic model for assisting with program planning, understanding data collection, communicating results, and making program decisions.

Table 19.

Frequency of Respondents Indicating Logic Model Use for Program Elements by Cohort

Item All Cohort I Cohort II

b. Understand what data needs to be collected related

to program outcomes 34/36 15/15 14/15

c. Communicate to stakeholders about program

content 34/36 15/15 12/13

d. Support a decision made prior to the evaluation 32/34 15/15 13/14

e. Communicate program results 33/35 15/15 13/14

f. Make program decisions 32/34 15/15 12/13

g. Assist in designing my program curricula 32/34 14/14 13/14

These data indicate programs are using the evaluation logic model to make programmatic decisions. In fact, one of those respondents said, “I looked forward to the evaluation after each event to gauge the effectiveness of our programs and identify the areas/resources that need strengthening.” Moreover, the frequency of logic model use present by key element shows a level of confidence in its use as a programmatic feature.

Much of the interview case study discourse around evaluation also included recognition by leadership to use the logic model as a map or plan to guide the organization through

reinvention, relationships, and restructuring for efficiency with funds based on community needs and program goals. Use of logic models beyond higher education service projects to include other YMCA program areas speak to the value and importance of evaluation use in program planning.

And we were using that logic model to develop what a quality program would look like. It was more aligned with not only the pre collegiate opportunities and services but also the work we were doing with the college students themselves. The whole infrastructure – looking at it from the ground up, systemically - is what this whole opportunity gave us. What is the process, the entire logic model process, how will we do the work, what are we going to do, and the assessment. (Beta #1)

Here is what our test grant’s about. Help foster youth make connections and then we develop activities and even smaller activities. I was able to use that same map in our strategic plan about safety. (Alpha #1)

We are doing autistic buddy system swim program teaching kids to have fun and support one another. Again, we are trying to reach out and get something different and with that program, we are using the road map to measure outcomes that the autistic society agrees and what we feel we should be getting out of that. I am putting it in the heart of my swim lesson time. (Alpha #1)

Just understanding how to walk with the logic model, has helped me as we build our case to show that we are really meeting our outcomes and that it is not just about outputs and working backwards how do we implement activities, different way to work through and your work as well as team members has been really crucial. (Delta, #1)

Even in the two years they were not doing as much with local colleges, it [logic model] helped guide us to that and now it is more of an expectation now. These doors are now open and it is helping our students see, guided us towards improving our programs. (Gamma, #1)

Logic model as a framework seemed to support a new program-planning model. When asked about how they were evaluating programs prior to the YHESP grant, all but one

interviewee described limited knowledge coupled with unrealistic demands. In many ways, the culture prior to YHESP described how programmatic decisions themselves were based on funding. Interestingly when discussing the current state, the interviewees still tied funding streams to program decisions through reliance on evaluation data but for both evidence of outcomes and information to improve program quality.

4.4.1 Conclusion

Results from this section point to YMCA respondents’ increased use of evaluations and, more specifically, logic models in program planning. Even more, when comparing it to their use of evaluation prior to this effort, the majority of respondents found they were using evaluation more. In most cases, use was for program planning decisions with no notable variation among cohort or initial knowledge capacity. These data support the claim that this collaborative