Describe the program by updating or creating a new logic model

We cannot emphasize too strongly how important it is to have a good de- scription of a program before attempting to design an impact evaluation. The description must include a clear statement of the goals, objectives, and strate- gies associated with a program. The goals should be clearly stated in a “what is expected to be accomplished by when format.” A good description of the program tells how the “what” is to be reached in the expected time frame and makes it easier to identify the aspects of a program that need to be evaluated. A good description also helps to assure that you identify all of the factors that may need to be taken into account in the evaluation.

Periodically or at the end of an evaluation, the evaluator and the program manager should revisit the goals. The purpose is to assess whether the content of the evaluation goals is appropriate in light of the program goals and whether the program goals are realistic or whether they need to be re- evaluated.

One of the best ways of generating a good program description is to up- date or create a program logic model. A logic model is usually a two dimen- sional figure with a logical sequence of program activities in one dimension and the resources, activities, outputs, and short and long term outcomes associated with each of the activities arrayed in the other. Some programs have logic models while others may not. Even if a program has an existing logic model, it is important to re-examine it before starting an evaluation and ask how the program has changed since the logic model was last updated. Program managers are constantly making changes to their programs for a variety of reasons. For example, they come up with a new approach, discover that they cannot do something that they thought they could, improve some-

State goals, objectives, and strategies clearly. A good format for a goal statement is “to specify what will happen by when.”

Update or develop a program logic model to describe your program.

A logic model helps simplify the under- standing of complex programs, shows the linkages among elements, and highlights what you expect of target audiences and partners.

thing that has not worked, or discover forces at work in the external environ- ment that need to be leveraged or taken into account.

Figure 6 is one version of a high-level logic model for EERE’s Industrial Technology Delivery Program (ITDP). A logic model like this is very helpful because it provides evaluators with a lot of program information. One striking fact about the ITDP is the complexity and the many inter-connections among its various components. For example, the program has an activity where it creates software and a variety of other activities that result in the software being used, for example, basic software downloads or training programs.

More importantly, the logic model helps evaluators quickly identify the large range of outcomes that can result from program outputs. Examples include a student learning to use the software while doing an industrial audit, a graduate of the Industrial Assessment Center (IAC) program using the soft- ware for her employer, an industry consultant downloading the software to see what might be improved in his own spreadsheets for analyzing steam systems, a facility engineer looking for ways to control costs, and so forth. Under- standing these pathways can help the program manager to more fully identify their potential impacts and to identify evaluation priorities.

Logic models are sometimes criticized because feedback loops are not typically shown. People who use and develop logic models understand that the interconnections and feedback loops are implicit. For example, it is understood that feedback from architectural and engineering partners to software developers in the knowledge domain might be an outcome of train- ing, or that use of the software could lead to changes in the software. Part of the job of an evaluator, and an important use of logic models, is to pinpoint areas where feedback needs to occur and ask if the organizational connections are in place to facilitate the feedback.

Tailor the scope of the impact evalua- tion to the needs of your program and stakeholders.

Logic models are often used to describe a view of how a program cur- rently works. That is a useful first step. Some suggest that this is a major limitation of logic models because it may discourage program managers and evaluators from thinking beyond the obvious.

Logic models are much more useful when they are forward looking, an- ticipating what actors, what actions, and what connections between actors are required to accomplish program goals. Logic models can be used to analyze what needs to occur as well as what is occurring. Indeed this is one of the main themes in this document. It is easier to think about what the other connections may be if you have a story line or theory to help guide your thinking.

In addition, useful logic models identify external factors that impact the efficacy of programs. In the ITDP model, outsourcing is listed as an external factor. There has been a general trend for industrial firms to use more outside engineering staff than in the past. Managers in industrial plants report that this leaves fewer internal personnel to identify, promote, manage, and imple- ment energy efficiency projects. It may also make it more difficult to main- tain gains in energy efficiency. Identifying external factors is quite different than describing the current state of the program and should not be overlooked.