Once the campaign has been implemented and the target audience is beginning to pay attention to the message, it could be very easy to sit back and enjoy the “success”
of all the planning and effort by everyone involved. But work on the campaign is not yet finished! Evaluation is a critical component of this process. Evaluation involves the use of research procedures to determine whether the campaign was effective, how it did and did not achieve its objectives, and the efficiency with which it achieved them. In other words, the campaign’s “success” cannot be concluded until evaluation research has proven it.
Three main findings should result from the evaluation phase in the form of answers to the following questions:
1. To what degree did the campaign reach its objectives?
2. How or why did the campaign work?
3. What lessons can be learned for future public information campaigns?
Two different types of research help examine the effectiveness of the cam-paign: process research and outcome research. Process research usually involves collecting data on when, where, and for how long the campaign is broadcast.
For example, the evaluator might want to listen to the radio at prespecified times to verify that the ad was played the number of times it was supposed to be played. Outcome research is conducted by collecting data to measure the
program’s impact. This data collection could be done using quantitative mea-sures such as surveys, or qualitative methods such as interviews or focus groups, but no matter which method is selected, it is important to include both people who have been exposed to the campaign and those who have not been exposed to the campaign in the evaluation research. This control group provides a com-parison and allows the evaluator to determine just how effective the campaign has been.
Some of the critical variables that should be assessed during the evaluation phase include the following:
◾ Campaign exposure. This is the extent to which the targeted audience has seen, can recall, and/or can recognize the campaign materials. Process evaluation can help determine the level of possible exposure to the campaign, but only outcome research can provide information about whether members of the target audience recognize or recall the campaign message or components of the campaign.
◾ Interpersonal communication. Public communication campaigns can be very effective at stimulating interpersonal communication. If a target audience member sees one of the campaign tactics, it may or may not change his or her behavior. But frequently this target audience member is quite likely to men-tion what he or she saw to a friend, coworker, or family member. Now the information in the campaign has spread even further even though this next group of people has not yet seen any campaign materials. When these friends, coworkers, and family members are actually exposed to one of the campaign messages, research suggests they are more likely to pay attention because they had previously heard some of this information. Outcome research can help the evaluator determine if interpersonal communication played a role in the campaign’s effectiveness, and if so, how.
◾ Campaign impact. Clearly, the goals of a campaign are to change the audi-ence’s knowledge level, attitude, and behavior, so it is critical to measure changes in these variables. Assessing any other variables that comprise the theory selected to guide the campaign development also is recommended.
A good way to approach evaluation is to consider each objective separately. The Don’t Be That Guy Alcohol Reduction Education Campaign used this approach in its evaluation. Its first objective was to raise awareness for the That Guy campaign and the negative effects of excessive alcohol consumption. Process research pro-vided the campaign evaluators with some initial information. For example, it was noted that the www.ThatGuy.com website recorded 235,082 different visitors who spent an average of six minutes visiting the site and viewed a total of 954,798 pages.
The That Guy MySpace profile registered 1,800 friends, over 167,750 promotional materials were distributed, and specific counts were obtained for how many times the video PSA ran in movie theaters, the radio spots were aired, and the streaming
Public Information Campaigns ◾ 97
website ads were run. This information provided evaluation of the campaign expo-sure, but outcome research was needed to demonstrate campaign impact.
The DOD conducted two types of outcome research to assess the first objec-tive: focus groups and a web survey. The focus groups allowed the evaluators to determine several important evaluation outcomes. First, it was found that there was a high level of awareness of the campaign among the target population. Focus group participants not only knew about the campaign, but indicated that the That Guy messages were relevant and real. Second, evaluators were able to determine that the communication vehicles they selected were effectively delivering their mes-sages. Third, the focus group participants indicated that the campaign was helping service members be more cautious regarding what they do when they drink. While the focus groups provided evaluators with valuable information about campaign effectiveness, another important component of the evaluation was a web survey featured on www.ThatGuy.com that collected additional data. The survey indi-cated that 37 percent of those who visited the site and viewed campaign messages were likely to think twice about their actions so they don’t become That Guy, thus demonstrating an impact on behavioral intentions. The respondents also provided information indicating that the campaign’s strategy of using humor and entertain-ment to reach the audience was working. Finally, many of the survey respondents reported that they visited the website at a friend’s recommendation, which showed that interpersonal communication was contributing to the success of the campaign by making it viral.
The second objective of the campaign was to motivate 50 military installations to implement the campaign in the first year of the program while leveraging results to set the stage for increasing engagement among additional installations the fol-lowing year. Process research indicated that by the end of the first year, 145 military installations were engaged in the campaign, well over the stated objective. At the same time, 900 military sponsors in more than 40 states, the District of Columbia, and 9 foreign countries also were promoting the That Guy campaign, so that the increased engagement was achieved within the first year of the campaign. Building on the campaign’s first-year success, the DOD created and aired a suite of six addi-tional public service announcements on American Forces Network, generating a total of nearly 120 pro bono airings per week on three different stations. Their success also allowed them to formalize partnerships with many different military organizations as the campaign grew even larger.
Only by collecting all of the evaluation research, both process and outcome, and both quantitative and qualitative, could the DOD reach the conclusion that the That Guy campaign had met both of the specified objectives, and therefore, was successful. The evaluation research also allowed communicators to revisit the theory they selected at the beginning of their campaign process to determine if progress had been made. Based on the information obtained, many members of the target audience clearly had moved from the precontemplation stage of the TTM into other stages. While many of the enlisted men had moved to contemplation
or preparation, some had progressed to action and maintenance stages. Revisiting theory is another important component of the evaluation process that allows the campaign planners to assess their success.
When evaluation shows that campaigns have been successful in achieving their objectives, everyone involved is pleased. But what happens when evaluation does not demonstrate success? Most importantly, if this happens, remember not to panic.
This outcome should be used as a learning opportunity to discern which parts of the campaign process need more attention the next time. Questions to be asked include: Was more research needed to gain a more comprehensive picture before moving to the action phase? Was the theory selected appropriate for guiding cam-paign development or might a different theory be better? Were the goals and objec-tives reasonable and achievable? Did we have enough information about the target audience? Were the strategies and tactics effective in achieving campaign goals?
Were the time and budget allocations sufficient for what we were trying to accom-plish? Was the right message used in the campaign? Did our evaluation assess all important aspects of the campaign? Answers to these questions will allow planners to be better prepared for the next adventure in public information campaigns.