Relevant Methods

A range of methods can be used to evaluate the potential effectiveness of a risk communication before implementation, depending on the resources available. Conducting a literature review, systematic review, or meta-analysis (i.e., a synthesis approach) of the existing literature can accomplish several goals. It can highlight the risk information that needs to be communicated and the sources of risk communication to which the receivers of information may have access. Moreover, it can provide guidance on the most effective means of presenting and designing a risk communication. This is often the first step in the design of an effective risk communication. A slightly more labour-intensive method is to conduct a textual analysis of current and previously issued risk communications to understand common types of content, designs, and delivery channels. Taken together, the synthesis approach and textual analysis can provide an accurate risk characterization, understanding of risk decision-making, and overview of the

most essential features of an effective risk communication. With these benefits in mind, the generality of these methods provides limited guidance about a specific risk communication in a specific context.

Interviews and focus groups examine the information needs, beliefs, and behaviours

of the receivers of information in the context of a specific health product risk communication. This provides an initial characterization of the information needed, including preferred wording and style of presentation (Jardine, 2008; Bruine de Bruin & Bostrom, 2013). Moreover, focus groups provide a relatively straightforward way to engage the receivers of information and other stakeholders. Once a risk communication has been designed, but before it is implemented, an evaluation using these methods can also test whether it will be effective by directly asking participants about the relevance of the communication content and how it may influence their decision-making and behaviour (Fischhoff, 2009). There are, however, a number of challenges associated with this type of testing. Participants may be unaware and/or unable to predict how information will inform their decision-making. It may also be difficult to determine the accuracy of responses and how they compare across individuals. Finally, these methods do not provide a means to compare the decision-making of individuals who receive a risk communication with those who do not.

At this stage, RCTs can be used by randomly assigning individuals to receive different types of information by altering the content, design, or delivery of risk communications. For example, as described in Chapter 3, Schwartz et al. (2009) conducted two RCTs to test the effectiveness of drug fact boxes relative to direct- to-consumer drug ads. One trial involved information related to a drugs ability to alleviate allergy or heartburn symptoms (symptom drug boxes), while the other focused on a drugs ability to prevent cardiovascular events (prevention drug boxes). In both trials, the control groups received two actual drug ads that consisted of a front page and a summary while the treatment group received the same ads with the summary replaced by a drug fact box. In the trial using symptom drug boxes, 68% of the treatment group chose the superior drug (compared to 31% in the control group) while in the trial using prevention drug boxes, 72% of the treatment correctly identified the drug benefit (compared to 9% in the control group) (Schwartz et al., 2009). This study provides clear evidence that a drug fact box is an effective risk communication tool. RCTs are an effective method for conducting controlled tests of various content, design, and delivery channels across the receivers of information; however, these results are specific to a given setting and cannot be easily generalized to other situations (Cartwright & Munro, 2010; DFID, 2012). Of the methods described in this section, RCTs are the most resource intensive and difficult to conduct.

With sufficient resources and commitment, mixed methods provide the most comprehensive way to identify the appropriate populations, sources, content, design, and delivery channels for a risk communication, ultimately informing its development. Box 4.1 provides a case study of one such evaluation.

Box 4.1

Developing Risk Communication Tools with Mixed Methods Evaluation

A study of the beliefs of adolescent girls related to sexual behaviour provides an example of how mixed methods evaluation can inform risk communication development. Bruine de Bruin et al. (2007) note, based on a literature review, that adults and teens view sexual health risks differently, and that the former group may make erroneous assumptions about why teens engage in risky sexual behaviour. To develop a more effective risk communication tool about sexual health and use of related health products (e.g., condoms), needs assessment and pre- testing were carried out using mental models (Morgan et al., 2002). This process involved examining the available scientific model and the model of a sample of adolescent girls. To develop the adolescent girl model, engagement with this group was needed. This was achieved through in-depth individual interviews using non- directive questions and increasingly specific prompts (to explore understanding, including intuitive theories), followed by written surveys to “[measure] the prevalence and correlates of potentially critical beliefs, as identified by the interviews” (Bruine de Bruin et al., 2007).

Based on the gap between the scientific and adolescent models, an interactive video format on DVD (What Could You Do?) was developed to empower young women in making sexual health decisions (Bruine de Bruin et al., 2007). As the authors describe, “empowering [young women]…means helping them to understand their world and work it to their best advantage. Given the complexity of sexual decisions, young women need a broad perspective to create and evaluate options, adapt to unanticipated obstacles and opportunities, and consider the broader context that gives meaning to their lives and relationships.” Further evaluation methods that used think-aloud protocols helped to make the content realistic, compelling, and understandable (Bruine de Bruin et al., 2007).

4.3 REACH: MATCHING EVALUATION METHODS