Prioritize messages

esti-mated construct salience scores for the provider’s current month of performance. The system uses rules to create priority scores for five feedback message types which correspond with the five different messages and tailoring approaches included in Figure 13. The rules that establish the message priorities represent theoretical causal mechanisms within the tailoring knowledge-base are the following:

1. If knowledge is salient as a barrier, then deprioritize withholding feedback 2. If skills are salient a barrier, then prioritize withholding feedback

3. If negative feedback has been delivered repeatedly, then prioritize withholding feedback 4. If self-efficacy is a barrier, then prioritize withholding feedback

5. If knowledge is a barrier, then prioritize current-score format 6. If skills are a barrier, then prioritize current-score format

7. If the salience of skills as a barrier is above 100, increase the priority of current-score format by salience-100

8. If self-comparison is relevant, increase the priority of self-comparison format 9. If skills are a barrier, then deprioritize peer-comparison

10. If peer pressure is a barrier then deprioritize peer-comparison

11. If peer pressure is a not barrier and comparison is relevant, then prioritize peer-comparison

12. If self-efficacy is a barrier then deprioritize peer-comparison

13. If peer comparison and historical peer comparison are relevant, then increase the priority of historical peer comparison

After calculating the priority scores for each message type, for each individual and month of performance, these data are written to the performance database. After the conclusion of this assessment, a feedback tailoring system could generate a menu of tailored messages for a clinical supervisor to use for any individual who worked in any month during the two year period.

13.3 DISCUSSION

AF interventions can significantly impact the implementation of evidence-based practice.

However, significant research effort in recent decades has been unable to answer the ques-tions of how and when AF intervenques-tions will work.¹⁰ In response to Ivers and colleagues’ call for new approaches to AF research¹¹, I argue that AF research should address a promising and novel AF component: automated feedback message tailoring systems. The potential significance of the systems I envision is growing with our increasing understanding of how to use eHealth data for comparative effectiveness research¹³ and with the development of standardized terminologies199,200,206 and common theoretical frameworks^136,138 that create a basis for the use of computer-interpretable implementation knowledge. Furthermore, evi-dence about the use of computer-based message tailoring for health behavior change¹⁸⁴and a significant understanding of knowledge-based computer systems in biomedical informatics¹⁶³ reveal a foundation of knowledge and tools that could support the development feedback tailoring systems. Perhaps most importantly, systems that provide support for the practice of giving performance feedback could create a helpful structure for clinical supervisors, who deal with much uncertainty and unanticipated reactions when giving feedback to health-care providers. I view this work as supporting a recognition of the complexity in providing evidence-based care that calls for improved judgment on the part of providers, rather than improved rule-following.²⁰⁷

The system architecture that I describe represents a new mode of AF that is adaptive and may potentially withstand the complexity of the clinical environment and individual differences in provider capability and motivation to improve feedback effectiveness. I have

outlined an approach to using knowledge representation methods to adaptively tailor feed-back messages. A central part of this approach is to use the features of an individual’s clinical behavior to make rule-based inferences about the causal mechanisms through which feedback influences future behavior.

13.3.1 LIMITATIONS

This research has several limitations. First, I have not evaluated the cost of development and maintenance of such a message tailoring system. It would seem that the use of message tailoring systems would be most cost-effective in larger health systems where eHealth is already used to support performance measurement, but the cost-effectiveness of this approach is an important area of future research.

While I believe the rule-based approach to modeling construct salience that I used was adequate for the purpose of an exploratory analysis, it is likely to be inadequate for a large number of rules or to model the complexity of additional constructs. Using a Bayesian net-work to probabilistically model the netnet-work of factors influencing feedback effectiveness is likely to be a more viable approach. The benefits of such an approach have been discussed in the context of intelligent tutoring systems.²⁰⁸ The primary benefits of using a probabilistic approach are that it can adequately represent complex interactions resulting from multiple observations, and it can allow for the explicit representation of supervisors’ beliefs about feedback recipient’s barriers to behavior change. These beliefs could be modeled and revised over time as supervisors observe the effect of repeated feedback on individual performance, and change their beliefs about the effectiveness of feedback message designs for individual providers. This network would require a “recipient model” that could probabilistically rep-resent the feedback recipient’s capability, opportunity, and motivation factors with regard to each performance indicator. Additionally, the network could represent the recipient’s re-ported preferences for receiving feedback to estimate the probability that a tailored feedback message in a menu would lead to improved performance.

Another limitation is that the system’s ability to provide effective feedback is contin-gent on the ability of a supervisor to accurately perceive specific barriers for each

individ-ual. Supervisors’ ability to identify barriers can be expected to vary across supervisors and situations, and could contribute to the ineffectiveness of feedback. However, I note that, compared to feedback which is not tailored for specific barriers, we can reasonably expect that a message tailoring tool could provide relative improvement to the effect of feedback messages. Nevertheless, I do not know the extent to which making inaccurate assumptions about barriers to behavior change could negatively impact performance. For the purpose of our examples, I did not validate the tailoring rules that represent the relationship between a theoretical causal mechanism and the performance features found in the data.

13.4 CONCLUSION

Understanding how to tailor feedback messages holds significant potential for the improve-ment of AF interventions. In pursuing the goal of understanding how to develop tools for automated feedback tailoring, I plan to evaluate a prototype feedback message tailoring system in disparate AF intervention settings. This work is perhaps best characterized as embracing the complexity of healthcare by developing adaptive tools to target individual providers’ specific barriers to the adoption of evidence-based practice.

14.0 AUTOMATED FEEDBACK TAILORING IN LOW-RESOURCE