Refereed papers
Improving information management in
primary care: the proof is in the pudding
Virginia A Reed PhD
Research Associate Professor of Psychological and Brain Sciences, Dartmouth College, Adjunct Associate Professor of Community and Family Medicine, Dartmouth Medical School, Co-director, Center for Educational Outcomes, Dartmouth, Hanover, NH, USA
Karen E Schifferdecker PhD
Research Assistant Professor of Community and Family Medicine, Dartmouth Medical School, Hanover, NH, USA
Karen Homa PhD
Improvement Specialist, Leadership Preventive Medicine Residency Program, Dartmouth–Hitchcock Medical Center, Lebanon, NH, USA
Background
In 2002, Haux and his colleagues2described a vision of health care in 2013 in which health information systems would be crucial for documentation, communication,
medical knowledge, decision support, research and reporting. Well before 2013, information and com-munication technologies have become important in
ABSTRACT
Generalists in both the USA and UK have been at the forefront of improving information manage-ment skills, defined here as the abilities required to locate and utilise synthesised information for patient care that is accessible, current, relevant and valid.1 Over the past decade, a variety of interventions designed to improve knowledge and skills relative to information management has been implemented. The goals of training are for learners to demonstrate long-term retention of knowledge and skills gained and to be able to transfer this learning from the context of training into different situations and contexts, such as those encountered in the work-place. Thus, to conclude that learning has taken place, it is essential to study performance after learners have acquired knowledge and skills to see how well those have been retained and generalised. The current study builds on previous work con-ducted by the authors that described and evaluated an intervention designed to improve information management knowledge, skills and use of Web-based resources by participants from generalist primary care practices. This cross-over study found
that both groups of participants – those who received training initially and those who received training later – showed the same improvements when assessed 15 months and three months, respectively, after training. Given the definition of learning as ‘rela-tively permanent’, we wondered if these improve-ments would last.
Participants in the original three phases of the study completed questionnaires during each phase; for the current study they were asked to complete a fourth questionnaire administered 27 and 15 months, respectively, after their original training. All vari-ables showed non-significant differences between participants’ scores at the end of the original study, where learning was assessed as having occurred, and the current administration of the questionnaire. Demonstrated long-term retention of knowledge and skills and generalisation to the workplace show that the goals of training have been met.
many capacities in healthcare systems. The term ‘information management’ has been used to describe the role of technology in healthcare systems, including linkages with the National Health Service in the UK,3 and management of chronic diseases using telemedicine, clinical decision support and home-based monitoring in Australia.4Others have described it as a mechanism for establishing large datasets to facilitate research and the application of clinical trial evidence.5More nar-rowly, information management functions have been described as identifying needs, obtaining information to meet those needs and ascertaining the value of the information.6
In this last vein, Sackett and Rosenberg7noted that ‘As the quantity of valid evidence increases so does the requirement for each of us to develop the skills necessary to assimilate, evaluate and make best use of that evidence for patients’. Slawson and Shaughnessy1 elaborated on this by defining a broader skill set of information management – the abilities required to locate and utilise synthesised information for patient care that is accessible, current, relevant, and valid – rather than the narrower focus on best evidence alone. Family medicine and general practitioners have been leaders in focusing on information management skills8–11and have identified lack of knowledge and skills in the use of electronically-based resources as one factor affecting providers’ use of evidence-based resources.1,9,10,12Various interventions aimed at im-proving information management knowledge and skills have been implemented,13–18 with most reporting positive findings. In each case, learning was assumed to have occurred. But what is learning? Learning has been defined as ‘relatively permanent change in behavior potentiality which occurs as a result of reinforced practice’.19The goals of training are for learners to demonstrate long-term retention of knowledge and skills and to transfer or generalise this learning into different situations and contexts, such as the work-place.20,21 Thus, to conclude that learning has taken place, it is essential to study how well learners have retained and generalised knowledge and skills.21
Relatively little research has studied retention and transfer relative to information management learning. Various research designs17,22–24and timeframes18,22,24,25 have been used, but few have gone beyond 12 months. In a cross-over study, the current authors described improvements in knowledge, skills and frequency of Web-based resource use for two groups of participants when assessed 15 months and three months after training.26Given the definition of learning as ‘rela-tively permanent’, we wondered if these improve-ments would last.
Method
The original study26began in July 2004 and involved 24 primary care practices located in New Hampshire and Vermont. Practices were matched by size, specialty and location. A cross-over design was used, with one practice in each pair randomly assigned to the initial intervention group and one to the delayed training group. Dartmouth College’s Committee for the Pro-tection of Human Subjects approved the human subjects protocol.
Prior to randomisation, each practice completed baseline measures, then received two computers and high-speed Internet connections to ensure that all sites had at least the minimum of equipment and access to the Web. At baseline, 67% of the practices had high-speed connections and 54% had limited computer resources, numbers in line with other descriptions of primary care information technology infrastructure.27 No differences in resources were found between the groups: high speed Internet connections (2= 0.540, P= 0.46) and computer resources (2= 0.691,P = 0.43).
The participants consisted of small teams – one to three healthcare providers and clinical and office staff – selected by the practice, making a total of 32 par-ticipants in each group. Training focused on know-ledge and skills related to information management and on creation of an improvement plan to incorpor-ate information management into the practice via use of Web-based resources for clinical decision making and patient education. The purpose of the teams was to enhance perspectives on care delivery processes in the practice, which helped inform the development of the practice-based improvement plans.28,29
Training consisted of two day-long workshops that utilised a variety of experiential learning method-ologies.26Workshop I focused on efficiently accessing Web-based resources and on developing initial prac-tice-specific plans to use these resources. Between the workshops, participants completed a practice-specific search exercise and implemented initial plans. A phone call prior to the second workshop was used to answer questions and to review progress. Workshop II focused on reviewing search strategies and on revising and expanding improvement plans. One month after Workshop II, a site visit was made to each practice to review progress.
self-assessed knowledge and skills, b) frequency of resource use, c) communication with patients about using resources for health information, d) perceptions about the use of resources and e) incorporating use of resources into daily practice. A self-report questionnaire was utilised mainly for reasons of practicality, with know-ledge of the pluses and minuses of such a tool. As indicated in Figure 1, the questionnaire was admin-istered three times during the original study.
In the original study,26participants in both initial and delayed training groups reported increases in knowledge, skills and frequency of use of electronic evidence-based resources for patient care decisions that were associated with the training intervention. Participants also reported increases in frequency of resource use for patient education, communication with patients about evidence-based resources and incorporating use of these resources into daily prac-tice. These increases occurred for both groups over the initial timeframe of the study and were not related to training.
To determine whether improvements were sus-tained (the focus of the current study) the question-naire was administered a fourth time (follow-up III), as shown in Figure 1. This reflected a period of no further training for approximately 27 months for the initial training group and 15 months for the delayed training group.
Analysis
Prior analyses26had examined the differences between initial implementation and delayed implementation participants’ scores at baseline and follow-up I, and scores at baseline and follow-up II. These analyses assessed the equivalency of both study arms, as well as initial learning for both groups of participants. Since both groups reported scores that were substantively the same after training, it was concluded that there was no significant effect of group. As the current question of interest was whether the improvements reported
initially would last, the present analysis utilised paired t-tests to examine the difference between all partici-pants’ scores at follow-up II and follow-up III. Main-tenance of learning at follow-up III would be indicated by no change in scores (non-significant t-tests). To ensure that such findings indicated maintenance of learning rather than Type II error due to smallN,post hoceffect size and power issues were addressed.
Results
Of the 64 original participants, 40 (63%) completed questionnaires at each of the initial three survey points. Thirty participants (47% of original and 75% of those completing three prior surveys) completed the follow-up III questionnaire. In most cases, par-ticipants who had completed the three prior surveys but not the follow-up III questionnaire had left the practice.
The participants were predominately female (74%) and averaged 50 years of age (standard deviation (SD) 9.3). They had been with their current practice for an average of 10.5 years (SD 8.2).
Table 1 displays the mean scores, standard devi-ations, and pairedt-test results for all variables.
No paired t-tests were significant, indicating that scores at follow-up III did not differ significantly from scores at follow-up II. Specific results for each con-ceptual domain are described below.
Knowledge and skills
These items asked participants how well they could do particular Web tasks. In the original study, each of the four items representing knowledge and skills showed improved scores associated with training. In the present analysis, there were no significant differences between scores at follow-up II and follow-up III for any of the items: a) finding information on the Web, b) finding educational material on the Web for patients, c) a sum of specific search skills and d) skills using a variety of Web-based resources.
Frequency
These items asked participants to estimate how often in a typical month they used a variety of Web-based resources for patient care decisions and patient edu-cation. In the original study, the item indicating how often participants reported using a variety of Web-based resources for patient care decisions showed im-proved scores associated with training. All participants
reported a significant increase in the frequency with which they used Web-based materials for patient education; this occurred over the initial timeframe of the study and was not related to training. In the current analysis, there were no significant differences between scores at follow-up II and follow-up III for either item.
Communication
This item asked participants to estimate how often they referred patients to specific Web sites. In the original study, all participants reported a significant increase; this occurred over the initial timeframe of the study and was not related to training. In the current analysis, there was no significant difference between scores at follow-up II and follow-up III for this item.
Perception
These items asked participants how they felt about their experiences, given their current patients and practice setting. In the original study, the item indi-cating the degree to which participants like it when patients bring in Web-based information showed an increase associated with training. Additionally, there were two items with non-significant findings: partici-pants reported no difference in feeling the Web could help them provide patients with either better clinical care or better health education materials. In the present analysis, there were no significant changes between scores at follow-up II and follow-up III for any of these items.
Incorporation
Table 1 Mean scores, standard deviations and pairedt-tests between follow-up II and follow-up III
Measures Follow-up II Follow-up III
Mean (SD) Mean (SD) t(df) P-value
Knowledge and skills
Ability to find information on the Web 3.83 (0.91) 3.70 (0.75) 0.78 (29) 0.442
Ability to find educational materials on the Web for patients
3.36 (1.22) 3.36 (1.11) 0.00 (24) 1.000
Composite of skills for finding information on the Web for patients
11.62 (5.05) 12.08 (4.33) –1.13 (25) 0.269
Skills in using a variety of Web-based resources
21.88 (7.66) 19.47 (9.78) 1.61 (33) 0.117
Frequency of resource use
How often in a typical month I use Web-based materials for patient care decisions
12.35 (3.41) 12.50 (4.52) –0.21 (25) 0.833
How often in a typical month I use Web-based materials for patient education
2.16 (1.14) 2.52 (1.00) –1.89 (24) 0.071
Communication with patients about using resources for health information
How often I refer patients to specific Web sites
2.93 (1.05) 2.89 (1.03) 0.25 (27) 0.802
Perceptions about the use of resources
I like it when patients bring in information they find on the Web
3.88 (0.67) 3.72 (0.89) 0.70 (24) 0.491
I feel that the Web does or could help us provide better health education materials to patients
4.17 (0.59) 4.00 (0.83) 1.04 (29) 0.305
I feel that the Web does or could help us provide better clinical care to patients
3.97 (0.63) 3.90 (1.01) 0.34 (28) 0.738
Incorporating use of resources into daily practice
I use the Web during my work hours at this practice for patient education
3.04 (1.06 ) 3.32 (1.11) –1.43 (24) 0.166
I use the Web during my work hours at this practice for patient care decisions
3.39 (1.31) 3.70 (1.11) –1.50 (22) 0.148
I think there is leadership at this practice that encourages using Web resources for patient education
3.50 (0.89) 3.67 (0.92) –0.81 (23) 0.426
I think there is leadership at this practice that encourages using Web resources for patient care decisions.
Discussion
The research question of interest in this study was whether the improvements in information manage-ment knowledge and skills that occurred in the original study had persisted. All items showed non-significant differences between participants’ scores at follow-up II and follow-up III, indicating that the improvements were maintained.
These findings are subject to a number of limi-tations. The first is the small sample size combined with an overall response rate of 47% of the original participants. Three-quarters of those who completed the first three questionnaires also completed the fourth questionnaire. Given that the study was conducted over more than three years, turnover in practices was expected. The major concern of the small sample size is assuring that the findings of non-significant differ-ences support the conclusion of maintenance of learn-ing rather than Type II error. Thus, it is important to consider issues of effect size and power. Cohen30has described parameters of small (0.2), medium (0.5) and large (0.8) effect sizes, and has suggested that a power level of 0.80 is the conventional level needed when an
-level of 0.05 is used to denote significance. Using the scores from baseline and follow-up III (a conservative estimator for the original study due to the smallerN available at follow-up III), we calculated the actual effect size of items. These calculations showed that nearly all of the effect sizes achieved, based on our actual sample of 30 participants at follow-up III, were >0.5, indicating at least a moderate effect. In these
instances, the actual power level was well above the recommended level of 0.80 for all items where we indicated that change had taken place in the original study. The results of these calculations provide assur-ance that the small sample size did not unduly bias the findings.
Another potential limitation to the study concerns internal validity. It is likely that changes in expec-tations and the environment for use of Web-based healthcare resources occurred over the 37 months of the study. The original study’s findings, illustrated in Figure 2, show two types of increases: those associated with training and those not related to training that were likely due to passage of time and enhanced computer technology in the practices. The findings from the current study, also schematically represented in Figure 2, show no reported change over the inter-vening time. Given adequate statistical power, these patterns of change provide some evidence that the intervention had an effect and that other changes were associated with the practices having enhanced com-puter equipment.
Another limitation of the study concerns general-isability. Many practices in New Hampshire and Vermont are rural and may not be representative of practices in other areas.
Self-reports are often considered a limitation. Shrauger and Osberg31have noted that, given the appropriate circumstances, judgements of one’s own behaviour can be as effective as those made in other ways. This opinion has been seconded by others who have observed that self-report can provide valid and valuable measures when employed in sensible research designs,32such as
longitudinal studies,33 and where observational or other more objective measures would be obtrusive and impractical.34Howard and his colleagues35reported on a series of studies aimed at testing and comparing behavioural and self-report measures. They noted that while self-report measures may be ‘contaminated’ by a variety of response style effects,36behavioural measures are subject to distortion due to variety of types of variance.37They noted both that it was costlier and less efficient to collect behavioural measures than self-report measures, and that behavioural measures yielded infor-mation that was of no better quality.35
It is possible that social desirability or response bias could have played a role in participants’ responses. It has been noted that self-deception is minimised by use of behavioural items,38such as those used in the study questionnaire. As noted earlier, relatively little re-search has addressed retention and transfer of knowl-edge and skills relative to information management. Schmitt34has opined that it is appropriate in newly developing areas of research to use methods and research designs that might be less appropriate in more mature areas of research.
The goals of training are long-term retention of knowledge and skills, and transfer of learning into other environments, such as the workplace.21 Work-place performance is influenced by many factors including learner characteristics, the type of skills learned and workplace culture and environment.20It has been argued that successful learning and transfer in organ-isations requires: a) knowledge of guiding principles, culture and purpose; b) infrastructure including time and support to learn; c) tools needed to do the job.39 These are among the factors that should be considered for educational interventions focused on information management in clinical practices. That said, the proof is in the pudding: in this study, demonstrated long-term retention of knowledge and skills and generalis-ation to the workplace show that the goals of training have been met. Further work will explore the relative contributions of the educational intervention and the organisational culture to this success.
ACKNOWLEDGEMENTS
This work was supported by the National Library of Medicine grant 5 G08 LM 008110, the Office of Community-based Education and Research at Dartmouth Medical School and Dartmouth Biomedical Libraries.
REFERENCES
1 Slawson DC and Shaughnessy AF. Teaching evidence-based medicine: should we be teaching information management instead?Academic Medicine2005;80:685–9. 2 Haux R, Ammenwerth E, Herzog W and Knaup P.
Health care in the information society: a prognosis for
the year 2013.International Journal of Medical Inform-atics2002;66:3–21.
3 Howcroft D and Mitev N. An empirical study of Internet usage and difficulties among medical practice manage-ment in the UK.Internet Research: Electronic Networking Applications and Policy2000;10:170–81.
4 Celler BG, Lovell NH and Basilakis J. Using information technology to improve the management of chronic disease.
Medical Journal of Australia2003;1998:242–6.
5 Cordell WH, Overhage JM and Waeckerie JF. Strategies for improving information management in emergency medicine to meet clinical, research, and administrative needs.Annals of Emergency Medicine1998;31:172–8. 6 Williamson JW, German PS, Weiss R, Skinner EA and
Bowes III F. Health science information management and continuing education of physicians: a survey of US primary care practitioners and their opinion leaders.
Annals of Internal Medicine1989;110:151–60.
7 Sackett DL and Rosenberg WMC. On the need for evidence-based medicine.Journal of Public Health Medi-cine1995;17:330–34.
8 Ebell MH. Point-of-care information that changes prac-tice: it’s closer than we think.Family Medicine 2003; 35:261–3.
9 Guyatt GH, Meade MO, Jaeschke RZ, Cook DJ and Haynes RB. Practitioners of evidence based care.British Medical Journal2000;320:954–5.
10 McColl A, Smith H, White P and Field J. General practitioners’ perceptions of the route to evidence based medicine: a questionnaire survey.British Medical Journal
1998;316:361–6.
11 White B. Making evidence-based medicine doable in everyday practice.Family Practice Management2004;11: 51–8.
12 Wilson SM. Impact of the Internet on primary care staff in Glasgow.Journal of Medical Internet Research1999; 1:e7, www.jmir.org/1999/2/e7/ (accessed 07/03/08). 13 Bradley P, Oterholt C, Herrin J, Nordheim L and
Bjorndal A. Comparison of directed and self-directed learning in evidence-based medicine: a randomised con-trolled trial.Medical Education2005;39:1027–35. 14 Brettle A. Information skills training: a systematic review
of the literature. Health Information Library Journal
2003;20(Suppl 1):3–9.
15 Crites G, Chrisagis X, Patel V, Little D and Drehmer T. A locally created EBM course for faculty development.
Medical Teacher2004;26:74–8.
16 Schilling K, Wiecha J, Polineni D and Kahalil S. An interactive web-based curriculum on evidence-based medicine: design and effectiveness.Family Medicine2006; 38:126–32.
17 Straus SE, Ball C, Balcombe N, Sheldon J and McAllister FA. Teaching evidence-based medicine skills can change practice in a community hospital. Journal of General Internal Medicine2004;20:340–43.
20 Baldwin TT and Ford JK. Transfer of training: a review and directions for future research.Personnel Psychology
1988;41:63–105.
21 Schmidt RA and Bjork RA. New conceptualizations of practice: common principles in three paradigms suggest new concepts for training.Psychological Science1992;3: 207–17.
22 Alper BS and Vinson DC. Experiential curriculum improves medical students’ ability to answer clinical questions using the Internet.Family Medicine2005;37: 565–9.
23 Ross R and Verdieck A. Introducing an evidence-based medicine curriculum into a family practice residency: is it effective?Academic Medicine2003;78:412–17. 24 Smith SA, Ganschow PS, Reilly BM et al. Teaching
residents evidence-based medicine skills: a controlled trial of effectiveness and assessment of durability.Journal of General Internal Medicine2000;15:10–15.
25 Urquhart TJ, Durbin J and Ryan J. Changes in infor-mation behavior in clinical teams after introduction of a clinical librarian service.Journal of the Medical Library Association2007;95:14–22.
26 Schifferdecker KE, Reed VA and Homa K. A training intervention to improve information management in primary care.Family Medicine2008;40:423–32. 27 Andrews JE, Pearce KA, Sydney C, Ireson C and Love M.
Current state of information technology use in a US primary care practice-based research network. Inform-atics in Primary Care2004;12:11–8.
28 Nelson EC, Batalden PB and Godfrey MM.Quality by design: a clinical microsystems approach. San Francisco, CA: Jossey Bass, 2007, pp.106–23.
29 Nelson E, Batalden PB and Lazar JS (eds). Practiced-based learning and improvement: a clinical improvement action guide. Oak Brook, IL: Joint Commission Re-sources, 2007, pp.81–94.
30 Cohen J. A power primer.Psychological Bulletin1992; 112:155–9.
31 Shrauger J and Osberg TM. The relative accuracy of self-prediction and judgments by others in psychological assessment.Psychological Bulletin1981;90:322–51. 32 Howard GS. Why do people say nasty things about
self-reports? Journal of Organizational Behavior 1994;15: 399–404.
33 Spector P. Using self-report questionnaires in OB re-search: a comment on the use of a controversial method.
Journal of Organizational Behavior1994;15:385–92. 34 Schmitt N. Method bias: the importance of theory and
measurement.Journal of Organizational Behavior1994; 15:393–8.
35 Howard GS, Maxwell SE, Wiener RL, Boynton KS and Rooney WM. Is a behavioral measure the best estimate of behavioral parameters? Perhaps not.Applied Psycho-logical Measurement1980;4:293–311.
36 Millham J and Jacobson LL. Social desirability and need for social approval. In: London H and Exner J (eds)
Dimensions of Personality. Wiley, 1978 pp.365–90. 37 Campbell DT and Stanley JC. Experimental and
quasi-experimental designs for research on teaching. In: Gage NL (ed)Handbook of Research on Teaching.Rand McNally, 1963, pp.171–246.
38 Zerbe WJ and Paulhus DL. Socially desirable responding in organizational behavior: a reconception.The Acad-emy of Management Review1987;12:25–64.
39 Cowley C, Daws L and Ellis B. Health informatics and modernisation: bridging the gap.Informatics in Primary Care2003;11:207–14.
CONFLICTS OF INTEREST
None.
ADDRESS FOR CORRESPONDENCE
Virginia A Reed
HB 7016, Dartmouth College 46 Centerra Parkway
Lebanon, NH 03766 USA
Tel: +1 603 653 3449
Email: [email protected]
