Teaching a Systematic Search Strategy Improves Literature
Retrieval Skills of Pharmacy Students
Kathryn L. Grant, Richard N. Herrier and Edward P. Armstrong
College of Pharmacy, The University of Arizona, P.O. Box 210207 Tucson AZ 85721-0207The present investigation tested a lecture/computerized demonstration of a systematic approach to computerized literature retrieval. Forty-eight students in their third professional year who had been taught how to use OVID (version 3.0) in a one hour class, were tested on their ability to construct search strategies using a paper/pencil examination. Results of the pretest demonstrated that students were able to define Boolean operators but were unable to use them in constructing search strategies. After a lecture and computerized demonstration of search strategies was presented, homework using OVID was assigned. A paper/pencil posttest was given after the homework was returned and discussed. The posttest showed a statistically significant increase in scores (P<0.001). The importance of improving pharmacy students’ literature retrieval skills and having this level of training early in their curriculum is discussed.
Information networks straddle the world. Nothing remains concealed. But the sheer volume of information dissolves
the information. We are unable to take it all in.
Gűnther Grass
The greatest crisis facing modern civilizations is going to be how to transform information into structured
knowledge.
Carlos Fuentes
INTRODUCTION
The level of competence that pharmacy students require for computerized literature retrieval has not been established. Lack of access to computerized searching technology and lack of time to conduct searches themselves are the most frequent reasons health care professionals use intermediar-ies such as librarians and drug information specialists to perform literature searches(1). However, drug information centers are curtailing services(2) and librarians have real-ized their manpower limits to provide intermediary search-ing on a timely basis for all health care practitioners(3). At
the same time, university medical libraries have invested considerable resources in making user-friendly computer-ized bibliographic databases such as MEDLINE, one of the bibliographic databases of Medlars developed by the Na-tional Library of Medicine (NLM), readily available to health care professionals(3).
In training students and health care professionals to use computerized bibliographic databases, research has shown that access to these databases does not assure competence to use them effectively or efficiently(4-6). More extensive training not only leads to better search retrieval but the likelihood that computer searching will be continued after graduation(5-10).
In addition, bibliographic software screen layouts affect the use of the system. Searching MEDLINE databases using simplified commands with search aids conveniently located on the screen is achieved at the expense of complete under-standing of the underlying search principles, especially for novice users. Without an understanding of the limitations of controlled vocabularies and how the Medlars databases are indexed, literature searches can often be unsuccessful(11). Another limitation is that several studies have verified that
Fig. 1. Pretest.
computer searching retrieves fewer citations than manual searching(12-14).
It is important for pharmacy faculty to assess student literature retrieval skills. During recent curriculum revi-sions at the University of Arizona College of Pharmacy, faculty identified insufficient computer-retrieval skills of students as a significant curricular issue. In addition, the faculty recommended that the enhanced bibliographic search training be taught much earlier in the curriculum so students would be skilled at primary literature retrieval throughout their four-year doctor of pharmacy training. Therefore, the purpose of this study was to evaluate the impact on literature retrieval skills using a specifically designed educational effort that focused on underlying Medlars databases’ prin-ciples and a systematic approach to constructing search strategies.
METHODS
The study population was 48 students enrolled in their third professional year of the pharmacy curriculum. The semester
Fig. 2. Lecture Outline.
before this investigation, these students were required to complete a one hour bibliographic retrieval training course with OVID (version 3.0. copyright 1993-1995, OVID Tech-nologies, Inc.). This study began by requiring students to complete a pretest that asked them to write specific search strategies locating pertinent articles for answering two clini-cally oriented questions and that evaluated their knowledge of Boolean operators, in particular, the logical connectors “and” and “or” (Figure 1)(15).
Five weeks later, a 50-minute lecture on a systematic approach to developing search strategies was presented to the students (Figure 2). This was combined with a 50-minute on-line demonstration with OVID using a notebook computer connected to a local area network (LAN) line, an overhead projector and LCD screen. The primary goal of the lecture and demonstration was to emphasize the importance of a system-atic approach to literature retrieval compared to haphazardly entering terms and scanning hundreds of articles. Secondary goals were to make sure the students understood the differ-ence between using MEDLINE’s controlled vocabulary called MeSH (Medical Subject Headings) and free-text searching; the inherent problems with MeSH headings; when to use MESH terms, free text terms, subheadings and explosions; and the appropriate use of Boolean operators. The students were shown that combining MeSH terms and free text is imperative to retrieve the maximum number of pertinent citations. The students were then shown appropriate ways to further refine the retrieval using limits such as “human,” “English” and age groups. The lecture outline is presented in Figure 2. Figures 3– 7 provide a selection of some of the examples used during the demonstration to emphasize the key points. The lecturer
Fig. 3. Demonstration of a simple exhaustive search on a single medication -all the searches were conducted in the 1995 to 1991 MEDLINE database using OVID (version 3.0) in August, 1995.
Fig. 4. Demonstration of an exhaustive search when a suitable MeSH (Medical Subjects Heading) term is not available.
presented the answers to the pretest during the demonstration, but the pretests were not returned. At the end of the demon-stration, students received a homework assignment to practice the principles presented.
Eight different homework assignment sets were distrib-uted to the 48 students. The homework was designed to give examples of exhaustive and precise literature searches. Each homework assignment had four questions in which the students were to perform a literature search. They were required to print out their search strategies and list of articles the strategy retrieved. Two weeks later, after the homework was returned and graded, a third 50-minute session reviewed the homework assignments.
One week later a midterm exam (posttest) was admin-istered. The posttest included two questions on Boolean operators and two new clinical questions similar to those presented in the pretest. Students were again asked to write specific search strategies that would retrieve pertinent ar-ticles (Figure 8).
Fig. 5. Demonstration of an exhaustive literature search which would include a route of administration.
Search strategy grading was done by the authors using pre-established scoring criteria. If the examinations had been conducted in a computer laboratory, the answer would have been graded as to the number and pertinence of the retrieved articles (as was done with the homework assign-ment). However, since the examinations were paper and pencil tests, each search statement expected to be written by the student within the search strategy was assigned a specific point value. Students were not penalized for search state-ments beyond what was necessary to complete an adequate search. Students did not have to conform to the exact strategy worked out by the instructor. Any reasonable strategy the student provided was run on OVID and, if successful in retrieving the same citations, full credit was given.
The scores were analyzed with descriptive statistics including means with standard deviations, medians and modes. Mean scores were compared using a two-tailed student paired r-test. Pearson’s coefficient of correlation was used to assess the relationship between pretest scores and posttest scores, as well as homework scores and posttest scores(16).
RESULTS
Forty-eight students completed the pretest. Their overall average score and average scores on each question are presented in Table I. The test revealed that students could
Fig. 6. Demonstration of the importance of combining MeSH and free text terms.
Table I. Pretest and posttest scores
Values Pretest Posttest
Overall Scores (perfect score = 60 points)
Mean±standard deviation 31.5±8.0 56.8±4.1a
Percent 52.5 94.7
Median 32 59
Mode 32 60
Question #1 mean±standard deviation 9.1±2.8 10 (%) (perfect score = 10) 91% 100% Question #2 mean±standard deviation 8.4±3.6 10
(%) (perfect score = 10) 84% 100% Question #3 mean±standard deviation 8.2±2.2 19.1±1.6
(%) (perfect score = 20) 41% 95.5% Question #4 mean±standard deviation 5.7±3.3 17.7±3.3
(%) (perfect score = 20) 28.5% 88.5%
aP=<0.001 (two-tailed student paired t-test (Microsoft Excel©)
define Boolean operators. Pretest responses on the next two questions showed that being able to define the operators did not necessarily translate into an ability to construct an efficient search statement.
The homework assignment was worth 100 points and scores ranged from 47 to 100. The mean score was 91.3±10.5 with a median of 93 and a mode of 100. Only two students received scores less than 80 points.
One week after the homework was discussed and re-turned, all students completed a midterm examination in-cluding a four question posttest on constructing literature searches. The group mean scores overall and on each ques-tion are listed in Table I. Students improved significantly (P<0.001) compared with the pretest. The two-tailed stu-dent paired t-test showed statistical significance for the group means at a level of P<0.001.The Pearson Correlation
Fig. 8. Posttest.
describing the relationship between the pretest and posttest was 0.35. Although a direct relationship between the scores existed, that relationship lacked statistical significance. The Pearson Correlation comparing the homework to the posttest was 0.08. again demonstrating a lack of a statistically signifi-cant relationship between the homework and the posttest.
No student score decreased from the pretest to the posttest. Only four students improved their scores by less than 15 points between the pre-and posttest. The students receiving the lowest scores on the pretest showed the greatest improvement, with one student increasing their score from 12 points on the pretest to 60 points (a perfect score) on the posttest.
DISCUSSION
The results showed that exposing students to a systematic approach to constructing strategies improved their ability to write search strategy statements. Evidence that the level of training will actually correlate to the ability to retrieve relevant citations is limited. Two studies determined that literature
retrieval was greatest when conducted by experienced or well-trained searchers. Kirby and Miller reviewed 52 literature searches done by untrained biomedical personnel searching MEDLINE using Colleague®, a modem accessible search
program(5). Compared with searches run by experienced medical librarians, 60 percent of searches by untrained person-nel were incomplete. Most unsuccessful searches (21/31, 68 percent) were the result of inadequate search strategies. Seven (23 percent) were unsuccessful because users did not ad-equately understand the system’s keyboard commands. In addition, Poisson found that medical staff performed better literature retrieval if taught to search by experienced librar-ians(6). Their training emphasized MeSH vocabulary, tech-niques of exploding, use of check tags and subheadings that were all components of the current research.
Exposing these pharmacy students to more computer-ized search training and experience should not only increase their skills but also increase the likelihood that they will use these skills during clerkship experiences and after gradua-tion. Such a correlation was found by Pao and colleagues with third year medical students. Pao showed a strong relationship existed between the quantity of computerized search experience in the medical school curriculum and the frequency of use in medical practice(7). For pharmacy stu-dents, Ikeda and colleagues trained 151 pharmacy students over a four-year period on key board commands and formu-lating search strategies with Boolean logic(10). They sur-veyed 151 graduates and found that in the 90 respondents 72 percent used some form of computerized searching and 42 percent did their own searches.
Whether we should consume valuable pharmacy cur-riculum time teaching literature retrieval skills has been debated. For one, should pharmacists spend their time as practitioners doing literature searchers? Do librarians do a better job at retrieving pertinent citations? Research com-paring literature retrieval by drug information specialists to medical librarians has shown that with adequate end user training, drug information specialists can do as well as, if not better than, medical librarians in retrieving relevant cita-tions to answer drug information inquiries. Schneiweiss and Comer compared literature retrieval by a medical librarian and a drug information pharmacist using 20 sequential questions received by a drug information center(9). The drug information pharmacist retrieved more relevant refer-ences than the librarian. The librarian retrieved relevant citations to help answer 30 percent of the questions and the drug information pharmacist retrieved relevant citations for 75 percent of the questions. However, how the drug infor-mation pharmacist was trained was not documented. Wanke and Hewison compared the usefulness of MEDLINE searches done by three reference librarians with extensive NLM online training to a drug information specialist who was taught as an “end user”(6). The drug information specialist’s end user training involved a detailed course that included a systematic approach to constructing a search strategy. The searching system used was modem access directly to NLM that charges a variable fee for searches. The pharmacist’s searches cost $9.00 more than the librarians’ for the 48 literature searches performed. However, despite the slight increase in cost per search, the authors concluded that searches done by a pharmacist trained in this detail were as useful as those done by the reference librarian.
Thus, if adequately trained, drug information special-ists can search the literature with equal, if not greater
efficacy, as compared to librarians, but that does not answer the question of whether pharmacy students should receive more extensive literature search retrieval training. Several factors support the need for more extensive pharmacy stu-dent training. For one, drug information centers are closing or curtailing services leaving many pharmacy practitioners without access to these centers(2,17,18). For another, medi-cal librarians cannot fulfill all health care professionals literature searching needs. Sewell and Teitelbaum called this the “law of saturation(3).” As more literature retrieval requests are received by health sciences libraries, the de-mand for librarians increases. However, in the current eco-nomic climate, library staffs cannot experience unlimited growth. To deal with the inability to add personnel, many health sciences libraries have decided to make online ser-vices readily available. Library staff then provide assistance with only the most difficult searches. Finally, pharmacists are experiencing increased access to bibliographic data-bases. Frequently, MEDLINE is available free or at reason-able prices from regional medical or health sciences librar-ies. Grateful Med®, a user friendly PC software available
from the NLM, is relatively inexpensive (less than $50) and is available through the Internet. Literature search capabili-ties are also available free through Helix® (St. Louis College
of Pharmacy from a grant from Glaxo) which offers its users IPA® (International Pharmaceutical Abstracts. ASHP.
Bethesda, MD). Given this expanding access to computer-ized retrieval and declining access to literature search inter-mediaries such as librarians and drug information special-ists, pharmacists will frequently need to perform their own searches to obtain adequate information to provide direct patient care. To ensure that future practitioners have the literature retrieval skills they need, more extensive training needs to be part of the pharmacy curriculum.
In the present study, while students” performance in-creased dramatically, certain limitations must be considered in attempting to apply this investigation to other situations. Although the examination was using a paper and pencil exami-nation rather than directly using computer software, scoring was adjusted to compensate for the testing method. Points were not subtracted for incorrect terminology or for state-ments that could not have been determined to be incorrect by students, because they did not have online responses to en-tered search statements. In addition, students’ prior computer experience was not measured. Given that improvement oc-curred between the pretest and posttest, even for students who had received higher scores on the pretest, differences in prior computer experience were probably not significant. Finally, exposure to the pretest may have contributed to the students’ performance on the posttest. However, the eight week interval between the pre- and posttest may have decreased any effect of testing on student performance.
This study confirms that teaching a systematic approach to constructing search strategies as a complement to key-board command training improved pharmacy student per-formance. Although, not systematically documented, the students studied here showed improved literature retrieval abilities during a required fourth year drug information clerkship. Compared to previous clerkship students, the group that received this more intensive training level re-quired less preceptor assistance, appeared more confident when completing a search, and were more likely to find obscure citations.
CONCLUSION
This research adds to the body of literature demonstrating the importance of training students more rigorously in search strategy techniques. Students need to be exposed to bibliographic literature searching early in their pharmacy education. As with any skill, pharmacy students need oppor-tunities to practice literature retrieval before embarking on clerkship experiences. Whenever possible, each pharmacy course should incorporate opportunities for students to use bibliographic searching from the first professional year onward.
As medical library collections move from repositories of “secondhand knowledge” to “online knowledge servers,” the health care professionals must keep pace with their abilities to use the new technology(19). Computer literacy and search strategy knowledge will be key for fast and efficient use of the medical libraries of the future.
Am. J. Pharm. Educ., 60, 281-286(19%); received 3/15/96, accepted 5/17/96, References
(1) Cornick.D., “Being an end-user is not for everyone,” ONLINE,
(March 1989).
(2) Rosenberg. J.M., Fuentes. R.J., Starr C.H., Kirschenbaum. H.L. and McGuire. H., “Pharmacist-operated drug information centers in the United States,” Amer. J. Health-Sys. Pharm., 52, 991-996(1995). (3) Sewell. W. and Teitelbaum. S., “Observations of end-user online
searching behavior over eleven years,” J.Am. Soc. Inf. Sci., 37 ,324-345(1986).
(4) Abate. M. A., Shumway. J.M. and Jacknowitz.A.I., “Use of two online services as drug information sources forhealth professionals.” Meth-ods Inf. Med., 31,153-158(1992).
(5) Kirby. M. and Miller. N., “MEDLINE searching on Colleague: Rea-sons for failure or success of untrained end users,” Med. Ref.Serv. Q., 5, 17-30(1986).
(6) Poisson. E.H., “End-user searching in medicine.” Bull. Med. Libr. Assoc., 74, 293-299(1986).
(7) Pao. M.L., Gerfsheim. S.F., Barclay. M.L., Woolliscroft. J.O., McQuillan. M. and Shipman.B.L., “Factors affecting students’ use of MEDLINE,” Comput. Biom. Res., 26, 541-555(1993).
(8) Wanke. L.A. and Hewison. N.S., “Comparative usefulness of MEDLINE searches performed by a drug information pharmacist and by medical librarians,” Am. J. Hosp. Pharm., 45,2507-2510(1988). (9) Schneiweiss. F. and Comer. J.B., “Using MEDLINE to answer drug
information questions: Hospital library or drug information center?” Drug. Inf. J., 15, 94-100(1981).
(10) Ikeda. N.R., Cushing. H. and Schwartz. D.G., “Impact of end-user search training on pharmacy students: A four-year follow-up study,”
Bull. Med. Libr. Assoc., 80,124-130(1992).
(11) Lowe. H.J. and Barnett. G.O., “Understanding and using the medical subject headings (MeSH) vocabulary to perform literature searches.”
JAMA, 271,1103-1108(1994).
(12) Kirpalani. H., Schmidt. B., McKibbon. K.A., Haynes. B. and Sinclair. J.C., “Searching MEDLINE for randomized clinical trials involving care of the newborn,” Pediatrics, 83, 543-546(1989).
(13) Bernstein. F., “The retrieval of randomized clinical trials in liver diseases from the medical literature: Manual versus Medlars search,”
Control Clin. Trials, 9, 23-31(1988).
(14) Poynard. T. and Conn. H.O., “The retrieval ofrandomized clinical trials in liver disease from the medical literature: A comparison of Medlars and manual methods.” ibid., 6,271-279(1985).
(15) Morton. D., “Boolean AND (searching OR retrieval).” Online, 17, 57(1993).
(16) Anon., Microsoft Excel©, Ver. 5.0a, Microsoft Corp., Seattle WA (1985-1993).
(17) Hallgarth. S.A., Dunham. D.B., Goshman. L.M. and Vermeulen. L.C., “Evaluation of the effects of implementing a voice-mail system on workload in a drug information center,” ASHP Midyear Clinical Meeting, 30(Dec) P-169(E)(1995).
(18) Anderson. P.O. “Establishment of a 900 telephone number at a university-based drug information service,” Amer. J. Hosp. Pharm., 51, 2684-2687(1994).
(19) Matheson. N.W. and Welch. W.H., “The idea of the library in the twenty-first century,” Bull. Med. Libr. Assoc., 83,1-7(1995).
