An evaluative case
study of a real-time
online reference service
Louise Kloss and
Yin Zhang
The authors
Louise Klossis at Morley Library, Painesville, Ohio, USA. Yin Zhangis at the School of Library and Information Science, Kent State University, Kent, Ohio, USA.
Keywords
Reference services, Information, Libraries, Services
Abstract
This case study evaluated a real-time online interactive reference service. Three data gathering methods were used. First, general usage statistics were generated electronically from the service provider. Second, patron evaluations were collected electronically. Third,
questionnaires were mailed to the librarians who provided the service. The general usage statistics revealed an overall pattern of growth in the number of participating libraries, the number of patrons, the number of questions and the total time spent answering them in minutes. Patron evaluations of the service were positive, most of the patrons’ questions were answered. They all liked the service and intended to use it again. The librarians’ responses were also positive. Most of the librarians were able to nd quality Internet Web sites containing the answers to their patrons’ questions and when they could not, they often found the answers in a print resource. This nding suggested that regardless of the setting, print resources still had an important role to play in the provision of reference service.
Electronic access
The Emerald Research Register for this journal is available at
www.emeraldinsight.com/researchregister The current issue and full text archive of this journal is available at
www.emeraldinsight.com/0264-0473.htm
Real-time reference services are becoming widely available in libraries. As a new means of offering a traditional library service, it is essential to evaluate what is currently being offered in order to create effective real-time reference guidelines. The purpose of this study was to evaluate the materials, outputs, and outcomes involved in the provision of a real-time online reference service using a multi-method approach. In order to ful ll that purpose, data were gathered from both the patrons who used the service and the librarians who provided the service. Additionally, the service software collected data that revealed the patrons usage pattern during the initial stage of operation. The major goals of this study were:
. to determine whether or not the service
met both the patrons information needs and the patrons expectations;
. to determine whether or not the service
met the librarians expectations; and
. to determine what areas of the service
appeared to be working and what areas of service appeared to need improvement. The real-time reference service evaluated in this study is the service provided by the Northeastern Ohio Library Association (NOLA) Regional Library System. Since November 2000, this consortium has offered its member libraries the opportunity to participate in the provision of live interactive reference assistance to their patrons in real time via the Internet through
AskUsQuestions.com (NOLA, 2001). Reference librarians from a number of participating libraries serve as after-hours operators and provide the online reference service from their homes. In order to access the service, patrons must rst, click on the
AskUsQuestions.com button which is found in various locations on their libraries Web sites. On the next screen, they must enter their patron ID. This connects them with an e-reference librarian. Chat software allows the two parties to communicate with each other. Generally, the patrons type queries, and the librarians type responses while simultaneously searching the Web for answers. Then, as soon as the librarian nds the appropriate Web sites, he/she pushes The Electronic Library
Volume 21 · Number 6 · 2003 · pp. 565-575 Emerald Group Publishing Limited · ISSN 0264-0473 DOI 10.1108/02640470310509126
The authors thank the NOLA Regional Library System for providing system usage statistics and the real-time reference librarians who participated in the study.
pages from the sites into the patrons browsers. This study focused on
AskUsQuestions.com s rst several months of operation. It is hoped that the results will be of some assistance to those who are interested in planning similar real-time online reference services.
Literature review
Real-time online reference is a relatively new way to offer a traditional service. A review of relevant literature reveals that most published articles are largely descriptive in nature in explaining what real-time online reference is, how it works, what types of online reference services exist, and why it is needed (Eichler and Halperin, 2000; Lipow, 1999; Oder, 2001).
Early published studies on digital reference largely focus on the use of e-mail or Web forms as means of providing reference service to remote patrons (Ables, 1996; Bushallow-Wilberet al., 1996; Fishman, 1998; Garnsey and Powell, 2000; Kasowitzet al., 2000). Additional studies indicate that offering live, or real-time reference service, is a fairly recent addition to the more established digital reference services mentioned earlier. One such study was a survey of 150 academic libraries conducted by Janeset al.(1999) in which they reported that none of the libraries were providing reference services in real-time . Instead, e-mail and Web forms were the digital reference methods of choice. Their survey appears to con rm both how recently and how swiftly some libraries have developed and implemented real-time reference services. Tenopir and Ennis (2002) conducted a longitudinal study of reference services in academic libraries. They surveyed the academic members of the Association of Research Libraries four times during the past decade from 1991 to 2000. Their ndings suggest that digital information sources and services were being used at an increasingly accelerated rate in the 1990s and that the Internet and the World Wide Web are changing how reference services are delivered.
A number of papers have been devoted to the use of the Internet as a resource for answering reference questions. Devlin and Burke (1997) conducted a study to determine whether or not the Internet was a reliable and an authoritative source of information. They concluded that the Internet provides no comprehensive mechanism for identifying
quality information and called for a uni ed service interface to help alleviate this problem. However, in another study, Zumalt and Pasicznyuk (1998) used depth, accuracy and durability as three measures to determine the effectiveness of the Internet in answering reference questions. They concluded that the Internet could provide answers to a signi cant percentage of actual reference questions and that there was little signi cant difference in the accuracy of information obtained from traditional reference sources and Internet sources. Koutnik (1997) conducted a systematic study on using the Internet to answer 104 reference questions. He
concluded that librarians should be using the World Wide Web to supplement their print reference collections. However, Koutnik noted that additional time is needed to weed through the list of Web sites returned as relevant to a query.
More recent studies have focused on investigating users and librarians in digital reference services. For example, Ross and Nilsen s (2000) study provides insight into how Internet sources may have changed reference transactions and affected users experiences with reference services. Another example is Janes s (2002) survey, which not only reported reference librarians experiences and attitudes towards digital reference services, but also provided a detailed account of online reference services. Sloan s (2003) report about a collaborative online reference service offered by eight academic libraries in Illinois provides rst-hand usage statistics and a list of questions asked by the users of the service.
Still, evaluations by users and librarians of real-time online reference services are few. It is hoped that the results of this study of
AskUsQuestions.com, conducted during its initial stage of service, will add to the real-time reference service knowledge base in three areas: usage statistics; patron evaluations; and librarians perceptions.
Methodology
To achieve the goals of this study, the following three types of data were gathered.
(1) System usage data. The service software automatically recorded NOLA real-time reference transactions and then statistics were generated to reveal both the usage patterns of the service and the workload of service librarians.
(2) Patron evaluations. At the end of a real-time reference session, patrons were directed to a service evaluation form. These data were gathered within the same timeframe as the system usage data. To ensure patron con dentiality, no personal or demographic data were gathered.
(3) Reference librarian/operator survey. A questionnaire (see Appendix) was sent to all participating librarians during the rst 6 months of real-time reference operation. The following categories provided the underlying framework for the librarians questionnaire:
materials, outputs, and outcomes
(Allen, 1995, p. 209). Additional questions were included to gather demographic data about the librarians. The questionnaire was pre-tested by two reference librarians who had similar backgrounds with the participating librarians, but neither of whom was involved in the research project. They reviewed the questionnaire and made suggestions for improvement. Finally, the questionnaire was mailed to all seven real-time reference librarians/operators. All seven librarians returned the survey within two weeks.
Results
General usage statistics
Table I summarizes the monthly totals in each of the following general usage categories: the number of participating libraries, number of patrons, number of questions and the total time in minutes. There was an overall pattern of growth in the number of participating libraries, patrons, and questions, as well as in the total time in minutes. However, a drop in usage occurred during the month of April. This might have been due to spring or Easter break. Elementary and secondary school
children in Northeastern Ohio Public School Systems would have had at least a one-week break from school sometime during this month. The librarian responses reported later appear to reinforce this as a cause for the drop in usage because all the librarians reported providing homework assistance for junior high and high school students, and most of the librarians reported providing homework assistance to elementary students.
Table II comprises the weekly highs, lows, and averages in the following categories: patrons, questions, and time in minutes. NOLA s real-time online reference service was in operation ve days per week and averaged 7.6 patrons per day. The average number of patrons per week and the average number of questions per week imply that most of the AskUsQuestions.com patrons were asking only one question per session. Patron evaluations
Patron evaluations of the service were very positive, as shown in Table III. For 1 month, during the same time period when the usage statistics were being gathered, patron evaluations were also gathered. In March 2001, 193 patrons tried the service and 12 of them were randomly directed to the patron evaluation form. Of these 12 patrons, only eight responded. The response rate was low largely due to the fact that the survey site was pushed at the end of a real-time reference
Table IGeneral usage statistics January-May 2001
Month
Participating
libraries Patrons Questions
Total time (min) January 14 123 148 1,176 February 14 160 195 1,196 March 16 193 204 1,446 April 17 161 170 1,360 May 17 199 211 1,700
Table IIWeekly highest/lowest/average statistics January-May 2001
Highest Lowest Average
Patrons 59 12 38
Questions 78 14 42.18
Times (min) No data No data 8:13
Table IIIPatron evaluations of service How helpful was the librarian?
Very helpful 6
Helpful 1
Not helpful 0
Was your question answered?
Yes 7
No 1
Did you like the service (AskUsQuestions)?
Yes 8
No 0
Do you plan to use our service again?
Yes 8
session. In most cases, patrons had logged off before the survey page showed up after the service. While the number of respondents was small, it could be considered as a random sample. Patrons responses suggest that their experience with the online reference service was a positive one. In fact, all patrons indicated that they would use the service again. Seven out of eight patrons stated that their question was answered, and six out of seven thought that the librarian who provided the service was very helpful.
Reference librarian survey results Librarians’ backgrounds and training for real-time reference
The demographic data provided by the reference librarians indicated that the librarians had both the technical skills and practical experience required to conduct the online reference transactions. Five out of seven had a Master of Library Science (MLS) or Master of Library and Information Science (MLIS) degree, and the other two had a Bachelor s Degree. Six out of seven indicated that they had used the Web to answer reference questions at least once a day during the past year, and all of them had independent reference experience in their libraries. They indicated that they had learned how to conduct Web searches and how to evaluate the quality of Web sources mostly through continuing education workshops,
self-learning, and in-service training sessions. The librarians experiences serving as
reference librarians ranged from almost ve to 22 years, and their Web reference experiences ranged from four to six years.
All librarians received training in all aspects of the software that they would be using for the real-time reference work. They received at least a three-hour lecture demonstration training session with the balance of the day for hands-on practice. Further, each librarian received an operator s manual. When asked to rate how dif cult it is to use the NetAgent software interface, six of the seven librarians rated it as easy to very easy, and one rated it as neutral. When asked how well prepared they felt when they logged on for their rst real-time interactive online reference shift, three of the seven felt prepared, three felt neutral, and one felt less prepared. Workload and experiences
The librarians worked two to 2.5 hours (one to 1.25 two-hour shifts) each week providing
real-time reference services. In the rst four months after the real-time reference service was launched, the librarians experienced three to eight times when they did not have any patron to assist during their shifts. Six of the seven librarians found it easy to work alone and commented that they were comfortable providing reference assistance in a non-traditional setting located outside the library walls, for example, from home.
It is interesting to note how they described their rst real-time reference experience.
. Hard remembering how to use the
software. Fumbled around a bit, but I think that is normal when using something new .
. A bit nervous, but well motivated and
energized.
. Scary Ðbut turned out to be easier than
I feared it would. Also Ðhad a wonderful co-librarian at my side.
. It was a little scary and awkward to try
and do two things at once Ðchat and seriously search the Web. Sometimes your mind goes blank as you watch the clock tick away and record the transaction time.
. I d practiced a bit and the workload
wasn t heavy, so it wasn t dif cult. The only thing dif cult was not knowing what to expect.
. I was familiar with instant messaging
and chat, but had not done it too often. That was my biggest problem at rst. Apparently, even though all the librarians were trained and had the necessary experience for real-time reference work, their rst time using the system still appeared to be challenging.
The Internet vs traditional print sources as reference sources in real-time online reference
Although the majority of librarians speci ed that they were able to meet their patrons information needs using Internet Web sites, their experiences demonstrated that print resources continued to play an important role in the provision of reference services
regardless of the setting.
The majority of the librarians were able to locate quality Web sites containing answers to the patrons questions. This result is
consistent with the data on patron satisfaction evaluations reported earlier. Only one librarian was able to answer all questions
using the Internet for reference sources. The other six reported that they had at least one time when they could not use Internet sources to answer patron questions. In addition, four of the seven librarians reported that they could always nd Web sites that contained authoritative, accurate, reliable and
up-to-date information, but only three could locate Web sites that contained comprehensive information about a subject all the time.
It is interesting to note that although all of the librarians had a collection of pre-selected Web sites for their reference service, they did not rely solely upon their collections to answer patrons questions. Instead, they relied upon a combination of approaches that involved searches in their Web reference collection, searches of the Web and occasional searches through print resources. At times, some of the librarians consulted either print resources housed within the traditional library or print resources that they had on hand in their homes. Sometimes, the librarians requested a patron s e-mail address so that they could forward on any information that they later located in their library s print collection. Types of questions in online reference transactions
Most of the librarians found it dif cult to give the same level of service to two or more patrons simultaneously. This implies that before the librarians logged on for their rst real-time reference shift, more time to practice answering more than one question at a time via chat might have been bene cial.
Table IV summarizes types of questions received during real-time reference work as reported by the real-time reference service librarians. It appears that homework questions were most common in real-time reference services. All of the librarians reported receiving homework questions for Grades 7-12; ve of them received homework questions for Grade K-6. Six of seven librarians also received questions that were
ready reference in nature, or questions that were for recreation/entertainment and personal interest; four of them received questions that were about business or consumer/product information. Librarians’ perceptions of patrons’ expectations about real-time reference service
Most librarians ( ve of seven) felt that their patrons had unrealistic expectations about
how long it would take to search for information. Also, most librarians indicated that their patrons were unaware of the limitations of Web resources. Librarians responses were mixed regarding whether patrons could clearly communicate
information they needed in writing. Two of seven librarians were on the positive scale, one was neutral, and four were on the negative scale, including one very negative.
Bene ts and limitations of real-time online reference service and suggestions for improvements
Convenience and service after library
operation hours were mentioned most often as bene ts of real-time reference. Other bene ts mentioned by librarians included the
following.
. Library can extend hours in an
inexpensive way.
. Can reach a net-based clientele that may
never walk into a library.
. Improve the image of libraries. Make
libraries seem high-tech and with-it.
. Additional educational and training for
patrons by helping people nd authoritative and accurate answers and sources.
Most librarians felt that the hindrances to provide real-time online reference were both
Table IVTypes of questions in real-time online reference transaction
Types of questions Received by number of operators Homework help: 7-12 7 Ready reference 6 Recreation/entertainment 6 Personal interest 6 Homework help: K-6 5 Business 4 Consumer/product information 4 Do-it-yourself/how-to-do-it 3 Government 3 Health 3
Other (Ohio information/Local history/Sports) 3
Hobbies 2
Research assistance: undergraduate 1
Economics 1
Politics 1
Legal 1
Taxes 1
Research assistance: graduate 0
the amount of time it took to search the Web and the dif culty involved in nding
information on the Web. This reinforces the results reported earlier regarding the librarians perceptions of the patrons expectations of real-time reference
transactions. The following are limitations of real-time reference service as mentioned by the participating librarians.
. In depth research. We provide breadth of
knowledge, not in-depth, generally speaking.
. Only text based chat to communicate.
Not everything is on the Web. Searching the Web can be dif cult.
. Many people seem to think that when
they send us a question, we are a search engine and will instantly fetch the correct answer. I often have to tell them that I have to keep looking because the information was not at the rst Web site I found, etc.
. Can t always nd info on the Web
expediently. Reference interviews can be imperfect.
. Communication is limited and
sometimes things can get misinterpreted. Harder to conduct reference interview.
. Not enough authoritative info. on the
Web.
. Some questions are dif cult, and it is
very time-consuming to locate the exact answer. As a result, instant answers aren t always possible, and I need to consult print sources. So, having a dial-up access service for the Internet can be very slow and frustrating.
In terms of improvement in real-time service, most of the librarians felt that voice-over IP would improve the quality of service that they provided to AskUsQuestions.com patrons. This reinforces the results reported earlier that most of the librarians reported feeling that both they and their patrons experienced varying levels of dif culty communicating clearly via chat. Some suggestions offered by the librarians are as follows.
. De nitely need cable for this. 56 K is too
slow. Voice over IP.
. A good PR and marketing strategy plan
will add to the success of an innovative service.
. When most people have high speed
connections, voice-over IP
communications would be helpful.
. I would like more people to give me their
e-mail addresses so I could do further research and send them the information.
. Expansion of available hours. Voice
contact.
. Lists of good search engines and
metasearch engines that we can
package and send to patrons for their own further searches.
Conclusions and discussion
The overall pattern of growth re ected in the general usage statistics suggests that an increasing number of patrons of the participating Northeastern Ohio libraries began using the service once it was in operation. Further, the patron evaluations indicated that patrons were satis ed with the services they received and that they intended to use the service again.
The results of the real-time reference librarians survey indicated that the librarians felt con dent about their ability in using the Internet both as a resource for answering reference queries and as the means of delivering information. However, despite the training and experience each librarian received before his or her rst shift in
providing real-time reference service, the rst time still appeared to be challenging. In addition, most of the librarians found it dif cult to give the same level of service to two or more patrons simultaneously. This implies that before the librarians logged on for their rst real-time reference shifts, more time to practice answering more than one question at a time via chat might have been bene cial. On the other hand, it may also suggest that assisting one patron at a time would ensure quality service to all patrons.
Problems experienced by the librarians during the rst six months in real-time reference service are largely due to the limitations of the service. For example, communicating clearly using text based chat appears to be challenging; searching for answers is sometimes more complicated and time-consuming than patrons realize; and patrons may have unrealistic expectations on timely feedback during the reference
transaction with a slow network connection. As a result, the majority of librarians recommended voice over IP as a possible solution to some of these problems.
Although the majority of librarians speci ed that they were able to meet their patrons information needs using Internet Web sites, print resources continued to play an important role in the provision of reference services regardless of the setting. As a result, this study broadens Koutnick s (1997) conclusion that in the traditional reference setting librarians should use the Internet to supplement their traditional print reference resources. Data collected in this study demonstrate that in this non-traditional reference setting, the librarians used traditional print reference resources to supplement their use of the Internet as the primary reference resource. As a result, one can conclude that print resources still have an important role to play not only in the
traditional library reference setting, but also in the non-traditional virtual reference setting.
A follow-up study would be necessary to compare how the usage data change over time and how the reference librarians perceptions of this service change as they gain more experience. In addition, suggestions for improvement of online reference service can be gathered directly from patrons.
Furthermore, a follow-up study could address some of the questions that have not been answered in related studies: Are any of the patrons repeat users? How many e-mail questions are being answered in addition to real-time reference questions? How many of the total questions asked are unanswered by the librarians? How many of the total number of questions asked are not answered during the session, but are answered via e-mail at a later time? What type(s) of questions are most frequently asked? How many are answered using Web sites, and how many are answered with print resources? Answers to these questions would be helpful when it comes to making decisions about how to allocate library resources for improving reference services.
Real-time online references services are increasing in number across the country and around the world. The hours of operation, the staf ng patterns and the settings in which they are being conducted vary a great deal. For instance, some libraries may provide real-time reference only during their hours of operation and staff it with their librarians. Other real-time online reference services are provided by commercial enterprises for a fee and are generally not staffed by professional librarians. Future studies could be conducted to gather information about both the patrons
and the service providers in these various settings. Additional studies could compare and contrast the various types of real-time online reference services.
References
Abels, E.G. (1996), “The e-mail reference interview”,RQ, Vol. 35 No. 3, pp. 345-58.
Allen, B. (1995), “Evaluation of reference services”, in Bopp, R.E. and Smith, L.C. (Eds),Reference and Information Services: An Introduction, Libraries Unlimited, Englewood, CO, pp. 207-29. Bushallow-Wilbur, L., DeVinney, G. and Whitcomb, F.
(1996), “Electronic mail reference service: a study”, RQ, Vol. 35 No. 3, pp. 359-71.
Devlin, B. and Burke, M. (1997), “Internet: the ultimate reference tool?”,Internet Research, Vol. 7 No. 2, pp. 101-8.
Eichler, L. and Halperin, M. (2000), “LivePerson: keeping reference alive and clicking”,EContent, Vol. 23 No. 3, pp. 63-6.
Fishman, D.L. (1998), “Managing the virtual reference desk: how to plan an effective reference e-mail system”,Medical Reference Services Quarterly, Vol. 17 No. 1, pp. 1-10.
Garnsey, B.A. and Powell, R.R. (2000), “Electronic mail reference services in the public library”,Reference & User Services Quarterly, Vol. 39 No. 3, pp. 245-54. Janes, J. (2002), “Digital reference: reference librarians’
experiences and attitudes”,Journal of the American Society for Information Science and Technology, Vol. 53 No. 7, pp. 549-66.
Janes, J., Carter, D. and Memmott, P. (1999), “Digital reference services in academic libraries”,Reference & User Services Quarterly, Vol. 39 No. 2, pp. 145-50. Kasowitz, A., Bennett, B. and Lankes, R.D. (2000), “Quality
standards for digital reference consortia”,Reference & User Services Quarterly, Vol. 39 No. 4, pp. 355-63. Koutnik, C. (1997), “The World Wide Web is here: is the
end of printed reference sources near?”,RQ, Vol. 36 No. 3, pp. 422-9.
Lipow, A.G. (1999), “In your face’ reference service”, Library Journal, Vol. 124 No. 13, pp. 50-2. NOLA (2001), “AskUsQuestions.com: building cooperative
Web-based reference services”, NOLA Online, available at:www.nolanet.org(accessed 20 January 2003).
Oder, N. (2001), “The shape of e-reference”,Library Journal, Vol. 126 No. 2, pp. 46-50.
Ross, C.S. and Nilsen, K. (2000), “Has the Internet changed anything in reference?”,Reference & User Services Quarterly, Vol. 40 No. 2, pp. 147-55.
Sloan, B. (2001), “Ready for reference: academic libraries offer live web-based reference: evaluating system use”, available at:www.lis.uiuc.edu/~b-sloan/r4r. nal.htm(accessed 20 January 2003).
Tenopir, C. and Ennis, L. (2002), “A decade of digital reference – 1991-2001”,Reference & User Services Quarterly, Vol. 41 No. 3, pp. 264-73.
Zumalt, J.R. and Pasicznyuk, R.W. (1998), “The Internet and reference services”,Reference & User Services Quarterly, Vol. 38 No. 2, pp. 165-72.
Appendix. Real-time reference service
librarian/operator questionnaire
Biographies
Louise Kloss has a Master s Degree in Library and Information Science from Kent State University. She has been an adult services reference librarian at Morley Library in Painesville, Ohio for the past nine years. In addition to providing reference services, she is Morley Library s Database Coordinator; works part-time providing online reference assistance as one of NOLA s AskUsQuestions.com operators and is a member of the Ohio Library Council. Louise can be contacted at: Louise Kloss, Morley Library, 184 Phelps Street, Painesville, OH 44077, USA. Tel: (440) 352-3382; E-mail: [email protected]
Yin Zhang is an assistant professor at Kent State University School of Library and
Information Science. She holds a PhD in Library and Information Science from the University of Illinois at Urbana-Champaign. Her research interests include information seeking behaviour in the Internet environment, evaluation of information uses and services, and distance learning. Her papers have appeared inJournal of the American Society for Information Science and
Technology,Journal of Information Science,Online Information Review, andLibrary Trends. Yin can be contacted at: Yin Zhang, School of Library and Information Science, Kent State University, Kent, OH 44242, USA. Tel: (330) 672-2782; Fax: (330) 672-7965; E-mail: [email protected]
