One Site to Rule Them All: Usability
Testing of a Responsively Designed
Library Website
Junior Tidal
Introduction
Users of academic libraries are utilizing mobile de-vices to access the web. According to the Pew Inter-net & American Life Project, 68% of users access the web through Smartphones or tablet computers.1 In
response, librarians and library web developers have created mobile sites. More recently, some libraries have adopted responsive design to serve their patrons, a practice of rendering a singular website experience among cross-platform devices. However, despite this more unified experience, usability obstacles may exist that prevent users from accessing information. Can users truly reach their information seeking goals on a library website regardless of what kind of device is used? This paper examines the on-going usability testing of a designed website for an urban, academic library. It will explore themes applicable to other li-braries, including the importance of reiterative usabil-ity testing, obstacles found in responsive design, and comparing task-testing performance across several different device platforms.
The Importance of Usability Testing
A library is an institution of knowledge and empow-erment, only if patrons can access our resources. Li-brary websites can be confusing; jargon, the startling experience of being whisked away to external infor-mation resources, archaic catalogs, or a combination of the above, can be frustrating at best and prohibitiveat worst. Accessing a library website within the con-text of a mobile device can amplify these problems. Ideally, patrons should be able to find, access, and consume information in a reasonable process regard-less of what device they are using.
Librarians have used usability studies to get a better understanding of their users and to provide better access to their resources. Usability testing can pinpoint obstacles that librarians may find difficult to see. Librarians typically know jargon, the technical know-how to find resources in electronic databases, know which search terms to find materials, as well as the specific idiosyncrasies of interfaces, yet these can confuse the end-user.
By identifying a library website users’ specific us-ability obstacles, librarians can help strategize an ef-fective process to provide better information access and help users reach their information-seeking goals. At the code4lib 2014 National Conference, Sumana Harihareswara’s opening keynote deftly summarized the importance of the user experience, stating that li-brarians and developers removing usability obstacles “are working to achieve social justice goals.”2 Winhall
describes user-centered design, stating that “design-ers observe people in context to und“design-erstand the com-plex experiences, needs and wishes of individuals, and are able to represent and champion those needs throughout the design process.”3 Identifying and
re-moving usability obstacles is one step to rere-moving in-Junior Tidal is Web Services & Multimedia Librarian, and Assistant Professor at New York City College of Technology, e-mail:
ACRL
2015
herent, technological gatekeepers to our institutional resources.
Background
The Ursula C. Schwerin Library serves the New York City College of Technology (City Tech), CUNY com-munity of 1,700 full and part-time faculty, and over 17,000 students. The main library website (http://li-brary.citytech.cuny.edu) is managed using the Drupal 6 (D6) content management system. The D6 CMS uses community created modules that expand the func-tionality of the site. In 2012, a separate mobile website (http://m.library.citytech.cuny.edu) was launched us-ing Drupal 7 (D7) to compensate for an increase of users visiting the site through mobile devices.
Accom-modating mobile users is a trend that many academic libraries have developed (Jackson, 2013).4 However,
the maintenance of these two separate sites resulted in the development of a redesign adopting responsive design techniques which would replace both.
Matthew Reidsma describes responsive design as providing “the same content to all users on the same URL, regardless of the device.”5 Numerous libraries
have employed this design technique. Eric Rumsey, a librarian at the Hardin Library, Iowa University, keeps an updated list of libraries who have utilized the tech-nique.6 Libraries have implemented studies regarding
responsive designed websites. Oregon State Univer-sity use survey and analytics data to justify the im-plementation of responsive design.7 California State
FIGURE 1
Polytechnic University-Pomona utilized a number of different devices to test their site.8
Methodology
There were two objectives sought while conducting this study. The first was to determine if a responsive designed site could support students’ research needs efficiently. The second was to determine if there was a consistent experience in using different devices. A heuristic evaluation was used to reach these objec-tives.
Google Analytics and Piwik, an open-source ana-lytics tool, were used to pinpoint what pages were uti-lized in the desktop and mobile websites prior to the redesign. Logging visitor data, these tools determined
which devices were used the most to access the li-brary’s desktop and mobile site. Data was also collect-ed to determine most uscollect-ed functions or pages of the site. Each of these tools use a piece of JavaScript that is inserted into each web page, which sends data back to the tool, which is presented in a human-readable format. Other than tracking devices and page visited, these tools can also determine how long a user was on a page, the general geographic location visitors origi-nate from (based on IP address), operating system used, a wide spectrum of data on users.
The heuristic evaluation, or usability test, was widely advertised. Social media, flyers, and e-mail were all used to inform students and faculty about the study. Participants of the study were open to City Tech students and faculty, with students receiving a $5 Am-azon gift card as an incentive. Prior to testing, partici-pants were screened using a web survey to determine which device or platform they have had prior experi-ence with (see Appendix A). So not to confuse the us-ability of the website with that of the device, users uti-lized devices they were familiar with. The URL for the screening survey was included on all advertisements. The usability test, related surveys, and incentives were approved through the City Tech Institutional Review Board (IRB).
A usability test typically consists of two rounds of testing. In the first round, participants are asked to complete a series of tasks and to use the “think-aloud” protocol (TAP). TAP allows users to give feedback as they attempt to complete the task, providing proctors with their reactions and opinions. TAP gives further insights that may not be available through qualitative metrics. Based on the success and failures of these tasks and the identification of usability problems, the website would be modified for the second round of testing. Using the same tasks from the first round, it can then be evaluated how well the modification im-prove the usability of the site.
Testing was conducted on a working prototype of the library’s responsive redesign (see image 3). This site uses D7 as its content management system, aug-mented with a Bootstrap-based responsive theme.
FIGURE 2
ACRL
2015
Bootstrap is a popular front-end web framework that is accessible and responsive. In development for over a year, this redesigned site has migrated from the City Tech servers to the Amazon Web Service in the cloud. The library website is also comprised of several differ-ent sub-sites running CMSes, such as WordPress, Me-diaWiki, and more recently LibGuides. However, the usability testing of the site was focused on the D7 in-stallation. Due to the nature of the tasks participants were asked to perform, they were taken to external sites such as the CUNY Catalog and the different elec-tronic resources. The librarians at City Tech do not have direct control of these external sites, but are nec-essary components to the library’s online presence.
Usability testing is inherently user-centered, in-corporating the experiences of participants in order to better improve the system or design. It builds upon
observing users, rather than just asking questions or opinions, as there is much more to be found watch-ing someone use a website rather than gatherwatch-ing data from surveys or focus groups.
The library utilized a series of 12 usability tasks for the user to undertake (see Appendix B). These tasks were derived from the analytics tools described above, and were written as scenarios for the user to complete.
The test utilized 9 different devices, including a MacBook Pro, a Windows 8 laptop, an iPad 2, an iPad Mini, an iPod Touch (to simulate an iPhone), a Samsung Galaxy S4 Android Smartphone, a Samsung Galaxy Tablet, a Microsoft Surface Tablet, and an An-droid based Google Nexus. These devices were select-ed basselect-ed on their popularity found through analytics data. Testing was conducted within the library in a
FIGURE 3
closed room away from other distractions. The City Tech wireless network is closed, and only available to users with a username and password. However, the principle investigator of this study received prior ap-proval for all 9 devices to access the network.
Scenarios were read aloud to participants, and were also given a written list for reference. Each sce-nario began at the home page as a starting point. Par-ticipants and their screens were digitally recorded for note-taking purposes. Mobile device screens were originally going to be recorded through device mir-ror, where screens are shown on a computer which can then be recorded. However, due to the infra-structure limitations of the City Tech WiFi record-ing jittery and slow digital recordrecord-ings, a web camera was used to record screens as they used the site. The camera was positioned so it did not interfere with par-ticipants interacting with the device. A Blue Snowball USB microphone was also used to record the audio of the participant studies.
After completing each task, participants were asked to rate the ease of their experience using a 5-point Likert-scale immediately after they attempted the scenario. At the end of the usability session, us-ers were given a post-test survey to fill out (see Ap-pendix C). Likert-scales were also used here, rating aesthetics overall ease of use, and other factors (see Appendix B). Other information such as prior use of the library website on mobile devices and users’ opin-ion of using the label “citatopin-ion finder” were queried on the post-testing survey. Dougan and Fulton posed a similar question for their usability testing to deter-mine if “Class Guides” was an appropriate term for their research guide.9
Results
Initially, the test was intended to utilize 90 partici-pants: 5 participants using 9 devices over 2 rounds of usability testing. Unfortunately, there weren’t enough participants available to utilize each device. Regard-less, enough data was gathered by dividing partici-pants by device type: workstations (Windows and Macintosh), tablets, and smartphones. Tablets
includ-ed a Microsoft Surface using the Windows RT operat-ing system, a Google Nexus device usoperat-ing the Android operating system, and an iPad Mini. Participants were given the option to use an iPad or an iPad Mini and most chose the Mini.
5-8 users are the typical sample of web usability studies according to Nielsen, although he notes that for statistically significant results, 20 participants should be used.10 Overall, 20 students participated in
the study. A breakdown of participants by device can be found below (see table 1).
Among the device types, there weren’t that many differences among the success and ease of the tasks. Workstation (100% success rate) and tablet users (80%) were successful in finding videos using the li-brary website, but Smartphone users were the least successful (37.5%). Conversely, workstation users had difficulty finding borrowing policies (57%) in comparison with tablet users and Smartphone users (100% and 87.5%).
Even though participants failed some tasks, they generally perceived the tasks easy to complete. The most difficult tasks for most participants included finding a research guide despite a large banner image advertising them on the homepage. Finding citation resources, newspaper articles, and library faculty sub-ject specialists were found to be difficult to complete. Despite this, all users were successful in finding links to connect with reference librarians and library hours.
Some users were also recorded as partially com-pleting a task, labeled as Success/Fail. For a task to be ranked as success/fail, the participant may be unaware they were on the correct page, or they completed a task other in a way than it was intended to be com-pleted. For example, some participants found cita-tion guides on the website, but were not satisfied and click on to another page. Another user who was asked to find a citation guide found the citations provided within an article in an EBSCO database.
TAP responses were very useful in identifying usability problems. Users were vocal about their dif-ficulty finding eBooks and videos. The CUNY Cata-log displays eBooks as “Electronic Resources” and
ACRL
2015
users were not able to determine if that signified an eBook or not. Secondly, users were unable to sort search results by format type. Users also expressed how branding can be confusing, as some were un-able to identify the Lexis-Nexis database. Mobile and tablet users noted that they didn’t mind scroll-ing through long pages of electronic resources in
or-der to complete a task. Participants also gave a mix of feedback regarding the site, with mostly positive reactions (see table 2).
Responsive Design Post-Survey
At the end of the usability test, participants were giv-en a pgiv-en and paper survey. The purpose of the
sur-TABLE 1
Success/Failure Rate and Perceived Ease of Use Per Task Windows and Mac
Machines (n = 7) Tablets (n = 5 ) Smartphones (n = 8) Success Rate Avg. Ease of Use (1–5) Success Rate Avg. Ease of Use (1–5) Success Rate Avg. Ease of Use (1–5) Find a book 85% 2.14 80% 1.6 87.5% 1.88 Find an eBook 71% 2.71 80% 3 75% 2.88 Find a film 100% 2.14 80% 3.6 37.5% 3.75
Find a specific database 71% 2.00 60% 3.4 75% 2.38
Find a newspaper article 14% 2.71 60% 2.4 50% 2.38
Find a research guide 0 1.57 20% 1.8 25% 1.75
Find a citation resource 14% 0.86 20% 2 25% 2.38
Find a tutorial 43% 1.86 80% 2.2 50% 2.25
Ask a Librarian 85% 1.57 80% 1.6 100% 1.13
Find the subject librarian for your department 28% 1.29 60% 1.8 37.5% 1.63 Find how long you can check out a book 57% 1.43 100% 1.6 87.5% 1.63
Find the library’s hours 85% 1.43 100% 1.2 87.5% 1.75
TABLE 2
Overall Reactions of the Site
Please give any opinions or comments about the site you may have.
Positive Reactions: Negative Reactions:
“It’s a great website for research” “…needs more improvements like subjects added for faster research as well as a separate link to find dvd’s, etc..” “Good site, very easy to navigate around :)” “I couldn’t find the video and tutorials should be easier to
access and the library hours when click hours didn’t come up.” “Easy to understand.” “More color to help with visual organization, bigger more
noticeable buttons.” “Works well on mobile.”
“The site has very good tools for students at City Tech. Everything is very accessible and clear.” “The site is easy to navigate especially on a mobile device.”
vey is to gather specific information that may not be necessary apparent in a usability test. The first four questions used a Likert scale to rate user’s opinions of the library website, with 1 rated as “easy” or “good” and 5 rated as “difficult” or “bad,” depending on the question. Overall, there were not many discrepancies among users in all categories.
There were consistent scores among users among the survey questions (see table 3). This includes the heading of the website, finding information, and opinion of the library’s visual design. Smartphone us-ers and tablet usus-ers found the navigation easier to use compared to workstation users, however, Smartphone
users found the website harder to use navigate com-pared to tablet and desktop users.
The survey also gave insight on users’ use of the library website. Win/Mac users have accessed the li-brary website the least (14%) using a mobile device, followed by Smartphone users (37.5%) and tablet us-ers (60%). All tablet and workstation usus-ers have had prior experience with the library website, yet 50% of Smartphone users did not.
The last two questions of the survey were open-ended. One question asked participants about the label, “Citation Finder,” a CUNY-wide service where users enter information to find a specific article.
Us-TABLE 3 Post Survey Results Questions Windows and Mac
Machines (1 easy/ good—5 difficult/ bad) n = 7 Tablets (1 easy/ good—5 difficult/ bad) n = 5 Smartphones (1 easy/good—5 difficult/bad) n = 8
1. The City Tech Library website is easy to use
on this device 1.71 1.4 2.25
2. Headings on the City Tech Library website
are easy to understand 1.14 1.2 1.62
3. Finding information on the library website is 2.14 2.2 2.5
4. I would rate the library’s website navigation
as _______ to use 2.29 1.8 1.75
5. Have you used the library website before? 100% yes 100% yes 50% yes 6. Have you used a mobile device or
Smartphone to access the library website? 14% yes 60% yes 37.5% yes 7. On a scale of 1 to 5, how would you rate the
library website visual design? 2.43 2 2.12
TABLE 4
Suggestions for the Term “Citation Finder”
Something different "Article or Title Finder" No, it doesn't make sence (sp).
"I would rather use a term that isn't as technical, such as "article finder" to make things simplier."
I believe a more clear term can be used to find the link for articles. Possibly "Article Finder" or just "Articles." Citation Finder gives the impression that I would only be able to find citation of different articles but not the actual article. "Article finder” would be an appropriate term or “sources search.”
Yes, It's clear.
I don't even know what that is
ACRL
2015
ers were asked if the “Citation Finder” made sense to them, with 60% of all participants thought this sound-ed unclear. Participants gave several different alterna-tives for the label (see table 4).
The final question asked users of their opinions or comments about the new site, as these were the first students to use the site. Their insights could be used to make the site more usable. Although students were asked of their impression after completing a task, this question provided an opportunity for feedback of the site as a whole.
Application of Test Data
After this first round of usability testing, the data gathered here will be used to improve the site. Some improvements are technical. One example is repairing the RSS feed for library hours, as this was the main obstacle that had users fail the task. If the feed was unavailable, the hours of the library would not be dis-played on the front page. Another instance is the task
of users finding a New York Times article. Currently, our library is encouraging City Tech users to register for the New York Times for free online access, using their school email. As such, participants who used the eResources page to access the newspaper site were greeted with a login screen. This page dissuaded many participants from continuing or completing the task of finding a newspaper article, and they chose to look elsewhere.
As mentioned before, participants of the study had difficulty using the CUNY Catalog, an Aleph-based web OPAC. There was consistent difficulty finding call numbers for books, determining if an electronic resource was an eBook, or finding a spe-cific format. As a modification to the site, the second round of testing will incorporate the newly imple-mented Primo discovery layer for participants to find library materials. Unlike the CUNY Catalog, the dis-covery tool displays call numbers within the search results.
TABLE 5
Please Give Any Opinions or Comments about the Site You May Have
Workstation Users Tablet Users Smartphone Users
Have a direct e-mail link and phone
number Easy to understand. Maybe change placement of hours to "services [page]." It's a great website for research Works well on mobile. Looks
similar to current site. "I believe the site is easy to navigate." It is just a matter of playing with the links and seeing what comes up when you click on something. As long as you are familiar (sp) with the menu and links on the website, it is easy to use." "I feel that everything is good, but
some aspects should be simplified." "The site has very good tools for students at City Tech. Everything is very accesible and clear. (sp)"
I like it; very attractive and organized.
I was unclear on how to find an article
specifically from the New York Times. The site is easy to navigate especially on a mobile device. Just add a tab for video like find video and movies. Nothing else. "The site is overall good, needs some
improvements. like more subjects added for faster research as well as a seperate link to find dvd's, etc."
More color to help with visual organization, bigger more noticeable buttons." Good site, very easy to navigate
around :) I couldn't find the video and tutorials should be easier to access and the library hours when click more hours didn't come up. APA should be underneathe (sp) MLA.
In order for users to find call numbers within the catalog, they must first accurately spell their query (there is no spell check), and click on the link under availability. Since the catalog is shared by all 23 CUNY campuses, the user must click on the abbreviation for their school or NYCCT for City Tech. This will pro-vide the user with the call number. The implementa-tion of Primo may also have implicaimplementa-tion for finding materials based on format, since there isn’t an option to limit results or sort items based on format. The sec-ond round of testing will compare the success of using the CUNY Catalog versus Primo.
Finding research guides and tutorials were also a problem among all users. Users either had difficul-ty finding them or weren’t aware of their existence. Even with a front-page graphic heavily promoting them, research guides were difficult for participants to discover. Participants also had a difficult time finding subject librarians specific to their discipline. One idea to make this information accessible is to incorporate them on different pages throughout the site. For instance, placing links to the subject guides on the subject librarian directory and vice-versa may be helpful to the end-user. After these modi-fications, a second round of testing will take place. This will help determine if these changes affect the
success rate of tasks and the ease of use among par-ticipants.
Based on the overall reactions of users, many par-ticipants felt the site was easy to use, despite failing a number of tasks. This is evident in the scores found in table 1 and some of the reactions found in table 5. The reactions also support some of the difficulty that was encountered in finding a tutorial, subject guide, video, article, or eBook. A few participants noted how the site was easy to use on mobile devices.
Conclusion
One of the most important reasons for libraries to conduct usability testing is to create empathy for our users. This on-going usability test identified a number of issues that should be rectified as we improve our new responsive designed website. Overall, users per-ceived the responsive designed site as easy to navigate and use, despite the device they were using. After this first round of testing, it seems that this design simu-lates a consistent experience across devices. However, a second round of testing is necessary to support this conclusion.
Support for this project was provided by a PSC-CUNY Award, jointly funded by The Professional Staff Congress and The City University of New York.
ACRL
2015
Appendix A. Screening Tool
To be used as an online form
Thank you for your interest in participating in testing the library’s newly redesign library website. Please answer the following questions:
1. What is your status: q Student q Faculty
2. Do you use any of the following devices: q iPhone q iPad q iPod Touch q None of these 3. Do you use any of the following devices:
q Google Nexus 7 q Galaxy Note q Galaxy S4 q None of these 4. Do you use any of the following devices:
q Surface q None of these
5. Which is your preferred computer platform? q Apple Macintosh q Windows q None of these
Thank you! If you are interested in participating in the City Tech Library’s research study, please enter the fol-lowing contact information to schedule a session:
Name _________________________________________________________________________ Email __________________________________________________________________________ Phone Number __________________________________________________________________ If all answers are “None of these”:
Thank you, but unfortunately, you do not qualify to participate in the library’s research study at this time. You may close this browser.
Appendix B. Task Scenarios
1. You need to find a novel for your Sociology class. Your professor has told you to read George Orwell’s book, Animal Farm. Find the call number for this book using the library website.
2. Imagine you are off-campus and the library is closed. You need a copy of Mary Shelley’s Frankenstein to complete your English 1101 paper that is due tomorrow. Using the library website, find the eBook version of Frankenstein.
3. You need to watch a documentary based in Brooklyn for a sociology class. Using the library website, find a video on Brooklyn.
4. Your instructor has given you a history assignment. She has suggested that you find an article in the New York Times using Lexis-Nexis. Using the library website, find the Lexis-Nexis article database.
5. Imagine you are taking an English class. Your instructor has given you and assignment, and you need to find a research article on Internet Privacy from the New York Times.
6. You are a new freshman at City Tech and you are assigned a research paper for your nursing class. Using the library website, where would you go to get started with your research?
7. You are a typing a paper for a psychology class and your instructor has asked you to cite your sources in APA format. Using the library website, where would you go to find this?
8. You recently found out that the library offers several tutorials on the library website. Find the tutorials on the library website.
9. You are trying to find information about the Nike shoe company for your business class. You don’t know where to start and would like to get some help from a librarian. How would you contact a librarian for help using the library website?
10. Imagine you are writing a paper for your philosophy class. You want to have a one on one research session with a librarian. Using the library website, find the librarian who specializes in philosophy.
11. You need a book from the library, but want to know how long you can borrow it. Using the library website, find out how long you can check out a book.
12. You are planning to go to the City Tech library today to study for your finals. What are the hours for the week?
ACRL
2015
Appendix C. Responsive Design Post-Survey
1. The City Tech Library website is easy to use on this device
1 2 3 4 5
easy very difficult
2. Headings on the City Tech Library website are easy to understand
1 2 3 4 5
easy very difficult
3. Finding information on the library website is
1 2 3 4 5
easy very difficult
4. I would rate the library’s website navigation as _______ to use
1 2 3 4 5
easy very difficult
5. Have you used the library website before? q yes q no
6. Have you used a mobile device or Smartphone to access the library website? q yes q no 7. On a scale of 1 to 5, how would you rate the library website visual design?
1 2 3 4 5
good bad
8. We used the phrase “Citation Finder” to link a tool used to find articles. Is this term clear or would you use something different to describe this web page?11
9. Please give any opinions or comments about the site you may have.
Notes
1. Fox, Susannah, and Lee Raine. “The Web at 25 in the U.S.” Pew Research Internet Project. February 27, 2014. http:// www.pewinternet.org/2014/02/27/the-web-at-25-in-the-u-s/ 2. Harihareswara, Sumana “UX is a Social Justice Issue.”
Key-note address, code4lib National Conference, Raleigh, North Carolina, March 25, 2014.
3. Jennie Winhall, “Is Design Political?” Core 77, accessed December 10, 2014, http://www.core77.com/reactor/03.06_ winhall.asp
4. Jakob Nielsen, “How Many Test Users in a Usability Study?” Nielsen Norman Group, last modified June 4, 2012 http:// www.nngroup.com/articles/how-many-test-users/
5. Reidsma, Matthew. “Chapter 1.” In Responsive Web Design for Libraries: A LITA Guide, 4. Chicago: ALA TechSource, 2014.
6. Eric Ramsey “Responsive Design Sites: Higher Ed, Libraries, Notables” last modified May 3, 2012. http://blog.lib.uiowa.
edu/hardinmd/2012/05/03/responsive-design-sites-higher-ed-libraries-notables/
7. Rempel, Hannah G., and Laurie Bridges. “That Was Then, This Is Now: Replacing the Mobile-Optimized Site with Responsive Design.” Information Technology and Libraries 32, no. 4 (2013). http://ejournals.bc.edu/ojs/index.php/ital/ article/view/4636/pdf.
8. Conrad, Suzanna, and Julie Shen. “Designing a User-Centric Web Site for Handheld Devices: Incorporating Data-Driven Decision-Making Techniques with Surveys and Usability Testing.” Journal of Web Librarianship 8, no. 1 (October 2014): 1-35.: doi:10.1080/19322909.2014.969796. 9. Dougan, Kristin, and Camilla Fulton. “Side by side: What
a comparative usability study told us about a web site rede-sign.” Journal of Web Librarianship, 3 no. 3 (2009): 217-237. 10. Nielsen, “How Many Test Users in a Usability Study?” 11. Adapted from Dougan, K. & Fulton, C. (2009). “Side by
side: What a comparative usability study told us about a web site redesign.” Journal of Web Librarianship, 3(3).