Database Evaluation: Criteria Review

(1)

March 2012

These whitepapers are for reference use only and have not been updated by ERAC. The content within may not represent current information, technology or practice.

Database Evaluation: Criteria Review

March 2012

(2)

March 2012

1. Parts of a Database Evaluation

Databases are complex resources with three different areas that need evaluation as part of a whole resources evaluation: content, coverage, and technical.

Content Evaluation

This is evaluating the actual resources within the database and can be done using the same evaluation criteria used for print and multimedia resources that most match the content

(Johnson, 2009 & Hughes-Hassell & Mancall, 2005). i.e. The content of an online encyclopedia is its articles and can be assessed using the same criteria for print encyclopedias and multimedia formats it includes such as videos.

Coverage Evaluation

Coverage evaluation examines what resources are included in the database. How many journals, ebooks, magazines, newspapers, etc. are included in the database? What is the scope of the total coverage? Does it overlap with other databases or resources currently in the collection? There are some clear, basic criteria to complete coverage evaluation and no background information is needed to perform this part of the evaluation (Johnson, 2009).

Technical Evaluation

The technical evaluation involves specialized knowledge because it deals with all the unique functions of a database resource such as the search interface, results interface, index, and navigation (Johnson, 2009). This guide provides a summary of the key points needed to perform this part of the evaluation.

2. Developmental Characteristics Influencing Database Use

Students are growing and developing cognitively and this has an effect on their ability to find useful information efficiently. They do not have the physical, cognitive, or technical abilities of adults and therefore search differently. Students are also constantly developing and learning so go through many different stages of search abilities. Most researchers looking into child and young adult information retrieval (finding information) use developmental characteristics based

(4)

March 2012

on Piaget’s developmental stages (Bilal & Wang, 2005). Here are a few characteristics specific to each stage that are important for an understanding of student’s search methods and abilities:

•

Preoperationa

l (ages 2-6)

o They are aware that the world can be categorized and can conceptualize simple,

physical and sense based categories

o Hierarchies are beyond their understanding

•

Concrete Operational

(

ages 7-11)

o Developing basic problem solving skills, including being able to formulate search

queries and strategies

o Can categorize concrete concepts but abstractions are still too difficult o Initial understanding of hierarchies, but only with concrete concepts, abstract

categories are only frustrating

o Difficulty using more than one information source to answer an information

need

•

Formal Operational

(ages 11+)

o Developing the ability to use abstract concepts

o Developing the ability to synthesize multiple information sources

o Able to use multiple strategies and sources to meet information needs (Large,

Nesset, & Beheshti, 2009)

Keep in mind the age ranges for these developmental stages are fluid and each child develops at a different rate.

3. Search Methods

(5)

March 2012

Keyword Searching

Entering key terms/ words into a search box to retrieve information sources. Think Google.

Challenges of Keyword Searching

• Language and spelling skills of students. The younger the student the more limited their vocabulary and spelling skills. This can make it very difficult for students to come up with appropriate search terms and lead to frustration when misspelled terms return no results (Bilal, 2002 & Spink, Danby, Mallan, & Butler, 2009)

• Difficulty expressing or defining what it is they are looking for. The younger the child, the harder it is for them to identify exactly what it is they are looking for. This problem exists even for adults but it is accentuated in children especially before they develop problem solving skills and mental operations, which usually develop in the concrete operational stage (around age 7) (Bilal, 2002 & Meyers, Fisher, & Marcoux, 2009) • Synonyms. Effective searchers are good at coming up with synonyms for their search

terms in order to revise their searches and find more sources. Young students have difficulty coming up with synonyms and therefore often get stuck performing the same searches over and over (Wallace et al., 2000)

• Homonyms. Young searchers often don’t understand or recognize multiple meanings to their search terms and therefore get frustrated when results are based on the undesired meanings. If they are able to understand why they are getting irrelevant results, they often get stuck trying to use more words to clarify their desired meaning instead of using synonyms (Wallace et al., 2000)

• Boolean. Boolean searching and operators are simple but complicated and most adults don’t even use Boolean operators correctly or effectively. Most teenagers and children prefer using natural language and sentences or questions as queries rather than Boolean operators (Hirsh, 1999)

(6)

March 2012

Solutions

• Spell check offering alternative spellings are a must but sometimes child spellings are beyond computer understanding (Kuntz, 1999)

• Integrated thesaurus to help with synonyms (Bilal, 2002 & Kuntz, 1999)

• Term clarification such as statements on how the computer has interpreted the given search terms, including definitions and options for alternative meanings (Wolfram|Alpha) • Simplified Boolean interface which can take different formats based on the age of the

targeted audience (Hirsh, 1999). Some innovations include a graphical interface using puzzle pieces to visually represent and build the search query (Boolify) or using drop

down menus and multiple search boxes to help structure the query

Things to look for when assessing the search functionality:

• Spell check • Thesaurus

• Suggestions like “Did you mean” and “Related searches” or “Also try” • Advanced search options showing drop down menus

• Natural language search or sentence format search capability • Large search boxes

• Search and advanced search available from front page and always easy to access from anywhere in the database.

Browsing

Browsing means looking through a list of subject terms in order to access information sources, for example, using an index in a book or a website’s navigation menu.

Challenges of Browsing

• Hierarchies. Before the concrete operational stage (or around age 7) children have great difficulty understanding the concept of hierarchies, let alone navigating them. During the concrete operational stage hierarchies are only useful for concrete terms (Bar-Ilan &

(7)

March 2012

Belous, 2007 & Bilal & Wang, 2005). That being said, browsing is the search method of choice for young students because it is less cognitively demanding than keyword searching (Large, Nesset, & Beheshti, 2009)

• Abstract concepts. Broad categories in directories are often abstract concepts like “Entertainment and leisure” which young children don’t understand (Bar-Ilan & Belous, 2007)

• Getting lost. Students can get lost following a trail of subject links, getting more and more specific or clicking on subject terms that don’t match their research query. They can’t figure out how to navigate backward so they end up starting all over. Alternatively, it can be hard to go back to a source or heading they found useful once they have left it (Meyers, Fisher, & Marcoux, 2009)

• Hidden. Directories and subject heading lists are often buried within the interface (Kuntz, 1999)

Solutions

• Availability. Put the browsing options right on the front page with the search box (Meyers, Fisher, & Marcoux, 2009, Kuntz, 1999)

• Use audience appropriate subject heading and categories. This means a database will be age/ audience specific or will need different interfaces for different student groups. Appropriate headings could take the form of simple, concrete terms or icons and images integrated with the categories (Large & Behesti, 2005)

• Balance the breadth and depth of the hierarchy. There should be a balance between how deep or how many levels there are in a directory and how many initial categories there are (Larsen & Czerwinski, 1998)

• Clear navigation trail. It should be clear to users where they are in the database, how they got there, and how to move forward and backward. The most common form of this is a breadcrumb trail (Subject headings > Recipes > Desserts > Cookies > etc.) (Meyers, Fisher, & Marcoux, 2009)

• Multiple access points. An effective subject heading system or directory should have many ways to access the categories. This includes cross-listing subjects (“see” and “see also” references), alphabetical as well as hierarchical listing, making the subjects searchable (Kuntz, 1999)

(8)

March 2012

Things to look for when assessing the browsing functionality:

• Directory or subject heading access from front page • Age appropriate terminology

• Option to include “tags” or user created subject headings • Clear, simple navigation trail

• “See” and “see also” suggestions • Searchable subject headings

• Alphabetical as well as hierarchical category lists • Subject/ tag cloud

• Clear, consistent use of icons and labels

• “Hot” topics or special curriculum related sections

• Tools to mark or save good resources or good subject headings

4. How Students Use These Search Methods

Students favor a combination of browse and keyword searching. They will use their preferred method then switch as they run into difficulties, but only if an alternative method is accessible.

Browsing is favored by young (under 7) students because:

• It is less cognitively demanding as it relies on word recognition rather than recall • It is easier to use when the user is unsure of what terms to use, or unable to spell the

terms

• It is easier to use when you don’t have much foreknowledge of the subject being researched

• It is a good way to explore an unfamiliar resource (Large, Nesset, & Beheshti, 2009 & Bilal, 2005)

(9)

March 2012

Keyword searching is preferred because:

• It is a quick way to get some results. You can do a “quick and dirty” search • It can incorporate natural language queries

• It is the most familiar way to search, thanks to Google and other adult search engines. • It works better for complex or abstract concepts (Large, Nesset, & Beheshti, 2009 & Bilal,

2005)

Young students will switch to keyword searching when they get stuck while browsing, or if subject categories are not understandable.

Things to look for to support how students search

• Availability of both methods, and ability to switch at any stage of the search process from anywhere in the database

• Browse suggestions accompanying keyword search results • Clickable search options in browse results

• Highlighted and clickable subject headings in result descriptions • Query string included on results page

5. Search Results

The results page is where students review and choose their information sources. Very few students skim the actual resource or page. Instead they evaluate the relevance of the source based on the details provided on the results list.

When choosing sources students:

• Prefer familiar formats. For example, young students like encyclopedia entries because they are familiar with the print versions

(10)

March 2012

• Bounce between results list and sources without actually reading sources and often times getting lost (Hirsh, 1999 & Meyers, Fisher, & Marcoux, 2009)

Information students use to choose sources:

• Title • Format

• Listed subject headings

• Any bibliographic description or text snapshot included in results (Hirsh, 1999)

Things to look for in the search results:

• Full bibliographic description available on results page

• Short abstract or description of each source is preferable to the Google style text snapshot

• Highlighting of search terms in results listings • Clickable subject headings in results

• Menus for refining and revising search including suggestions for browse or keywords

6. Help Function

An integrated help function is essential for any database. When children are asked to draw their perfect search interface it usually includes a highly visible help option. But many help functions are more frustrating to use than the search interface.

A good help section is:

• Obvious

• Applicable. Designed with user group in mind

• Contextual. An innovative children’s search interface allows a user to click on the help button then move the question mark over whatever it is on the page the user wants help with (LS2Kids)

(11)

March 2012

• Designed to incorporate different learning styles by using graphics, flow charts, etc (Bilal, 2005)

A well designed, animated search buddy can be very useful, but also very age restricted. What is popular for a grade two student may be way too childish for a grade three student (Large, Nesset, & Beheshti, 2009)

Things to look for in the help section:

• Obvious help button everywhere within database. More than one help button • Contextual help

• Use of images, flow charts, step-by-step instructions • Online and offline help

7. Further Reading

If you would like more details on the above topics here are some suggested resources: For a good summary of recent research on children’s information retrieval:

• Large, A. & Beheshti, J. (2005). Interface design, web portals, and children. Library Trends, 54(2), 318

Developmental stages and their connection with information retrieval:

• Bilal, D., & Wang, P. (2005). Children's conceptual structures of science categories and the design of Web directories. Journal of the American Society for Information Science and

Technology, 56, 1303–1313

Children searching and browsing:

• Cooper, L. (2002). A case study of information-seeking behavior in 7-year-old children in a semistructured situation. Journal of the American Society for Information Science and

Technology, 53(11): 904-922

• Wallace, R., Kupperman, J., Krajcik, J., & Soloway, E. (2000). Science on the web: Students online in a sixth-grade classroom. Journal of the Learning Sciences, 9(1), 75-104.

(12)

March 2012

Evaluating resource relevancy and choosing sources:

• Hirsh, S. G. (1999). Children's relevance criteria and information seeking on electronic resources. Journal of the American Society for Information Science and Technology, 50(14)

(13)

March 2012

Things to Look For: Quick Sheet

Search:

• Spell check • Thesaurus

• Suggestions like “Did you mean” and “Related searches” or “Also try” • Advanced search options showing drop down menus

• Natural language search or Sentence format search capability • Large search boxes

• Search and advanced search available from front page and always easy to access from anywhere in the database

Browse:

• Directory or subject heading access from front page • Age appropriate terminology

• Option to include “tags” or user created subject headings • Clear, simple navigation trail

• “See” and “see also” suggestions • Searchable subject headings

• Alphabetical as well as hierarchical category lists • Subject/ tag cloud

• Clear, consistent use of icons and labels

• “Hot” topics or special curriculum related sections

• Tools to mark or save good resources or good subject headings

Results:

• Availability of both methods, and ability to switch at any stage of the search process from anywhere in the database

• Browse suggestions accompanying keyword search results • Clickable search options in browse results

• Highlighted and clickable subject headings in result descriptions • Query string included on results page

Help Function:

• Obvious help button everywhere within database. More than one help button • Contextual help

• Use of images, flow charts, step-by-step instructions • Online and offline help

(14)

March 2012

8. References

Bar-llan, J., Belous, Y. (2007). Children as architects of Web directories: An exploratory study. Journal of the American Society for Information Science and Technology. (58)6, 895-907. Bilal, D. (2001). Children's use of the yahooligans! web search engine: II. cognitive and physical behaviors on research tasks. Journal of the American Society for Information Science and Technology, 52(2), 118-136.

Bilal, D. (2002). Perspectives on children's navigation of the World Wide Web: Does the type of search task make a difference?. Online Information Review, 26(2): 108–17.

Bilal, D. (2005). Children's Information Seeking and the Design of Digital Interfaces in the Affective Paradigm. Library Trends, 54(2), 197-208.

Bilal, D., & Wang, P. (2005). Children's conceptual structures of science categories and the design of Web directories. Journal of the American Society for Information Science and Technology, 56, 1303–1313.

Boolify http://www.boolify.org/.

Brown, R. (1958). How shall a thing be called? Psychological Review, 65(1), 14-21.

Chen, C., Wu, F., Rau, P., Hung, Y. (2004). Preferences of young children regarding interface layouts in child community web sites, Interacting with Computers, 16(2), 311-330. Retrieved from http://www.sciencedirect.com/science/article/pii/S0953543803001164.

Cooper, L. (2002). A case study of information-seeking behavior in 7-year-old children in a semistructured situation. Journal of the American Society for Information Science and Technology, 53(11): 904-922.

Hirsh, S. G. (1999). Children's relevance criteria and information seeking on electronic resources. Journal of the American Society for Information Science and Technology, 50(14); Retrieved from ABI/INFORM Global.

Hughes-Hassell, S. & Mancall, J. C. (2005) Collection Management for Youth: Responding to the

Needs of Learners. Chicago: American Library Association.

Johnson, P. (2009) Fundamentals of Collection Development and Management, 2nd _{ed. Chicago:}

American Library Association.

Kuntz, J. (1999). Criteria for comparing children's web search tools. Library Computing, 18(3), 203.

Large, A. & Beheshti, J. (2005). Interface design, web portals, and children. Library Trends, 54(2), 318.

(15)

March 2012

Large, A., Beheshti, J., Tabatabaei, N. & Nesset, V. (2009) Developing a visual taxonomy: Children's views on aesthetics. Journal of the American Society for Information Science and Technology, 60(9),1808–1822, DOI: 10.1002/asi.21095.

Large, A., Nesset, V., & Beheshti, J. (2009). Children as information seekers: What researchers tell us. New Review of Children's Literature and Librarianship, 14(2), 121-140.

doi:10.1080/13614540902812631.

Larson, K. & Czerwinski, M. (1998). Web page design: Implications of memory, structure and scent for information retrieval. CHI '98: Proceedings of the SIGCHI conference on Human factors in computing systems. Los Angeles CA USA, 18-23, 25-32.

LS2 Kids. Retrieved from http://ls2.lafourche.org/kids.

Mansor, Y. (2002). Research in children's information seeking behavior. International Association of School Librarianship.Selected Papers from the ...Annual Conference, , 99-110. Retrieved from

http://ezproxy.library.ubc.ca/login?url=http://search.proquest.com/docview/236102611?accou ntid=14656

Masterson-Krum, H. (2001) How safe are kid-safe search engines? Teacher Librarian; 29(1); p. 20. Retrieved from CBCA Complete

Meyers, E. M., Fisher, K. E., & Marcoux, E. (2009). Making sense of an information world: The everyday-life information behavior of preteens. The Library Quarterly, 79(3), 301-341.

doi:10.1086/599125

Nielsen, J. (2002). Kid’s corner: Website usability for children. Alertbox. Retrieved from

http://www.useit.com/alertbox/children.html

Piaget, J. & Inhelder, B. (1969). The psychology of the child. New York: Basic Books.

Rorissa, A., & Iyer, H. (2008). Theories of cognition and image categorization: What category labels reveal about basic level theory. Journal of the American Society for Information Science and Technology, 59(9), 1383-1392. doi:10.1002/asi.20825

Shenton, A. K. (2007). The paradoxical world of young people's information behavior. School Libraries Worldwide,13(2), 1.

Wallace, R., Kupperman, J., Krajcik, J., & Soloway, E. (2000). Science on the web: Students online in a sixth-grade classroom. JOURNAL OF THE LEARNING SCIENCES, 9(1), 75-104.

doi:10.1207/s15327809jls0901_5.

Wolfram|Alpha. http://www.wolframalpha.com.

Zhang, Y., Robins, D., Holmes, J., & Salaba, A. (2009). Understanding internet searching

performance in a heterogeneous portal for K-12 students: Search success, search time, strategy, and effort. Journal of Web Librarianship, 3(1), 15-33. doi:10.1080/19322900802660235.

Database Evaluation: Criteria Review