Lancaster (1979) pointed out that ease-of-use is an important criterion for the selec- tion of an information retrieval system. Krichmar (1981) compared Dialog’s and ORBIT’s command language in terms of their ease of use from users’ attitudes and
perceptions. His study is based on the following factors that define ease of use: the difficulty of recalling a command, the effort and frustration involved in entering a
given command, the need to remember the sequence of argument values following a command, not completely understand the meaning of a command. The results showed that frustration with one or more important features of a system could have a negative impact on the perception of an entire system. Researchers have proposed measurable elements for ease of use, such as learnability, speed of user task per- formance, user error rates, and subjective user satisfaction (Hix & Hartson, 1993; Shneiderman & Plaisant, 2004). However, research on the standard measures for ease-of-use is ongoing. Furthermore, ease-of-use is a complicated concept involving different tradeoffs (Thimbleby, 1990).
Not every user prefers ease-of -use. Different users have different requirements for what they need IR systems to do for them. Ease-of -use vs. user control becomes an issue more for online databases because these IR systems were traditionally designed for information professionals and only recently started being designed for end users. These systems have to take into account needs of both novice and expert
users. Bates (1990), in her influential article, asked a reflective question about online
systems: “What capabilities should we design for the system, and what capabilities should we enable the searcher to exercise?” (p. 576).
Influenced by this idea, Xie (2003) studied users’ evaluation of features of a variety
of online databases in terms of ease-of-use and user control based on questionnaires, diaries, logs, and open-ended reports. The results showed that users considered both ease-of-use and user control as important for effective information retrieval. Us- ers’ requirements for ease-of-use and user control did change in the course of their interactions with the system and in the course of learning different systems. They needed more control after they had more understanding of IR systems and acquired more retrieval skills. The results also indicated that experienced users preferred more user control over novice users. While ease-of-use can mostly be achieved by system design, user control can only be accomplished by the collaboration between system design and user involvement. According to Vickery and Vickery (1993), user involvement is the decision that has to be made for interface design. Some interfaces only ask users for information statements, while others require users to be actively involved in the process of formulating search queries by providing guidance for
users. More research is needed to define ease-of-use and user control from users’
perspectives, in particular from different types of user groups to examine whether users have same perceptions of ease-of-use and user control.
Interactve IR n Onlne Database Envronments
Evaluation.Criteria.for.Interactive.Online.IR.Systems
Relevance is a traditional measurement for IR system evaluation, and it is also a crucial measurement for interactive IR systems. However there are issues that need to be dealt with for relevance judgment during user-system interactions. First, it is
difficult to control the situational dynamism of user-centered relevance estimation
during the interaction between users and systems. In studying subjects’ engaging
the LISA ondisc, Bruce (1994) identified a method to allow users to articulate the
cognitive schema for estimating relevance at each phase of the IR interaction: problem state, system interaction, and document interaction. This methodology pro- vides a mechanism for monitoring the impact of the IR interaction on user-centered
relevance judgment. Second, it is difficult for users to have dichotomous choices for relevance judgment for interactive online systems. Researchers have defined
the middle range of relevance to cover partially relevant and partially not relevant in addition to relevant and not relevant based mainly on what is missing and what is present by users (Greisdorf & Spink, 2001; Spink & Greisdorf, 2001; Spink, Greisdorf, & Bateman, 1998). After analyzing 32 users’ searching and evaluating results derived from Dialog, Greisdor (2003) suggested that the relevance judgment process is a problem-solving and decision-making exercise involving cognitive ac- tivities. According to Greisdor (2003), users went through a multiple-stage process of relevance evaluation during IR system interaction, and considered the topicality, pertinence, and then utility of a retrieved item in relevance judgment. Not on topic, not pertinent, not useful, and useful can be associated to not relevant, partially not relevant, partially relevant, and relevant, respectively.
IR system evaluation is a crucial component of IR research. The key question is what the unique criteria for evaluating interactive IR systems are. Su (1992, 1994) conducted a study to identify appropriate measures for evaluating interactive in- formation retrieval. After analyzing the data from 40 users’ interactions with six professional intermediaries searching large online systems, she tried to identify the best evaluation measures for interactive IR performance. The results revealed that value of search results is the best single measure for IR performance. Users’ satisfaction with search results and users’ satisfaction with precision of the search were strongly correlated with value of search results. However, precision is not sig-
nificantly correlated with success. To users, recall is more important than precision. There are several reasons for this: first, high precision does not mean high quality,
and users’ satisfaction with precision is a better indicator of IR performance. Second, users’ tasks that lead them to look for information also affect whether recall is more important to them. The high percentage of users in this study that require complete information to accomplish their tasks (e.g., dissertation/thesis, grant application, etc.)
also influences the result. Users’ satisfaction with the completeness of the search results, users’ confidence in the completeness of the search results, and users’ satis- faction with the precision of the search may serve as good measures of interactive
search performance. Both interaction and effectiveness factors are important in IR evaluation, and interaction factors are more important than effectiveness factors.
In addition, time is a significant factor of success.
Su’s findings demonstrate that relevance is not the only measurement for IR system evaluation. Her identified measurements were partly verified by other studies. Hersh,
Pentecost, and Hickam (1996) compared two commercial MEDLINE systems by applying a task-oriented approach to IR system evaluation, including measuring success at answering questions, user certainty in answering questions, time to answer
questions, ability to find relevant articles, and satisfaction with the user interface.
They concluded that the task-oriented approach was an effective evaluation method for assessing IR systems in terms of whether these systems can be used to solve real information problems. In their large-scale study, Saracevic and Kantor (1988b,
1988c) discussed the five utility measures (worth scale, user’s time, dollar value
assigned, problem resolution scale, and satisfaction scale) as effectiveness measures for IR systems in addition to precision and recall, especially their relationships with relevance odds. They found that when relevance and precision odds increased, users considered the results to be worth more time, to have high dollar values, to make a high contribution to the problem solution, and to provide a high level of satisfaction..
One utility measure is related to recall odds. When recall odds increased, less time was taken for users to evaluate results. The major contribution of this study is the
identification of the utility measures and their relationship to relevance odds. Although researchers used different terms to name evaluation criteria, they identified similar key evaluation criteria. However, the identified evaluation criteria mainly focused
on the search performance of online systems, they failed to assess the user-system interaction process in online searching.
Summary
One unique phenomenon in online database environments is that intermediary studies have accounted for a large portion of the interactive studies mainly be- cause professional intermediaries were the main searchers of online databases before the emergence of the Web. The cost and complexity of command language have contributed to the problem. At the same time, intermediary studies can shed some lights on how users interact with searchers, online systems, and documents. In online environments, intermediary studies have contributed to the research on domain knowledge’s impact on online searching by Shute and Smith (1993); types of interactive feedback by Spink (1997) and Spink and Saracevic (1998); cognitive styles affecting information seeking behavior by Ford et al. (2002); shifts in search problems/stages/focus by Robins (1997, 2000),Spink and Wilson (1999) and Olah (2005); intermediaries’ elicitation styles by Wu and Liu (2003); and evaluation criteria
Interactve IR n Onlne Database Envronments
for interactive IR systems by Su (1992, 1994). Many of these studies also suggest
how to incorporate their findings into system design, specifically to implement the
role of the intermediary into the design of online IR systems. Table 3.1 presents a summary of interaction studies in online database environments.
Task studies enable researchers to understand the impetus for information retrieval and to further develop theories of task-based IR process. The remaining question is: What is the relationship between tasks and user goals? User goals are also con- sidered the driving force of information retrieval, as discussed in chapter 2. Are
tasks a part of user goals? How can tasks and user goals be defined? In addition, how can the complexity of tasks be defined? Are levels of task complexity different for different users, or is there a standard way to define them? What are the other dimensions of tasks that influence online searching? These questions need to be
investigated further.
Levels of search strategies are the center of attention in interaction studies of online databases. Compared with OPAC studies, researchers have conducted more in-
depth studies on search strategies, and have identified different types of micro- and
macro-levels of strategies. However, the strategy studies are still on the level of the
identification of the types of search strategies; they do not go further to explore what
lead to the users’ application of different search strategies. In addition, researchers need to further examine the relationships among tactics, moves, and strategies. Are tactics and moves a part of strategies, and if so, how are strategies constituted by
them? Identification of shifts in search strategies, stages, and foci is just the first
step in understanding users’ information-seeking behavior during their interactions with intermediaries, IR systems, and information. In order to design IR systems to facilitate those shifts, we need to further identify the patterns between the shifts and the factors that lead to the shifts.
In general, researchers agree that domain knowledge and information retrieval knowledge affect users’ information-seeking behavior and search performance. Expert users can make better use of domain knowledge than novice users. While providing term selection is a popular tool for assisting domain knowledge, offer- ing different interfaces for expert users as well as novice users is a suggestion for offering retrieval knowledge help. However, research on knowledge structure has not been incorporated into the design of Help systems for online databases. That is why online Help is inadequate in existing online systems (Trenner, 1989; Xie & Cool, 2000). Further research needs to look into when and how users need differ- ent types of knowledge, and the interactions among different types of knowledge and their impact.
Although there is a disagreement about whether searcher characteristics affect search performance, researchers do agree that searcher characteristics, especially
their cognitive styles/search styles, do influence searchers’ behavior. Interactive
Table 3.1. Summary of interaction studies in online database environments
Types Research.Focus Problems/Questions Implications Tasks.and.their.
impact Impact of stages of tasks on search behavior and search performance; Impact of complex- ity of tasks on search behavior and search performance.
What are the relationships between tasks and user goals?
How can levels of task
complexity be defined?
What are the other dimen-
sions of tasks that influ- ence online searching?
Develop theory of task- based IR process; Understand the driving force for information retrieval;
Incorporate the stages of tasks and corresponding information seeking be- havior into system design.
Levels.of.search.
strategies Types of search tactics, moves, and search strategies.
What are the relationships among search tactics, moves, and search strate- gies?
What factors affect dif- ferent levels and types of search strategies?
Understand patterns of search behavior; Design IR systems to facilitate users apply- ing different levels and different types of search strategies.
Shifts.in.search.strat- egies,.seeking.stages,. and.foci
Shifts in search strate- gies, seeking stages, foci;
Factors leading to shifts in strategies, stages, and foci.
What are the patterns among factors that lead to the shifts and shifts in strategies/stages/.foci?
Understand the nature of interactions between users and intermediaries, IR systems, and information; Design IR systems to facilitate/guide the shifts.
Users’.knowledge.
structure. Domain knowledge and information retrieval knowledge affect search behavior and search performance
When and how users do need different types of knowledge?
What are the interplays among different types of knowledge and their im- pact on online searching?
Provide term selection to assist users with domain knowledge;
Offer multiple interfaces to novice users and expert users.
Searcher.charac- teristics/.Cognitive. Styles/Search.Styles
Attitudes, mathematic ability, cognitive styles, and search styles and their impact on search behavior and search outcome.
What are the relationships between cognitive styles and search styles? What role should systems play: try to help users with different styles or introduce all the styles to users so they can inte- grate different styles?
Understand the impact of users’ characteristics on their search behavior and search performance; Design IR systems to fa- cilitate users with different cognitive/search styles.
Interactve IR n Onlne Database Envronments
styles. While the cognitive styles of users affect their search styles, the relationships between cognitive styles and search styles need to be further explored. Each style
has its benefits and problems. The problem is whether system design should just
try to help users with different styles or it should guide users to integrate different styles.
In order to design effective interactive IR systems, it is important to understand what users desire for ease-of-use and user control. IR system design needs to consider system role as well as user involvement. The key point is that we need more research to understand users’ perceptions of ease-of-use and user control from diverse groups, such as novice users vs. expert users, female users vs. male users, younger users vs. older users, and so forth.
In order to design effective interactive IR systems, it is also important to identify the criteria for evaluating those systems. Research on the evaluation of online IR
systems focuses on the identification of the appropriate criteria for improving the
interactivity of existing IR systems. However, in these studies, there is no clear
definition of interactive IR systems. Moreover, the evaluation criteria are limited to
system performance and utility; they need to be extended to assess the interaction process between users and systems.
Ease-of-use.vs..user.
control Ease-of use vs. user control; System role and user involvement.
Do users from diverse user groups have the same perceptions of ease- of-use and user control?
Understand what users desire for ease-of-use and user control;
Design IR systems to bal- ance system role and user involvement.
Evaluation.of.inter-
active.IR.systems System performance criteria; Utility criteria.
How can interactive IR
systems be defined?
What are the criteria needed to evaluate the interaction process between users and online IR systems?
Determine the appropri- ate criteria for evaluating interactive IR systems; Improve the interactivity of existing IR systems based on evaluation results.
References
Anonymous. (2005). H. W. Wilson unveils WilsonWeb 2.5. Information Today, 22(8), 51-52.
Anonymous. (2006). Scientific online searching via handhelds. Information Today, 23(1), 36.
Anonymous. (2006b). H.W. Wilson periodicals databases offer color pages. Infor- mation Today, 23(2), 35.
Barker, A. L. (1998). DataStar Web: A comparison with “Classic” DataStar command language searching. Online and CD-ROM Review, 22(3), 155-167.
Bates, M. J. (1979a). Information search tactics. Journal of the American Society for Information Science, 30(4), 205-214.
Bates, M. J. (1979b). Idea tactics. Journal of the American Society for Information Science, 30(5), 280-289.
Bates, M. J. (1990). Where should the person stop and the information search inter- face start? Information Processing and Management, 26(5), 575-591. Bates, M. E. (1997). Knight-Ridder on the Web: A brave new world for searchers?
Searcher, 5(6), 28-37.
Bellardo, T. (1985). What do we really know about online searchers? Online Review, 9(3), 223-239.
Bruce, H. W. (1994). A cognitive view of the situational dynamism of user-centered relevance estimation. Journal of the American Society for Information Sci- ence, 45(3), 142-148.
Byström, K., & Järvelin, K. (1995). Task complexity affects information seeking and use. Information Processing and Management, 31(2), 191-213.
Chen, H., & Dhar, V. (1991). Cognitive processes as a basis for intelligent retrieval system design. Information Processing and Management, 27(5), 405-432. Davis, C. H. (1977). Computer programming for librarians. Journal of Education
for Librarianship, 18(1), 41-52.
Drabenstott, K. M. (2003). Do nondomain experts enlist the strategies of domain experts? Journal of the American Society for Information Science and Tech- nology, 54(9), 836-854.
Ellis, D., Wilson, T. D., Ford, N., Foster, A., Lam, H. M., Burton, R., et al. (2002). Information seeking and mediated searching: Part 5. User-intermediary interac- tion. Journal of the American Society for Information Science and Technology, 53(11), 883-893.
Interactve IR n Onlne Database Envronments
Feldman, S. E. (1996). Dow Jones Interactive News/Retrieval’s new software: Something for everyone. Searcher, 4(6), 42-48.
Fenichel, C. H. (1981). Online searching: Measures that discriminate among us- ers with different types of experience. Journal of the American Society for Information Science, 32(1), 23-32.
Fidel, R. (1985). Moves in online searching. Online Review, 9(1), 61-74.
Fidel, R. (1991). Searchers’ selection of search keys: I. The selection routine: II. Controlled vocabulary or free-text searching: III. Searching styles. Journal of the American Society for Information Science, 42(7), 490-527.
Ford, N., Wilson, T. D., Foster, A., Ellis, D., & Spink, A. (2002). Information seek- ing and mediated searching: Part 4. Cognitive styles in information seeking.
Journal of the American Society for Information Science and Technology, 53(9), 728-735.
Ford, N., Wood, F., & Walsh, C. (1994). Cognitive styles and searching. Online and CD-ROM Review, 18(2), 79-86.
Greisdorf, H. (2003). Relevance thresholds: A multi-stage predictive model of how users evaluate information. Information Processing and Management, 39(3), 403-423.
Greisdorf, H., & Spink, A. (2001). Median measure: An approach to IR system