Method similarities and differences

in interactive information retrieval and decision-making is how they prioritize information search vs. the underlying task motivating the search (e.g., decision, problem-solving, work, etc.). The decision-making

process and outcome is the primary focus in decision-making research. In these studies,information search

generally means external information search (i.e., seeking information from sources other than one’s memory). As information search and information source selection were not the focus of the studies, many experimental decision-making studies have often provided information to participants about the alternatives and attributes to consider using a matrix or information board such as MouseLab. This helps to isolate the decision from information acquisition as well as controls the amount and types of information available. Many decision- making studies have examined differences in patterns of information use during the decision-making process and decision quality under different time constraints.

In IIR research, the process of how people use search tools or other resources to acquire information is a primary focus. The outcome of the underlying task triggering the information-seeking (e.g., decision, problem-solving) may be often of secondary importance. Researchers generally provide one or more search systems for people to use to search for information by querying, examining results, and reading documents. The size of the choice set for the decision (i.e., the number of alternatives and attributes) is dependent on what the decision-maker already knows or finds through information-seeking or search.

In IIR studies, the search system might return results from the open web or from a closed corpus for which the researcher may know the set of documents which have been judged as relevant to a set of predefined tasks (e.g., the AQUAINT corpus used in Crescenzi et al. (2015)). While (I)IR researchers can use measures derived from expert relevance assessments such as precision (i.e., the proportion of the retrieved documents that are relevant) or recall (i.e., the proportion of the relevant documents retrieved) to assess the quality of a system or search, these measure topical relevance rather than the usefulness of the information to the user in their specific situation (Cole et al., 2009).

Lab-based experimental studies are used in both decision-making and information search research. As previously noted, studies of decision making and information-seeking investigate outcomes as well as the cognitive and metacognitive processes involved using process tracing methods. Although the methods in decision making and information-seeking are similar, the primary emphasis in a decision-making study is the decision rather than information seeking and information search. As such, it is common for the information source(s) and extent of information available to be experimentally controlled or manipulated so that the subprocesses of information acquisition and integration can be carefully examined. In contrast, the emphasis in an information-seeking study is the information-seeking and search (which may or may not be used to make a decision).

The outcome to be assessed in decision-making studies is dependent upon the type of decision (e.g., choice vs. judgment) and the task to be completed. Outcomes might include the number of people who chose the more risky alternative (Ben Zur & Breznitz, 1981; Huber & Kunz, 2007), or the frequency in which the better alternative is selected (i.e., dominant alternative selected or dominated alternative rejected). Other outcome measures include the number or rate of problems solved correctly (Baumann, 1998; Hertzum & Holmegaard, 2013a), or an assessment by experts’ of the quality of a written product (Karau & Kelly, 1992; van der Kleij, Lijkwan, Rasker, & De Dreu, 2009). Some researchers have also created measures that take into account the extent to which the decision made meets the criteria of the decision maker such as the proportion of satisfactory attributes (Hahn et al., 1992). Some types of decisions cannot be assessed for objective accuracy (e.g., preference decisions) whereas other decisions may be more objectively accurate than others (e.g., inference based on evidence) (Gigerenzer & Gaissmaier, 2011; Zajonc, 1980). For example, a preference decision is subjective; the participant makes a selection based on their own preferences. The accuracy of a preference decision can be assessed for the extent to which it meets the decision-maker’s preferences, but there may not be an objectively correct decision. On the other hand, a researcher can assess whether a participant made a correct inference based on the information provided such as choosing the less risky option.

Researchers also evaluate decision-making processes by analyzing the actual process of the decision- makers. The relative costs and benefits of decision strategies can be formally and quantitatively operational- ized to compare strategies along the dimensions of effort and accuracy. J. W. Payne, Bettman, and Johnson (1988, 1993) operationalized effort as the sum of the frequency of each type of individual actions in a decision and the accuracy of a decision was operationalized as a relative accuracy metric which compared the decision strategy used with the most effortful (i.e., that which maximized expected value) and the least effortful (random choice). An observed decision-making process can be compared to an optimal or compensatory de- cision process using the relative accuracy measure. Similar methods are used in human-computer interaction research to compare the process of an actual interaction with a system to an idealized process.

To measure the effort or cost of a decision strategy, decision-making and information-seeking researchers have decomposed processes into individual actions or sequences of actions at different levels of granularity. For example, Johnson and Payne (1985) decomposed the decision-making process into elementary information processes (EIPs). Strategies for decision-making and information search have been decomposed into sequences of actions at different levels of granularity (e.g., strategy, tactic, and move) for cognitive activities

(e.g., Bates, 1979b); physical actions such as tactics (Bates, 1979a; Marchionini, 1989) or moves within tactics (Wildemuth, 2004); or cognitive and physical activities (e.g., Wildemuth, Kelly, Boettcher, Moore, & Dimitrova, 2018; Xie & Joo, 2010, 2012). Similarly, methods like GOMS (Goals, Operators, Methods, and Selection rules) and variants are used in efficiency evaluations of information systems (John & Kieras, 1996).

2.2.3 Time pressure in experimental studies. Time pressure has been successfully induced in ex-