Mental models elicitation methods 44

Mental models are abstract and, as a result, difficult to measure. The most used

methods are interviews and think-aloud protocols, drawing, and naturalistic observation. A host of other techniques have also been used to elicit and represent mental models, such as repertory grid technique, concept listing, pair-wise rating, and concept mapping.

2.4.1.1 Interviews and think-aloud protocols

Researchers have employed both direct and indirect probing strategies in interviews. Direct probing asks participants to describe the system or how the system works (e.g. Borgman, 1984; Dimitroff, 1992; Muramatsu & Pratt, 2001; Slone, 2002). For example, Slone (2002) asked subjects the question, “Can you describe the Internet to me?”.

Indirect probing asks subjects to provide analogies or metaphors to the system under study (Bruce, 1999; Sasse, 1997). For example, Bruce (1999) asked subjects to articulate their conceptualization of the Internet by completing the sentence “Internet is like a…”. The indirect approach assumes that mental models are transferable from one system to another. In both cases, answers to the interview questions were transcribed and content analyzed to represent subjects’ mental models.

Both probing strategies are unstructured, imposing little restriction on users’

articulation of their mental models. Another form of interviews, semi-structured interviews, provide a loose framework for users to express their mental models, while allowing

subjects to express ideas in any way they want. Zhang (2008b) generated a set of interview questions intended to probe into four aspects of subjects’ mental models of the web (i.e., information sources, information organization, search mechanism, and interface). Subjects expressed their thoughts in the framework established by the interview questions. However, the researchers could adjust the order of the questions based on subjects’ responses so that subjects were able to talk freely and express any ideas that came into their minds. New

dimensions, such as affective aspects of mental models, were able to emerge from the interview. Mental models also can be elicited by structured interviews. Sharit, et al. (2008) designed 10 interview questions to access subjects’ mental models of the internet, web browsers, and search engines. For every subject, the questions were asked in the same order and the responses to each question were scored immediately by the researcher.

Think-aloud protocols are another frequently used method to get people’s verbal accounts of their mental models (e.g. Clement, 1983; Katzeff, 1990; Makri, et al, 2007). For example, Katzeff (1990) asked subjects to think aloud when they performed search tasks in a news database. The subjects’ mental models were constructed by analyzing the reasoning process reflected in the think-aloud comments. An example of a subject testing an incorrect mental model and finally forming a correct model is shown by the transcript: “If I write “next”… I will probably get the same article… now we’ll see … No, it is the next article … I should have done this “expand” in between in order to stay within the same article…”.

A variant of a think-aloud protocol, the teaching back technique, also has been used in mental models study. In a teaching back scenario, participants are asked to teach another person how to use the system by verbalizing their knowledge about the system (Sasse, 1991). The most frequently asked questions are of two types: “what is?” and “how to?” (Van der Veer & Puerta Melguizo, 2001). Sasse (1997) used this approach to elicit subjects’ mental models of a spreadsheet application.

2.4.1.2 Drawing

Drawing is a primitive form of communication. Gray (1990) pointed out that drawings can be used to elicit and illustrate structural aspects of users’ mental models. Denham (1993) used drawing as a means to examine children’s conceptions of computers. To investigate how people conceptualize Web search engines, Efthimiadis and Hendry (2005) asked 279 students at the University of Washington to draw sketches of how an Internet search engine works. Zhang (2008a) asked 44 undergraduate students to draw their

perceptions about the Web. In a proposed framework for IR systems evaluation and design, Pejtersen and Fidel (1998) pointed out that questions like, “Please draw a diagram or picture of it,” can help evoke mental models of the Internet.

Empirical research also suggests that drawing is an effective method to represent mental models (Kerr, 1990). Thatcher and Greyling (1998) used drawing to elicit and categorize people’s conceptualizations of the Internet. Six categories of mental models were derived: interface and utilitarian functionality, central database, user to the world, simple connectivity, simple modularity, and networking. The categories were found to be significantly correlated with the subjects’ experience with the Internet. Papastergiou (2005) collected sketches from 340 high school students in Greece. Similar categories of mental models, from utilitarian to structural ones, were identified. Significant correlations were found between students’ drawings and their answers to a set of internet related questions.

elicit people’s cognitive structure and mental models in psychology and education research. For example, Chang (2007) used the concept mapping technique to externalize students’ mental models regarding the homeostasis of blood sugar and concluded that concept mapping is able to differentiate students’ mental models and it is a workable method for representing mental models of complex and abstract concepts. However, there is a paucity of research using the method to represent users’ mental models of information systems.

2.4.1.3 Observation

Observing users’ interactions with the system when they conduct searching tasks is another commonly used method to study mental models. This type of study usually

analyzes errors, command patterns, or behavior patterns as a means of gaining insights into the mental models of the subjects. For example, Huang (1992) reported a study exploring subjects’ pause behavior when searching the DIALOG system to shed light on subjects’ mental models (although mental models was not the main theme of the research). Chen and Dhar (1990) identified users’ misconceptions of information retrieval systems by observing 30 subjects performing searches in an online library catalog. By this means, gaps in

subjects’ mental models of the system were identified.

Behavioral data gathered by observation are more objective and more reliable than verbal accounts gathered from the user (Norman, 1983). However, the conceptualization derived from observation does not reveal much information about a user’s cognitive model: the researcher will see what subjects did, but know nothing about why they did it (Sasse,

1991). Researchers have to construct users’ reasoning processes (mental models) based on the observed behaviors. Making accurate inferences about users’ mental models from the behavioral data is challenging (Marchionini, 1989a). Researchers with different

backgrounds might have different interpretations of the same sequence of behaviors. Because of the inherent constraints of this method, there are very few studies utilizing the single observation method to represent mental models. Rather, observation is used to complement verbal accounts (think-aloud protocols and interviews) in most of the mental model studies (e.g. Borgman,1989; Katzeff, 1990; Kerr, 1990; Makri et al., 2007; Marchionini, 1989a; Muramatsu & Pratt, 2001).

2.4.1.4 Other methods: repertory grid technique (RGT), concept listing, and pair-wise rating.

The repertory grid technique (RGT) generates a list of elements and a list of constructs based on the elements. An element is defined as things or events under investigation, such as books or search engines. A construct is an attribute of an element. It is a bipolar

dimension, where each pole represents the extreme of a particular view or observation of an element. For example, “has cache feature” and “no cache feature” consist of the two ends of a construct to evaluate Web search engines (Crudge & Johnson, 2004).

Zhang (1997) employed the RGT to represent users’ mental models of an IR system. In his study, nine concepts (elements) provided by experts were used to represent components of IR systems and three attributes of the concepts (constructs) represented properties of

those components. The nine concepts were browsing, classification, data structure, document content, feedback, information needs, interface, query, and search. The three attributes were three dimensions on which concepts were judged: form/process,

targeted/untargeted, and specific to IR systems/applicable to all information systems. Mental models thus were represented and measured by subjects' rating on nine concepts on the dimensions represented by the three attributes.

Wang et al. (2004) used the concept listing method to explore the development of students’ knowledge structure of a subject domain over a semester. In the study, a group of students enrolled in an information organization class were asked to list terms related to the class subject during a timed session over a semester. They found that the number of terms and concepts increased over the semester and the quality of the vocabulary also increased. More importantly, terminologies listed by some subjects formed distinct clusters on particular topics in information organization.

In the pair-wise ratings method, a predefined set of central concepts of a domain will be created and participants will be asked to rate the similarity or relatedness for each possible pair of concepts in the total concept pool. These ratings are transformed into a matrix, which is then analyzed by Pathfinder, a graph theoretic technique that derives network structures from rated data (Schvaneveldt, 1990). The concepts set can be created by domain experts or derived from system documents. The pair-wise rating method has been used to elicit subjects’ mental models of mobile phone networks (Langan-Fox et al.,

2006; Hanisch, Framer, & Hulin, 1991).

2.4.1.5 Summary

Each technique has its limitations in representing mental models. Verbal accounts depend on people’s ability to articulate their mental models and mental models inferred from the information retrieved from memory do not necessarily reflect the way that the information is stored in memory (Cacioppo, von Hippel, & Ernst, 1997). Drawing and concept mapping allow subjects to present their thoughts in an integrated manner, but the interpretation of drawings or concept maps is subject to researchers’ understanding. Meanwhile, drawings are not effective in representing abstract concepts (Zhang, 2008b). Observation allows researchers to see the real behavior of users, but inferring the cognitive mechanism underlying the behavior is very subjective. More structured methods, such as structured interviews, RGT, pair-wise ratings, and concept mapping, restrain the emergence of dimensions and might misrepresent mental models.

In fact, researchers often employ multiple methods in combination to explore mental models. Muramatsu and Pratt (2001) used comments from post-session interviews, along with video-taped search sessions, to infer subjects’ mental models of web search engines’ query processing mechanisms. Zhang (2008b) combined data from drawings, drawing descriptions, and interviews to represent subjects’ mental models of the web. It was found that data collected by drawing and by interview methods supplemented each other and, together, they provided a more holistic representation of mental models.