Critical Decision Method

Critical Decision Method (CDM) constitutes one of the Cognitive Task Analysis methods. Klein, et al (1989), developed the CDM for their naturalistic researches (Klein, 1989; Klein and Brezovic, 1986; Klein, Calderwood and Cirocco (1986). Cognitive Task Analysis: the purpose of the methods is to systematically uncover the decision making requirements and psychological processes used by expert individuals – for the purpose of this research experienced online users, in accomplishing results (mentamodels.mitre.org). Cognitive Task Analysis defines the actions of individuals so as to come to decisions. Critical Decision Method is based on Flanagan’s (1954) critical incident technique. According to Flanagan (1954), the critical incident technique consists of a set of procedures for collecting direct observations of human behaviour in such a way as to facilitate their potential usefulness in solving practical problems and developing broad psychological principles. These are important for identifying anomalies and states of mental unease in the decision making process.

The CDM as developed by Klein et al. (1989) is a descendent of the critical incident technique developed by Flanagan (1954). In common with Flanagan’ s technique, the CDM explores decision-making in previously experienced non-routine cases. Participants are requested to bring to mind an incident in which their expertise made a difference to the outcome. This incident is recalled in detail and a time-line constructed to establish the precise sequence of events (O’Hare et al, 2000).

5.4.1 First Sweep: Incident identification and selection: At this stage, the participants will report an event where they use decision making process in order to book online. Once the participant will identify the incident, he/she will be asked to recount the episode in its entirety. Once the participant complete his/her initial recounting, the analyst will retell the story.

65 5.4.2 Second Sweep: Timeline verification and decision point identification: In this phase of the interview, the participant will asked for a second time to go back through the incident in order to structure and organize the incident into ordered segments. It is important to construct an accurate timeline of the incident under analysis. The aim of this is to give the analyst a clear picture of the incident and its associated events.

5.4.3 Third Sweep: Progressive, Deeping and the Story Behind the Story: During this sweep the analyst will let the participant back over each identified segment (decision making process) the incident, while employing probes designed to focus attention on particular cognitive aspects of the incident. Each selected decisions of interest would then be probed or analysed further (Table 3: using probes of O’Hare et al, 2000).

5.4.4 Fourth Sweep: “What If” Queries: This sweep helps to uncover where the decision process might break down. According to Hann et al (2003), it points out vulnerabilities of a process, but also highlights where expertise is required.

Originally, these Critical Decision Method probes were developed by O’Hare et al (2000), but for the purposes of this research this was revised and expanded.

5.4.5 Critical Decision Method Probes: The Critical Decision Method (CDM) Klein et al, 1989) is a cognitive task analysis approach that uses semi-structured interviews and cognitive probes in order to analyze the cognitive processes underlying decision making in complex environments. Using this approach, critical incidents are decomposed into decision points, and so-called ‘cognitive probes’ (targeted interview probes focusing on cognition and decision making) are used to identify the cognitive processes underlying operator performance at each decision point.

Table 3: Critical Decision Method Probes

Cognitive Cue Sample Question Goal

specification

What were your specific goals at the various decision points? Goal

identification

What features were you looking at when your formulated your decision? How did you know that you needed to make the decision? How did you know when to make the decision?

Expectancy Were you expecting to make this type of decision during the course of the event? Describe how this affected your decision-making process?

Conceptual model

Are there situations in which your decision would have turned out differently? Describe the nature of these situations and the characteristics that would have changed the outcome of your decision?

66 Influence of

uncertainty

At any stage, were you uncertain about either the reliability or the relevance of information that you had available? At any stage, were you uncertain about the appropriateness of the decision?

Information integration

What was the most important piece of information that you used to formulate the decision?

Situation awareness

What information did you have available to you when formulating the decision? Situation

assessment

Did you use all the information available to you when formulating the decision? Was there any additional information that you might have used to assist in the formulation the decision?

Options Were there any other alternatives available to you other than the decision that you made? Why were these alternatives considered inappropriate?

Decision blocking

Was there any stage during the decision making process in which you found it difficult to process and integrate the information available? Describe precisely the nature of the situation

Basis of choice Do you think that you could develop a rule, based on your experience, which could assist another person to make the same decision successfully? Do you think that anyone else would be able to use this rule successfully? Why /. Why not?

Generalisation Were you at any time reminded of previous experiences in which a similar decision was made? Were you at any time reminded of previous experiences in which a different decision was made?

Source: O’Hare et al (2000)

The difference between what people think at the time of the experience and after the experience is important. One is a perception by reflecting on the past; the other is based on actual behavior at that time. Eye gaze does not lie but answers can misrepresent the truth.