7.4.1 Timings of Attitude Measurement
One factor to consider when assessing the applicability of results to real world sim-
ulation studies relates to the time frame of the experiment. In particular the ex-
periment measures the participant’s attitude immediately after the completion of
the simulation study. This could be quite different from a study in industry: real
decision makers may wait for, at least, a small period of time before committing to
any implementation action.
Dikjksterhuis et al. (2006) report on a series of experiments testing the ‘deliber-
ation without attention’ hypothesis. For example, in one experiment, when deciding
on which car to buy one group of participants were given time to consciously pro-
cess (‘deliberate’) information about the car. A second group were given the same
time, but were distracted (they solved anagrams). Participants in this experiment
(and field work) made better and more consistent decisions about complex pur-
chases when they were distracted. One possible mechanism leading to this result
is believed to be the relatively low capacity of conscious compared to subconscious
processing. Thus, if the participants in this research were asked to ‘sleep on it’ and
then report their attitudes the next day a different result may have occurred due to
subconscious processing.
The studies discussed by Dijksterhuis et al (2006) do not invalidate the current
research, but it does mean that the conclusions drawn from the results must be
considered carefully. Clearly in a real simulation study, incorporating either devel-
opment or reuse, clients would most probably have time to process (both consciously
One area the experiment can apply to is the snap or early judgements made
by decision makers. This is important because in real simulation studies decision
making is likely to be made in groups. Thus, as detailed by theories of persuasion
(E.g. Chaiken et al., 1989) and behaviour (E.g. Ajzen, 1991), a decision maker will
take into account the early judgements of other members of the group. The early
judgements of influential members of a group are likely to be important in how other
members of the group choose to view the results after they have ‘slept on it’. This
makes the mechanisms for learning discussed very important, if one is interested
in implementation actions following a simulation study. Especially since we cannot
expect decision makers to systematically review how and if they have changed their
minds (Rouwette, 2003).
7.4.2 Novice Decision Makers
A second difference between the experiment and real life is the level of expertise or
perceived self expertise that a client has in the behaviour of the system of study.
Empirical studies support the view that the greater the knowledge an individual
has about a topic (or the greater the self confidence in one’s own knowledge) then
the lower the likelihood that advice is accepted (Yaniv, 2004; Kantowitz et al.,
1992). This likelihood drops even further as the advice moves further away from
the opinion of the individual (Yaniv, 2004). The hypothesis put forward to explain
credibility assessment in simulation studies accounts for this by including the effect
of perception of expertise on the sufficiency threshold for V&V, i.e. more persuasion
is needed for individuals with higher belief in their knowledge of the system.
Clearly the participants in the experiment of this study are system novices and,
given the empirical evidence, their credibility assessments are easier to manipulate.
This means that the higher variability in MR credibility assessment is more striking
were fairly open to taking advice from the model. Similarly, the (albeit weak)
evidence of a lower self confidence in assessment of the model given the more rigid
structure of MBL might be magnified in a study with ‘domain experts’. This would
be especially so if the simulation model results were drastically different from those,
implicitly, expected by the client.
A similar perspective can be taken when interpreting the attitude change results.
The manufacturing managers and CEOs that partook in the Suri (1998) question-
naire might find it more difficult than students to reflect on their beliefs about
resource utilisation and be persuaded that 100% utilisation results in large queues.
In fact the students are system novices and it should be easier to persuade them
to do the right thing. Thus results that indicate failures to change the attitudes of
novices are useful as this suggests that experts would certainly not be persuaded.
Lastly, it is likely that the students will have reduced ‘buy-in’ (or motivation
to understand) to the case study problem compared to real managers and decision
makers with their own ideas and beliefs about how to improve performance. This
extra motivation may improve the chances of attitude change. Although motivation
in the experiment may not be as high as found in a real study some confidence in
motivation levels can be drawn from both observations in the experiment and other
empirical studies of statistical reasoning.
The first point that indicates that participants were highly motivated comes from
the MBL condition. Anecdotally it was pointed out that many of the MBL partici-
pants wanted to simulate additional scenarios beyond the three that were prescribed
without any prompting. Several of the participants became quite frustrated when
they were not allowed to perform the additional experimentation believing that they
had ideas that would improve performance. One participant, MBL21, took this a
step further and e-mailed a number of scenarios (regarding changing the shift times
In addition to the anecdotal evidence of this study, Brase et al. (2006) conducted
four experiments of participant recruitment methods and their impact on statistical
reasoning performance. Findings suggest that performance is influenced by the
ranking of the (U.S.A based) university (i.e. students from higher ranked universities
perform better) and financial incentives (i.e. students are more likely to engage
when paid or when a chance to win more money is present). As participants of the
current study come a top ten ranked U.K University and were given several financial
incentives some confidence can be placed in level of engagement achieved.