2.2 Model Building and Learning
2.2.3 Empirical Studies
The second category of literature contains empirical studies of decision maker learn-
ing from involvement in simulation model building. Although the studies discussed
are useful in understanding learning from the involvement of building simulation
models; these in general do not directly compare quantitative simulation model
building to model (re)use; they are either qualitative model building (Shields, 2001,
2002) or based on a full modelling interventions including substantial experimenta-
tion (Thomke, 1998; Rouwette et al., 2010). Moreover, none of the studies to be
discussed is focussed on discrete-event simulation. This section examines the sup-
section.
Before reviewing this literature in detail it is worth pointing out that a number of
experimental SD studies have compared the effect of the level of model transparency
on learning without including participants in the building of the model. Rouwette
et al. (2004) conduct a literature survey of the factors influencing individual decision
maker rationality when using SD simulation models. They find four studies that
support the conceptual discussion regarding transparency aiding learning. These
experiments provide transparency via introduction to the conceptual model (Young
et al., 1992), teaching of SD and the model (Gr¨oßler, 1998; Gr¨oßler et al., 2000) or
providing details of the conceptual model as the participants struggle with the model
(O’Neill, 1992). These studies indirectly support the belief that model building aids
learning through increased transparency.
Turning to the literature that explores learning from involvement in the mod-
elling process, the most recent example is an extensive study into attitude change
conducted by Rouwette et al. (2010). They develop and test a conceptual model for
attitude change in group model building (GMB) interventions based on the Theory
of Planned Behaviour (Ajzen, 1991) and theories of persuasion (Petty and Cacioppo,
1986; Chaiken et al., 1989). These theories are discussed in detail in Section 3.3.
Rouwette et al. use data from seven SD field studies they conduct for different
clients; for example, a housing association, an oil company and the ministry of
transport. The study uses a pre-test post-test (i.e. before and after the intervention)
design making use of content analysis (for project reports), questionnaires and post-
test interviews. In total 14 variables are measured across three areas: context,
mechanism and outcome. A full list of these along with an explanation can be
found in Rouwette et al. (2010); a more detailed account can be found in Rouwette
(2003). Here only an overview of the most relevant findings is given.
of persuasion: the elaboration likelihood model (Petty and Cacioppo, 1986) and the
heuristic-systematic model (Chaiken et al., 1989). Rouwette et al. measure the vari-
ables theorised to be the most important for systematic processing and evaluation of
information: argument quality, assessed by questionnaire and interview, and ability
to process information. These are assessed by a questionnaire taken from Vennix
et al. (1993) and Rouwette et al. (1998). Outcomes contained seven variables based
on another social psychology theory widely used to predict and explain individuals
intentions and behaviour called the Theory of Planned Behaviour (Ajzen, 1991).
This is reviewed in detail in Section 3.3. For now it enough to say that Rouwette
et al measure a change in participant’s attitudes towards options to improve sys-
tem performance. These attitudes are related to participant’s views about the other
members of the groups and the perceived control they have over the implementation
of the options they are investigating.
Rouwette et al. (2010) have three findings that are relevant to learning. Firstly,
they find that ability to process information has only a weak relation to pos-test atti-
tude; and no relationship with either the participant’s views of other members of the
group or the perceived control they have over implementing the options. Secondly,
the intervention appears to have no effect on the perceived control participants have
over implementing the options. Rouwette et al. remark that this is surprising as
most SD practioners would expect simulation to identify control variables with a
large impact on the problem. Lastly, they find that the participants often could not
identify learning outcomes in the study - even if their attitude measures showed sub-
stantial difference from pre to post-test. This agrees with the general view of social
psychologists that individuals struggle to understand their own learning (Nisbett
and Wilson, 1977). Rouwette et al. recommend that participants in a group model
building study are asked to list three options to improve performance before and
In a different simulation domain - finite element simulation - Thomke (1998)
explores learning during the research and development (R&D) of vehicle crash wor-
thiness at BMW. To illustrate learning over the course of the R&D project, Thomke
employs content analysis and extensive interviewing to construct time series illus-
trating engineers’ perceptions of the impact, both magnitude and direction, of differ-
ent variables on crashworthiness. The three examples included in the paper illustrate
the types of learning that could occur. For some variables, attitudes could remain
the same for the majority of the project and then experience a sudden jump in
importance. Others might fluctuate throughout the project.
Given the R&D setting and finite element approach in Thomke’s case study,
the model building approach may be similar to a DES study where the system
being modelled does not already exist. Building can largely be thought of as design:
engineers brainstorm a design for a vehicle based on their assumptions about how
the system will behave, outsource the construction of the model (called meshing)
and analyse results from the new model. Experimentation also clearly plays a role
in the learning process. Thomke points out that numerous learning points for the
engineers were based around the interaction of variables in the model that gave rise
to the crash dynamics observed.
In his closing comments Thomke reiterates the points outlined in the conceptual
discussion of the benefit of involving decision makers in model building.
‘Thus, I propose that the discipline required in developing computer
models will lead to advantages that go beyond making simulation models
available to users. It may also take a firm’s R&D knowledge from tacit to
explicit and, as a result, making it easily transferable within and between
firm boundaries’. (Thomke, 1998).
use. Thus it is difficult to pull out exactly what is learnt in building and what
is learnt in experimentation. For example, in Thomke’s study it might be that the
experimentation with variables in a crash model was the most beneficial for learning.
The one study found that directly compares model building and model use is
concerned with building qualitative models versus simulation use. Shields (2001)
performs an experimental study comparing the building of qualitative SD models
(i.e. casual loop diagrams) from a case study to the use of a SD simulation model
(generally referred to as a management flight simulator in the SD literature). Shields
manipulates these processes by either providing or not providing scripted facilitation.
Facilitators ask participants to describe their assumptions about interactions in the
model, predict what will happen when a chosen strategy is implemented and explain
results and outcomes following feedback on performance.
The study measured pre to post-test learning in five ways: an open ended ques-
tion asking participants to describe the system; two rating tasks where participants
rate the impact of variables on performance; and a diagramming task where par-
ticipants are provided with pre-labelled variables within the system. Marks were
given for the number of connections (complexity) and for inclusion of the direction
of feedback (SD).
Findings showed that the qualitative model building groups where a facilitator is
not provided demonstrated increased understanding of the complexity and system
dynamics of the problem. However, it was the facilitated simulation group that
demonstrated increased performance.