4.5 Experiment Procedure
4.5.4 Model Reuse
Participants in MR undergo the same simulation education as MB and MBL. How-
ever, MR participants are informed that a model was developed for a different
computer and conceptual models as well as the batch run results. The following
procedure is then followed:
1. The model is run in visual interactive mode and participants are talked through
the process; for example arrivals, priority queuing, emergency treatment and
visiting radiology;
2. The explanation for the result screen (Figure 4.5) given to MB and MBL at
stage one of model building is repeated;
3. Participants are presented with a list of assumptions and simplifications in the
model.
An alternative procedure could present the list of assumptions and simplifica-
tions to the participants prior to the Simul8 model. This would be equally valid,
but experience in the pilot suggested that participants were not really sure what
to do with them until after they had viewed the computer model. The order de-
tailed above appeared to help participants see the relevance of the assumptions and
simplifications and reflect on the simulation model.
Participants are reminded that the model was developed for another similar
hospital. Thus they must assess the models fitness for purpose. They are allowed to
ask questions about the models logic, results and simplifications and assumptions.
Once participants are satisfied that suitable V&V has been completed, the same
experimentation procedure is undertaken that was used in MB.
4.6
Summary
This chapter provides an overview of an experiment to test the hypotheses that
involvement of decision makers in model building aids learning - both single and
Figure 4.8: High level view of experiment
towards the management of three aspects of an A&E queuing system are measured
before and after the use of a simulation model to analyse the problem. The simu-
lation study process that participants are involved in is manipulated depending on
the condition: participants are either involved in development of a model followed
by extensive use (MB), involved in development of a model followed by limited
use (MBL) or involved in the reuse of a model (MR). It is predicted that the MB
and MBL should aid attitude change in the ‘correct direction’ and restrict attitude
change in the ‘incorrect direction’ more so than MR.
Following the post-test attitude questionnaire, participants answer reasoning
questions that present analogous queuing problems to the A&E department. These
transfer scenarios are classed as either close, i.e. within a healthcare context, or
far, i.e. within call centre and manufacturing contexts. It is predicted that the MB
and MBL conditions will achieve higher transfer success relative to MR. It is further
predicted that all conditions will show some extent of double-loop learning, i.e. a
correlation between attitude change and transfer, but that the relationship will be
strongest in the model building conditions.
The procedure developed for model building requires the participant to take the
role of a domain expert rather than a modeller. This means that the participant
is involved in conceptualising and validating the model, but not direct building.
Participants are able to make choices about the level of detail within the model
and review the results of their suggestions. Although the choices participants can
obviousness.
The next chapter provides a description of single-loop learning for participants
in each condition. This is followed by a formal comparison of the conditions and
Chapter 5
Single-Loop Learning Results
5.1
Introduction
A theory-of-action perspective on learning assumes that individuals have a defini-
tion of effective performance for a system. For example, an individual may define
effective performance of an A&E system as ‘very high utilisation of resources and
quick turnaround of admission, treatment and discharge of patients’. When working
on an instrumental problem, such as improving performance of an A&E department
against a target, individuals will strive to meet the objectives of this definition of
effective performance - perhaps without realising any relationships between objec-
tives or factors that may be missing. If this happens then an individual might
be surprised when the results of their actions do not fit with their expectations
of performance. Under theory-of-action assumptions an individual is more likely
to try to find solutions that fit with their definition of effective performance than
examine the definition itself. In other words attitudes in what action to take may
change, but deeper understanding and objectives may remain the same. This is called single-loop learning and is the focus of this results chapter.
view of attitude and supporting variables used in the analysis. The second section
discusses the analysis methodology used in the research and the alternatives. The
third section is organised by experimental condition and presents descriptive statis-
tics of the three attitude change variables followed by exploration of within group
differences using the process variables. The final section provides a summary of the
key findings of the single-loop results. Chapter 6 then builds on these descriptive
results with a comparison between each condition and a discussion of the support
for predictions.
5.1.1 Attitude Measures
Throughout this chapter and chapter 6 three attitude measures are of interest. These
all relate to management of the A&E queuing system. Table 5.1 details the meaning
and interpretation of these variables. The first two of these variables M axU til
and T radeU til relate to a participants attitude towards an aspect of managing
resource utilisation within the A&E department over the next six months. The
third attitude, ElimV ar, relates to a participants attitude towards reducing the
variation in radiology resource availability over the next six months.
5.1.2 Supporting Measures
In addition to the three attitude variables that measure learning outcomes, seven
process variables are analysed for each condition. No specific hypotheses are tested
within this chapter. Instead the results are used as a possible source of explanation
for different outcomes within a condition. Specifically the seven variables are used to
explore differences between correct and incorrect directions of attitude change within
a condition. For example, MB and MR participants experienced strong correct and
incorrect attitude change on T radeU til. The seven process variables are explored
Table 5.1: Attitude Measures
Attitude Description
M axU til The change in attitude towards pushing A&E resource utilisation to its maxi- mum. A negative change represents beneficial attitude change resulting from the simulation, as very high utilisation is detrimental to system performance;
T radeU til The change in attitude towards trading off some resource utilisation to achieve higher system performance. A positive change represents beneficial attitude change resulting from the simulation, as this indicates that participants recog- nise that utilisation has a relationship to system time;
ElimV ar The change in attitude towards reducing the variation in the availability of radi- ology resources. A positive change represents beneficial attitude change resulting from the simulation, as lower variation in the availability of radiology improves long term system time.
The process variables are divided into two groups. The first of these are credi-
bility measures.
• Median credibility assessment score;
• Median self confidence in the credibility assessment.
The second group of process variables give information on how participants
searched the solution space.
• Percentage of scenarios including resource reallocation;
• Percentage of scenarios including extra resource;
• Percentage of scenarios including reduced variability in the radiology depart- ment;
• Percentage of scenarios including other variables;
5.2
Analysis Considerations
Before proceeding to the results there are three issues that must be considered with
the analysis. Firstly, a sensible approach to identify and exclude outliers is required.
Given the small sample size of the experiment and the number of variables measured,
it was decided to adopt a multivariate approach. Secondly, a particular statistical
issue arises when working with pre-test post-test designs called regression to the
mean. This phenomenon is described along with reasons why the subgroup analysis
of conducted in this research warrants a specialised analysis of the data. Lastly, there
are several approaches available for dealing with regression to the mean. These are
briefly described along with the relative advantages and disadvantages.
5.2.1 Outlier Analysis
One problem with an experiment where multiple variables are measured is that there
are more chances that univariate outliers (outliers on single variables) will occur.
Given the small sample size of the experiment, it was deemed appropriate that cases
would be excluded only if they consisted of a unique combination across variables
(i.e. not extreme on an individual variables, but have unique multivariate profiles)
(Hair et al., 2006).
The outlier analysis was run in an iterative manner, i.e. as outliers can mask
other outliers (Wilcox, 2005) the analysis was repeated after outliers had been re-
moved to verify no further outliers were present. The outlier analysis consisted of
three stages. Firstly, univariate checks for outliers across all variables using box-
plots, histograms and z-scores (scores standardised to the normal distribution so
that extreme cases are more obvious). Secondly, bivariate checks were made using
scatter plots. Finally a multivariate profile check was run using Mahalanobis depth
(Hair et al., 2006; Wilcox, 2005).
were quite different from others cases in the experiment. For example, participant
MB4 appeared to provide a highly extreme reverse of the expected learning and gave
an exceptionally low credibility assessment score. This meant that initially MB4
believed (in fact, his or her score was the highest) that pushing for 100% resource
utilisation would reduce performance and that there is always a trade-off between
resource utilisation and performance. By the end of the experiment the participant
reversed these views (the change was extremely large). Reasons for this odd case
were investigated by reviewing the tape recording of the experiment. However, there
was no indication why the participant may have reported these values. In fact, the
participant appeared to cope quite well with the experiment and in fact learn the
opposite of what they reported. Similar results were found for the remaining two
cases identified as outliers.