B efo re experim entation can begin, the m odel must be "validated” . Carson (81)
distinguishes between verification, validation and credibility:
( i ) V erificatio n is the responsibility o f the m od eller to ensure that the model
performs in the w ay in which the m odeller intended it to perform
( i i ) V alidation is the responsibility o f the user (dom ain expert) and the m odeller to
ensure that the m odel is a representation o f the dom ain with sufficient accuracy to
be able to test different conditions
( i i i ) A "cred ib le" m odel is accepted by the dom ain expert as being valid, and can
be used as an aid in decision-making.
It appears that Carson's definition o f cred ib ility involves a higher level o f
con fidence and user acceptance than is generated by validation. L a w and Kelton
(8 2 ) sim ilarly o ffe r three stages o f examination, starting with verification and
validation as above, but having a third stage called "output analysis". Output
analysis is concerned with determining a sim ulation m odel’s true parameters or
characteristics, which may not necessarily be the characteristics o f the system.
Output analysis should therefore be continued until acceptable confidence limits
can be obtained fo r the results such that they are "credible".
In this w o rk , the author is the m odeller and the dom ain expert. Since the data is
fictitious, it is d ifficu lt to rigorously p ro ve valid ity and credibility. H ow ever,
statistical analysis o f the output yields a g o o d insight into the most important
factors in the decision-m aking process fo r alternative routes and identifies those
factors to w h ich the results are most sensitive.
7.1 Verification
Verificatio n gen erally refers to the process o f testing and checking the computer
code to "v e rify " that it truly represents the assumptions and data accurately (81,
82). Standard d ebu ggin g techniques are em ployed, and Carson also suggests:
(a ) Structured programming techniques
(b ) Program testing under a w ide variety o f input parameters, to check for
extrem e reactions and special cases
(c ) C ollectio n and display o f statistics, o v e r and above those o f interest
fo r experim ent, in order to expose any possible sources o f error
A ll the above techniques were used fo r this program. T h e fo llo w in g points from
L a w and K elton enhance Carson's list:
(d ) Som eone other than the programmer should read the code, because
the person w h o writes a particular subprogram m ay get into a mental
rut and be unable to evaluate its correctness; this procedure is
sometimes called a "structured walkthrough".
(e ) Use a "trace", where the state o f the simulation system, e.g. even t list,
state variables, certain statistical counters etc., is printed out after each
event. This can also be used to check the m odel's reaction to extreme
conditions.
( f ) Run the m odel under sim plifying assumptions i f possible, for which
the m odel's true characteristics are known or can be calculated.
In addition to the techniques listed above, the author adds the fo llo w in g two
techniques fro m experience and exploitation o f visual interactive system
capabilities respectively:
( g ) A fe w lon g run times are useful to check for unusual cases, rare bugs,
instability, or an unusual or catastrophic combination o f conditions.
(h ) Careful observations o f key events on the screen, "step p in g" through
each even t in turn, and checking the attributes and m ovem ents o f key
entities (an animated equivalent o f the trace, but not a complete
replacement).
7.2 Verification o f this model
Although the origin al model had been debugged, verified and validated for its
original application, exten sive verification was still required on the experimental
m odel because o f the m ajor revisions made to most o f the subroutines.
Functional testing was therefore required to ensure correct operation o f the main
capabilities described in Chapter 6, namely:
m ovem ent o f jobs between machines
processing a jo b on a machine
handling breakdowns and repair times
choosing the next jo b for an idle machine
choosing the next operation for a job
Functional testing was carried out on a dummy data set, using FCFS despatching
rules, and without alternative routes, and used most o f the techniques listed in the
previous section.
T h e third stage tested the action o f the code handling despatching rules and
alternative routes.
7.3 Components o f the validation process
Validation is often p o o rly treated in the literature on simulation m odelling, and
where consideration has been g iven to validation it often relies ex ten sively on
historical data o f existin g systems o r data collection on the real system (e.g. 82,
83). Much o f the w o rk carried out by the Simulation Group within the
Manufacturing Systems Engineering Group at W a rw ick University has been
in vo lved with the design o f production facilities not yet in existence and which
are expected to be considerably different from their predecessor facilities.
M od ellin g there proceeds using raw design data fo r input but output must be
verified as "sensible" o r "not sensible" by a team o f experienced engineers and
then "acceptable" o r " o f concern" o r "unacceptable" subsequently.
Sensitivity analysis can be used to identify factors on which the results are very
dependent and to check effects o f some o f the least reliable input data, such as
expected breakdown levels or forecast product mix.
Since simulation depends on random number generation to activate events such as
arrivals and breakdowns, any effects due to the random number streams being
used must be found, before any effe c t due to a change in conditions can be
quantified. The effects o f random number streams w ill be considered in a later
section (7.4) on "output analysis".
O ne advantage that a m odeller has o v e r other "experimental" conditions is much
better "control" o f the system, i.e. variables, factors and conditions can be
changed easily by changing the data, changing the progam code o r using other
random number streams (84).
F or the purposes o f validation, the ch o ice o f probability functions to generate
arrivals, time between breakdowns and repair times should be b riefly justified.
T im e between arrivals o f each com ponent type is sampled from a negative
exponential distribution, which produces entirely "random" arrivals. It is
questionable whether arrivals really are random in practice since they are lik ely to
have departed from a preceding process o r have arrived as a result o f an order on a
supplier. Both o f these cases might indicate that a normal distribution w ou ld have
been more appropriate.
T h e usual alternative is to assume that there is an unlimited supply o f material.
Parts are then drawn through the system as required, which creates another set o f
problem s such as w ork in progress and inflated loading o f a system which was not
intended to run at full capacity.
It is therefore com mon to use the negative exponential distribution fo r arrivals and
then prioritise or route the arrivals according to some schedule or final demand
pattern. The pattern o f time between arrivals in this m odel is shown by the graph
in Figure 21 fo r components 1 to 4. The typical exponential curve is achieved with
many short inter-airival times but also a fe w lo n ger times.
Sim ilarly, the time between failures is sam pled from a negative exponential
distribution in order to generate "random" breakdow n times.
Finally, it is common practice to generate repair times from an Erlang distribution
which is a variant o f the Gamma distribution. O f all the theoretical distributions
com m only applied to simulation this application appears to be the most obvious.
Depending on the type o f equipment, it is lik ely that the repair time pattern would
be made up o f a lot o f short breakdowns with a fe w long ones, or many lon g and
com plicated repairs with some short ones. Th ese cases can be generated by
specifying different " N " values. Graphs o f the repair times produced using "shape
factors" o f N = 1, 2 and 3 are shown in Figure 22, generated from the data in table
2. This is important to scheduling rules, and particularly to alternative routes since
different patterns could dictate different rule emphasis. W ith increasing data
collection on machine history, it is not d iffic u lt in many companies to actually
identify the repair pattern. In one com pany (7 9 ), analysis o f breakdown data over
the previous 5 years, showed similar patterns. H o w ever, an arbitrary decision was
made to adopt Erlang with a value o f N = 2.