Experimental set-up within a simulation model

When the model is validated, experiments can be run. However, the model is not ready to directly run experiments yet. For this, the experimental set-up must be determined. But first, it will be explained why this is necessary.

Simulation output

The output of a simulation model is stochastic. This simulation output can be transient or steady-state. With transient output, the distribution of the output is constantly changing. With steady-state output, the output is changing according to some fixed distribution. Steady-state output

is ‘moving around’ a certain value (Law, 2015, pp. 491-492). Next to these two familiar output types, other types have also been identified, like steady-state cycles and shifting steady-states. I will not describe these, but they are a variation of the main types (Robinson, 2014, pp. 368-371). Next, a simulation can be terminating or non-terminating. A terminating simulation stops at a natural endpoint. For example, a supermarket closes at the end of the day. A non-terminating simulation does not have such a natural endpoint (Law, 2015, pp. 491-493).

Usually, it takes a while before the model is in a steady state. For example, when a simulation is started there is no WIP in a production hall, whereas, in reality, WIP is present. This is the so-called initialization bias. In order to obtain accurate simulation results, multiple actions should be taken. Usually, simulations try to obtain an accurate estimate of mean performance. To acquire this, the initialization bias should be ignored, by determining the length until the model is in steady-state and removing data obtained in this period (warm-up period) or by setting initial conditions.

In addition to removing the initialization bias, sufficient data should be acquired such that a proper estimate can be calculated. Sufficient data can be acquired by multiple replications or a single long run. With multiple replications, the model is run multiple times with different random number streams, such that the sequence of random events also changes (Robinson, 2014, pp. 173-174). For both methods, the run length should be determined. By using replications, the replication run-length should be determined. Likewise, for a single long run, the length of this long run should be

determined. For a terminating simulation, the run-length does not have to be determined since this is determined by a natural endpoint.

Dealing with the initialization bias

As mentioned, the initialization bias can be removed with a warm-up period or by setting initial conditions (or a combination of the two). A warm-up period can be determined with multiple methods. Hoad et al. (2010) identified 44 different methods and classified them under five headings

- Graphical methods: involving visual inspection of time-series together with subjective judgment.

- Heuristic approaches: simple rules for determining the period. - Statistical methods: statistical principles for determining the period.

- Initialization bias test: iterative ways together with other methods to determine the period. - Hybrid methods: a combination of methods.

Not all approaches are equally good. The marginal standard error rule (MSER) performed consistently well. In addition, it does not rely on assumptions, parameters or complex calculations (Hoad,

Robinson, & Davies, 2010). The aim of the MSER is to minimize the width of the confidence interval of the mean by deleting initial observations. The MSER value can be calculated as follows:

42 𝑀𝑆𝐸𝑅(𝑑) = 1 (𝑚 − 𝑑)2 ∑ (𝑌𝑖− 𝑌̅(𝑚, 𝑑)) 2 𝑚 𝑖=𝑑+1

Where d is the proposed warm-up period, m the number of observations, and 𝑌̅(𝑚, 𝑑) the mean of the observations 𝑌𝑑+1 to 𝑌𝑚. The MSER value is calculated for every value of d up until m-5, so the last five values are not used. The value for d that minimizes the MSER value, is the warm-up period. If this value is higher than half of the used days, the conclusion for the warm-up period is rejected. If a model has more than one output (KPI), the warm-up length should be determined for every output. The maximum warm-up length for each KPI should be chosen as warm-up length. Moreover, the warm-up period should also be chosen for every input (experiments). In practice, this can become quite burdensome. Therefore, the warm-up length should be overestimated a little bit (Robinson, 2014, p. 179).

As an alternative to a warm-up period, initial conditions can be set by identifying values in the current real system or by determining the values of the system after the warm-up period. Initial conditions can be useful if the runtime of the simulation model is quite long.

The run length and number of replications

After a method for removing the initialization bias is found, the run-length and number of replications should be determined. The aim of both is to obtain sufficient output data from the simulation model. The number of replications will first be discussed.

For determining the number of replications, Robinson (2014) outlined three methods:

1. Rule of thumb

“At least three to five replications are performed” (Law & McComas, 1991). This rule shows that multiple replications should be run, however, it does not consider differences between models.

2. Graphical method

With the graphical method, an X number of replications should be run. The number of replications should be plotted against a cumulative mean, for example, the average internal throughput time. The point where the graph becomes flat should be chosen as the number of replications.

3. Confidence interval method

With the confidence interval method, the width of the confidence interval, relative to its average, should be sufficiently small. For this, a relative error should be determined which is usually 0,05. Eventually, the minimum number of replications for which the estimated relative error is smaller than the relative error (0,05) should be chosen. In formula form:

𝑛∗_{= 𝑚𝑖𝑛 {𝑖 ≥ 𝑛:} 𝑡𝑖−1,1−𝛼/2√𝑆𝑛

2_/𝑖

|𝑋̅𝑛|

≤ 𝑑}

Where n = the number of replications, t the value from the students t-distribution, S the standard deviation from the replications, X the average output data from the replications, i the replication number and d the chosen allowed relative error.

The run length of the simulation should be much longer then the warm-up period. According to our

lecturer DR. IR. M.R.K. Mes: “Say at least 10 times longer (2018)”. For determining the run-length, also a graphical approach can be used. This is similar to the graphical approach for the number of

replications. In this approach, an X number of replications should be run (let’s say 3-5) within the

Equation 1

43 simulation model. The run-length should be sufficiently long. Eventually, the point where the graphs of the replications become flat should be chosen as run-length. This is like Robinson’s method for the

run-length (2014, p. 191). With the type of simulation determined, initialization bias removed, the number or replications chosen, and the run-length set, experiments can be run within the simulation model.