Validation Method

Chance Effect and Multiple Replication Simulation Runs

Chance effect or randomness is one of the key features of HAIs (see Section 1.2.2). Given the same initial conditions, the transmission dynamics may be very different simply due to the intrinsic stochasticity of the spread of infectious diseases within a small patient population. Figure 5.3 demonstrates two different transmission dynamics realised by two simulation runs with exactly the same initial conditions and input parameter values. Due to chance effects, conclusions drawn from simulation models, either for model validation or experimentation, must be based on multiple replications so that the mean and the variations of model responses can be estimated. Therefore, it was decided that 500 replications of simulation runs should be performed for each distinctive set of input parameter values during model validation and experimentation. Furthermore, in order to get rid of the initial bias, a warm-up period of 50 days is added for each simulation run and consequently only simulation data after 50 days are collected for analysis. The time needed to run a single replication of the model for a year was about 1 second on a personal computer with a 2.2GHz Intel processor.

Figure 5.3 Realisation of two transmission dynamics by two simulation runs with the same initial condition and input parameter values (blue line: number of total patients in the ward; brown line: number of susceptible patients in the ward; green line: number of colonised patients in the ward)

Model Configuration and Validation

To demonstrate that the hospital ward has settled into a steady state after 50 days, Figure 5.4 shows the time-series of the mean number of patients in the hospital ward from 30 replications. The model is configured with model inputs of ward A during the pre-crossover period (see Table 5.10) assuming a transmission coefficient of 0.1. Figure 5.4 indicates that the mean number patients in the hospital ward increases from zero (the hospital ward is empty at the beginning of the simulation) to about 31 during the first 50 days; afterwards the mean number of patients levels off and varies in a small range between 29 and 33 patients.

Figure 5.4 Time-series of the mean number of patients in the hospital ward from 30 replications

To demonstrate that 500 replications are enough to obtain accurate model responses, the confidence interval method is applied (Robinson 2004). Figure 5.5 shows the cumulative mean and the confidence interval of the transmission ratio from 1000 replications of the model configured with inputs of ward A during the pre-crossover period assuming a transmission coefficient of 0.1. A significant level of 5% is used to construct the confidence interval which ensures a 95% probability that the value of the ‘true’ mean transmission ratio lies within the calculated confidence interval. For each replication, the simulation is run for 415 days with 50 days warm-up period and 365 days for data collection. Figure 5.5 demonstrates that the confidence interval is sufficiently narrow and the cumulative mean line (the thick line in the middle)

Model Configuration and Validation

model response, which is defined as half the width of the confidence Interval expressed as a percentage of the cumulative mean (Robinson 2004), is about 1.9% after 500 replications.

Figure 5.5 Cumulative mean and confidence interval of the transmission ratio from 1000 replications

The Calibration-Validation Process

The procedure to validate the model follows the calibration-validation process, in which all fourteen scenarios are randomly split up into two groups, one for the parameter calibration process and the other for the model validation process.

The only parameter to be calibrated is the transmission coefficient which is defined as the number of secondary cases incurred by one primary case per day assuming a large population of susceptible patients in the system. The transmission coefficient is the key parameter to define the transmission dynamics of infectious diseases. However, it is also difficult to estimate the parameter directly from observation (Cooper and Lipsitch 2004). As a result, the transmission coefficient has been the subject of many previous modelling studies in which mathematical compartmental models were fitted to observed data to calibrate the coefficient (Grundmann et al. 2002; Pelupessy et al. 2002; Cooper and Lipsitch 2004; Forrester et al. 2005).

Model Configuration and Validation

During the following model validation stage, the transmission coefficient calibrated in the calibration stage will be applied to the rest of the scenarios, and comparisons between observations and model responses will be performed to determine how close the model can represent the real system.

Parameter Calibration

During parameter calibration, the best estimate of the transmission coefficient for each scenario in the calibration group is estimated. All these best estimates of the transmission coefficient will then be used calculate the overall transmission coefficient for the following validation stage.

For each calibration scenario, the transmission coefficient is tested on a range of values to find the best-fit coefficient that leads to the closest match between the observed transmission ratio and the mean transmission ratio predicted by the model through 500 replications. During parameter calibration, all other input parameters, except the transmission coefficient, will take the values given in Table 5.10.

In order to find the best estimate of the transmission coefficient for each scenario, the coefficient is initially set at 0.1 which is an arbitrary value based on the estimations from previous studies (Hotchkiss et al. 2005; McBryde et al. 2007). Starting with the initial values, the model is run for 500 replications and the mean predicted transmission ratio is compared with the observed transmission ratio (see Table 5.11). If the former is larger than the latter, which indicates that the coefficient in the current model is too high, the coefficient is decreased by one small unit and the comparison will be repeated. Otherwise, if the mean transmission ratio is smaller than what is observed, which indicates the coefficient in the current model is too low, the comparison will be repeated with the coefficient being increased by one small unit. The small unit chosen in the study is 0.001. This iterative process continues until two consecutive comparisons give two distinctive results with one showing mean model response is slightly higher than the observation and the other showing the opposite. The search will stop at this point and the coefficient which gives the smaller absolute difference between the mean model response and the observation is chosen to be the

Model Configuration and Validation

Once the estimates of the transmission coefficient from all scenarios in the calibration group are obtained, a weighted mean coefficient is calculated to be the overall transmission coefficient. In this study, the number of secondary cases in each scenario is selected as the weighting factor since it is a reasonable indicator of the magnitude of MRSA transmission in each scenario and the data are available from observation.

Model Validation

During the model validation stage, the weighted mean transmission coefficient determined in the calibration stage will be used to test the fit of the model to the observed data for all scenarios in the validation group. For each scenario, the model will be run for 500 replications and the mean and distribution of various model responses (not only the transmission ratio, but also the absolute number of secondary cases and the average time to detection) will be compared with the corresponding observations.

Due to chance effects, each observed scenario is just one possible realisation which may take place for that ward during that study period with the same input parameter values. Therefore, the single observation itself does not necessarily represent the theoretical average transmission dynamics and it may even be an extreme case. As Robinson (2004) argued that real world data, even if “accurate”, are only a sample which in itself creates inaccuracy. Therefore, when a single outcome from the observation (e.g., the observed transmission ratio of one scenario) is compared with the corresponding model responses (e.g., the mean and distributions of transmission ratios predicted by 500 replications of the same scenario), they are unlikely to be an exact match.

In order to compare rationally a single observation with the corresponding model responses, it is reasonable to look at whether the single observation is within certain range of the model responses (e.g., within two standard deviations, or within first and third quartiles). The practice has been used for the validation of HAI models (Austin et al. 1999; Grundmann et al. 2002). In this study, for each validation scenario, the single observation will be compared with the first and third quartiles, and tenth and ninetieth percentiles of the model responses from 500 replications. Standard deviation is not used for the comparisons since the model responses do not follow the normal

Model Configuration and Validation

distribution (see Section 5.3.2). Paired-t statistical tests are also applied to test the difference between the observed and mean model responses.