Model verification and validation

The results of the OPD simulation model provide a great deal of information about the causes

and effects of congestion in the queueing system. The purpose of the model is to provide

insights that can be applied to the real-world problems, and it is therefore necessary to consider the accuracy and limitations of the simulation model and its results.

Verification

Verification tests are an important step in evaluating simulation models, although they are not directly concerned with the model’s ability to produce realistic results. Rather, the purpose of verification tests is to ensure that (a) the high-level/conceptual model of the system has been implemented correctly in the simulation; and (b) the technical aspects of the simulation function properly (Banks et al., 2004).

The OPD simulation model was verified through a close examination of the data generated in multiple simulation runs. This data was used to check various aspects of the model’s implemen- tation and logic, as outlined below.

1. The distributions of simulated patient counts, arrivals, and treatment needs/times were checked to ensure that they conformed to the input parameters.

2. Individual patient trajectories were examined to verify that (a) patients followed the correct path through the system; and (b) event times and waiting times were accurately recorded in the simulation results.

3. The simulated queues at each process were checked to confirm that (a) the appropriate queueing disciplines were followed; (b) queue lengths were accurately recorded in the simulation results; and (c) the composition of each queue matched the waiting times recorded for that queue.

4. The number of busy/available staff at each process was monitored to ensure that (a) the staff levels in the simulations matched the input data for staff schedules; (b) the staff break events were correctly implemented; and (c) the amount of time that staff were busy matched the treatment times for patients at that process.

Many of these checks were performed repeatedly during the process of implementing and debug- ging the simulation algorithm. They were also used to verify the final simulation model and the results presented in the previous sections. This analysis confirmed that the simulation model is an accurate implementation of the conceptual model outlined in Chapter 2.

Validation

The purpose of model validation is to evaluate both the accuracy and the usefulness of a model’s results. Accuracy is generally determined by how well the model matches the real-world be- haviour of the system, while usefulness is related to the model’s ability to provide the informa- tion/insight required for the task at hand. Robinson (1999) suggests that model validity does not depend on any absolute standards, but rather whether the model is “sufficiently accurate” to fulfil its intended purpose.

The accuracy of the OPD simulation model could not be tested through a direct comparison with real-world data, since the OPD does not monitor or record the flow of patients through the different queues. However, several other validation methods were used, as explained below.

1. Face validation

The simulation model was evaluated by the OPD clinical manager, Dr Ben Gaunt, as well as other members of the OPD staff. Based on their day-to-day experiences, staff were able to indicate whether the simulation results matched the observed trends in the OPD, such as (a) the normal/expected lengths of queues; (b) the peak queue lengths/busiest times at each process; (c) the flow of patients between different queues; and (d) the arrival/departure times of patients.

Discrepancies between the simulation results and the observed trends were discussed with the OPD staff to identify elements of the model that could be improved. Feed- back from these discussions resulted in numerous additions to the OPD model, such as time-dependent staff schedules, detailed treatment parameters, and priority queueing dis- ciplines. The OPD model was evaluated, improved, and re-evaluated many times during the course of this research.

Contact with the OPD staff also played an essential role in defining the purpose and scope of the model. These interactions helped to determine which aspects of the OPD system should be included in the model, as well as how to present the simulation results in a useful, informative manner.

2. Input-output validation

An important aspect of the simulation model is its ability to illustrate the effect of changes to the OPD system. This was tested during discussions with hospital staff by varying the model’s input parameters and confirming that the corresponding simulation results reflected the appropriate response. For example, increasing the number of staff at a process should result in shorter queues and waiting times at that process, while increasing the number of patients in the system should increase the general level of congestion.

3. Previous research

Bertscher (2015) highlights several issues affecting patient flow in the OPD system, based on research conducted during December 2014. Although this study did not involve rigorous data collection or analysis, it does confirm some of the general trends in the simulation results for the 2015 OPD set-up. For example, key findings in Bertscher’s report emphasise that

(a) the doctors queue is a bottleneck that limits the flow of patients through the OPD system;

(b) the long delays that patients experience in this queue are the main source of inefficiency in the OPD;

(c) fluctuations in patient arrivals have a significant effect on congestion — the OPD functions well when there are relatively few patients, but the system is not equipped to handle large numbers of patients efficiently.

Bertscher (2015) also mentions other sources of inefficiency in the OPD which are not included in the simulation model. Examples of these problems include

(d ) routing confusion — some patients may not follow the correct route through the OPD because they do not understand the system or receive incorrect instructions from staff;

(e) missing equipment/supplies — staff (particularly doctors) spend too much time searching for medical equipment and supplies that have been removed from their consultation rooms.

These observations illustrate some of the limitations of the simulation model, which does not reflect the time that staff spend on unforeseen tasks. There are several other sources of inefficiency that the model does not account for, such as equipment failures, missing patient records, and staff members arriving late for their shift or missing work.

These limitations should be taken into account when analysing the simulation model’s results, which reflect the flow of patients through the OPD in the absence of these types of delays and interruptions. In reality, patient’s waiting times may be longer and more varied than the waiting times in the simulation results, and there may also be greater variance in the length of the OPD queues. The simulation model’s results should therefore not be interpreted as a direct measure of the actual patient waiting times in the OPD.

Based on the information gathered from these sources, the simulation model is sufficiently accu- rate to fulfil its intended purpose. It can be used to develop a better understanding of the OPD system, including the behaviour of the OPD queues and the experiences of different patient profiles. The model can also provide insight into the causes and effects of congestion and the impact of changes to the system.