SIPP Testing and Model Validation

A number of approximations go into calculating the service level in this stochastic optimization model. Since the TSF level is the key driver of the staffing level, it is reasonable to question the accuracy of these approximations and to consider the SIPP adjustments discussed above. In this section I perform a numerical experiment to test the accuracy of each version of the SIPP model.

45 _{Throughout this dissertation, unless stated otherwise I use a worst case TSF of 50% and a minimum} staffing level of 2.

114 I test each SIPP version against models based on 3 of the model projects46 described in Section 2- 7.

4.3.6.2 Experimental Approach

As outlined above, the basic process involved with solving the stochastic program is to solve the model (4.1) - (4.7) against a set of scenarios, simulated realizations of call volume. During this process I calculate a TSF level and objective value, both of which are biased estimates of the true values. I then perform a post optimization analysis which tests the candidate solution against a set of evaluation scenarios. In this process I calculate an expected outcome (service level and cost) that is an unbiased estimate of the true solution47_{. The objective of this validation is to}

determine how well the unbiased, post optimization analysis is at predicting the actual realized service level. Note that since arrivals are random, the realized service level will be random.

In order to make this assessment I turn to Discrete Event Simulation (DES). DES is a well established methodology for examining complex queuing systems like this one. Using DES we can more closely model the specific behavior of the system to specific realized call patterns. The DES model used in this analysis uses the same algorithm shown in Figure 4-1 to generate a nonstationary call pattern. The model then generates individual simulated calls which are processed using the same theoretical distributions used in the Erlang A model. The simulation approach allows us to run the model for a large number of simulated arrival patterns and to calculated statistical bounds on key performance metrics such as TSF. See (Banks 2005) or (Law 2007) for a detailed description of the simulation process. Assuming that the DES model is a valid representation of the non-stationary Erlang-A queuing model, we can use this model to assess the accuracy of the TSF calculation in the optimization program.

The validation processed is outlined below:

1. Generate a set of 100 scenarios and use these to solve the stochastic optimization problem (4.1) - (4.7). 2. Using the solution found in step 1 as the candidate

solution, perform a post optimization evaluation

46 _{The fourth project outlined in section 2.7 is too small to be off interest in the scheduling model. Given} its very low volumes the project is almost always staffed at the minimal staffing level of two agents. I examine this project in the final model of this thesis which addresses project pooling.

115 against 500 independently generated scenarios to find the expected service level associated with the candidate solution.

3. Use the period by period staffing plan developed in step 1 to create the resource profile in a discrete event simulation model with an identical statistical distribution of call volumes.

4. Perform 50 replications of the DES model to calculate a point estimate of the expected TSF from implementing the solution found in step 1 using SIPP, SIPP Max, and SIPP Mix.

5. Compare the results found in step 2 to those found in step 4 to assess the error associated with each SIPP approach.

Figure 4-10 TSF Validation Approach

In the following table I summarize the results from applying this approach to compare the three SIPP models to Project J.

Optimization Std Max Mix

Scheduled Hours 1,160 1,200 1,200

Expected TSF 83.2% 81.0% 83.2%

Std. Dev of TSF 2.6% 3.0% 2.6%

Simulation Std Max Mix

Expected TSF 81.50% 84.00% 84.29%

Std. Dev of TSF 2.70% 2.87% 2.46%

Bias (Opt-DES) -1.72% 3.00% 1.07%

Error in Std Dev of Sim TSF -0.64 1.05 0.43

SIPP Method

Table 4-1 TSF Validation – Project J

As predicted by theory, the standard SIPP model overestimates the expected service level and under staffs the call center. However, at least in this case the error is rather low. The TSF estimated in the optimization program is only 1.72% above what is estimated by the DES model. Furthermore in the DES model the standard deviation of the TSF measure is 2.70%, so the estimate is within .64 standard deviations. Both the SIPP Max and SIPP Mix models use a more conservative estimate of the service level attained and as a result calculate a higher staffing level. The SIPP Max is the most conservative and underestimates the service level by over 3%. SIPP Mix is less conservative and underestimates the service level by 1.07%. SIPP Mix is arguably a better fit in this case as the error is slightly smaller and in a conservative direction. If I apply the same analysis to projects S and O we obtain the following results.

116

Optimization Std Max Mix Std Max Mix

Scheduled Hours 1,080 1,160 1,120 2,760 2,880 2,880 Expected TSF 81.5% 82.8% 81.9% 78.2% 78.3% 76.2% Std. Dev of TSF 2.9% 2.8% 3.0% 9.7% 10.4% 11.0% Simulation Expected TSF 80.99% 84.70% 84.30% 79.33% 81.80% 83.28% Std. Dev of TSF 3.22% 3.02% 3.40% 4.74% 4.80% 3.64% Error (Opt-DES) -0.54% 1.91% 2.45% 1.15% 3.50% 7.08%

Error in Std Dev of Sim TSF -0.17 0.63 0.72 0.24 0.73 1.95

SIPP Method

Project O Project S

SIPP Method

Table 4-2 TSF Validation – Projects O and S

Results from these two projects again show that the SIPP Standard method is the least conservative, but in the case of project O it overestimates the service level. The Standard SIPP model is in general the most accurate and I will utilize this approach in the remainder of this analysis.