Kriging is an increasingly popular metamodeling tool in simulation due to its flexibility in global fitting and prediction. In the fitting of this metamodel, the parameters are likely to be estimated from the simulation data and traditional plug-in estimators for prediction typically ignore these parameter estimation un-certainties. These can be substantial in stochastic simulations and can lead to overconfident results and misguided decisions. In this chapter, we propose a Bayesian kriging metamodeling approach for stochastic simulation that can ac-count for all parameter uncertainties as well as the inherent uncertainty of the simulation model itself, providing a more accurate approach for prediction and analyses. We derive the predictive distribution under certain distribution assump-tions and proposed a MCMC approach to address the general case. The results in Section 6.2 and Section 6.3 suggest that this proposed Bayesian approach can better account for the noise and heterogeneity in stochastic simulations. In ad-dition, we further consider the important problem of planning the experimental
design. We propose a two stage design approach that systematically balances the allocation of computing resources to new design points and replication numbers in order to reduce the uncertainties and improve the accuracy of the predictions.
We illustrate the design approach with several examples and provide some gen-eral suggestions on design plans under various scenarios. The current work can be extended in several directions. First, as observed in Section 6.3, selection of the prior mean functional form is important, especially in the presence of large noise. One possibility is to include model selection techniques within the model-ing framework to guard against model misspecification. Another possibility is to include this model uncertainty into the framework. Second, the proposed design criterion focuses on improving the overall predictive ability of the metamodel.
As metamodels have also been widely used for simulation optimization, our pro-posed metamodeling approach can be integrated into optimization techniques to develop sequential approaches for global optimization. For example, the predictor and its distribution form as described in Equation (6.5) can be substituted into optimization search point criterions like the expected improvement (EI) function (Jones et al.(1998)) or augmented forms of the EI function (Huang et al.(2006)) and applied in optimization algorithms like Efficient Global Optimization (EGO) for global optimization.
CONCLUSION
This study contributes to the design and analysis of computer experiment for stochastic systems. More specifically, we investigate both the modeling and ex-perimental design in the stochastic simulation context. In this chapter, we are going to conclude the study by summarizing the findings and limitation of this research, and also the possible future research topics.
7.1 Main findings
In this thesis, we first investigate the kriging model’s application in stochastic simulation, especially in heteroscedastic situations in Chapter 3. We discuss the kriging model’s behavior given the stochastic simulation outputs, and propose a modified nugget-effect model by relaxing the stationarity assumptions of the nugget-effect model and modeling the random noise as an independent random process. We then look into the behavior of the likelihood function in the pa-rameter estimation for the traditional OK model given the stochastic inputs, and note that the nugget-effect model and the modified nugget-effect model can re-duce the erratic behavior in the parameter estimation by penalizing the likelihood function affected by noisy input. This modified nugget-effect model is also com-pared with the stationary kriging model in terms of the mean squared error. We propose three methods to easily obtain the prediction variance of our new pre-dictor; namely, the kriging modeling, non-parametric bootstrapping method and the interpolation method. We illustrate our model and approach with several
numerical examples. Our results indicate that the proposed modified nugget-effect model is a promising method for applications in simulation systems with heterogeneous variances. In the following Chapter 4, we investigate the impact of parameter estimation uncertainty on three different kriging model forms in stochastic simulations. We analyzed theoretically a simple two-point problem and conducted three numerical studies. Based on the results of these studies, we find that the sensitivity parameter φZ estimated by kriging model is affected by the random noise in the stochastic system, and the additional prediction er-ror caused by parameter estimation uncertainty increases as the variance of the random noise ε (x) increases. The proportion of the additional prediction error caused by parameter estimation uncertainty in overall prediction error increases when the variance of the random noise ε (x) increases. In the case when the vari-ability of the noise is low and sufficient sample data is available, this additional error becomes negligible. Among the three kriging model forms studied in this chapter, the modified nugget effect model seems to have the best performance in both the overall prediction error and additional prediction error caused by pa-rameter estimation uncertainty. This phenomenon is partially explained in the two-point problem adopted in this chapter.
We further apply the modified nugget effect kriging model to the stochastic simulation optimization problem in the Chapter 5. With the modified nugget effect kriging model, we adopt the efficient global optimization framework and expected improvement function to allocate the available computing budget for the experimental design. Hence the traditional design methods can not balance the different design options in the scenarios with heterogeneous variance, we propose the two-stage sequential design framework to solve the problem. The proposed design framework tends to reduce the random variability in the first stage to certain level and then search for the new design point in the second stage. We test the proposed design framework with a simple one dimensional example then followed by a more realistic shipping liner service planning simulation example.
The results suggest that the proposed two-stage sequential design framework is a more reasonable design method to be used for the stochastic simulation than the traditional method.
After the development of the kriging model with modified nugget effect and related experimental design methods, we realize that the parameter estimation uncertainty problem mentioned in Chapter 4 still needs appropriate solution. In Chapter 6, we propose a Bayesian kriging metamodeling approach for stochastic simulation that can account for all parameter uncertainties as well as the inherent uncertainty of the simulation model itself, providing a more accurate approach for prediction and analysis. We derive the predictive distribution under certain distribution assumptions and proposed a MCMC approach to address the general case. The results in Section 6.3 suggest that this proposed Bayesian approach can better account for the noise and heterogeneity in stochastic simulations. In addition, we further consider the important problem of planning the experimental design. We propose a two stage design approach that systematically balances the allocation of computing resources to new design points and replication numbers in order to reduce the uncertainties and improve the accuracy of the predictions.
We illustrate the design approach with several examples and provide some general suggestions on design plans under various scenarios.