P. A. Farrington, H. B. Nembhard, D. T. Sturrock, and G. W. Evans, eds.
ADVANCED INPUT MODELING FOR SIMULATION EXPERIMENTATION
Bruce Schmeiser School of Industrial Engineering
Purdue University
West Lafayette, IN 47907–1287, U.S.A.
ABSTRACT
We discuss ideas useful to simulation practitioners when specifying the probability models used to represent stochas-tic behavior. Emphasis is on situations in which the classical simple models are inadequate. After discussing some gen-eral modeling issues, we consider univariate distributions, nonnormal random vectors and time series, and nonhomo-geneous Poisson processes.
1 INTRODUCTION
We assume that the reader is familiar with Nelson and Yamnitsky (1998), last year’s excellent advanced tutorial on simulation input modeling. That tutorial is advanced in the sense that it focuses on nonclassical univariate distributions and on dependence, both statistical and through time. In addition to being advanced, their tutorial is practical in that it focuses on ideas supported by easily obtainable software. (For an introductory tutorial, see any of Leemis 1996, 1997, 1998, 1999). Simulation textbooks often discuss input modeling, but typically not the advanced topics considered in Nelson and Yamnitsky (1998).
Rather than repeating the content of their advanced tu-torial, we expand upon it via comments and opinions. Some comments are historical, some point to additional references and ideas, some concern common errors. The opinions, we hope, are obviously opinions and that the reader, at least upon reflection, agrees with them. Such agreement is some-times difficult to attain, however, as evidenced by Kelton et al. (1990). In the spirit of a tutorial, no ideas here are new. A fairly extensive list of references is provided, with the large number from recent Winter Simulation Conferences indicating that input modeling remains a topic of substantial interest.
We organize our thoughts beginning with some gen-eral issues, followed by sections on univariate distributions, random vectors and time series, and the nonhomogeneous Poisson process.
2 GENERAL ISSUES
Underlying our discussion of input modeling is our view of simulation experiments (Nelson and Schmeiser 1986, Schmeiser 1990). We assume that the purpose of an ex-periment is to estimate a performance measure θ, a (not necessarily scalar) constant with unknown value. The sim-ulation experiment can be represented by the sequence of transformations
G → U → X → Y → ˆθ,
where ˆθ is a point estimator of θ, which is a characteristic (e.g., mean or quantile) of the unknown probability model of the output dataY . The output data Y are a known function g of the input data X, whose probability model is assumed to be known. The input data X are generated as random variates (see, e.g., Devroye 1986) using theU(0, 1) random numbers U, which are generated from a (pseudo)random number generator G.
2.1 The Logical and Input Models
on the issues that arise in simulation input modeling. Spe-cialized commercial software designed for simulation input modeling includes the ARENA input modeler (Kelton et al. 1998), BESTFIT (Jankauskas and McLafferty 1996) and EXPERTFIT (Law and McComas 1998). They focus on the important, but relatively simple, task of fitting classical univariate distributions to data and incorporating the fitted distributions into simulation software. The last few years have produced some easily attainable noncommercial soft-ware and ideas for creating more-complex input models, including Avramidis and Wilson (1994), Cario (1996), Cario and Nelson (1997a, 1997b), Chen (1999), Kuhl, Damerdji and Wilson (1997, 1998), Kuhl, Wilson and Johnson (1997), Song and Hsiao (1993), Song, Hsiao and Chen (1995), Stan-field et al. (1996), and Wagner and Wilson (1995, 1996a, 1996b). These (and some other ideas) are discussed in Nelson and Yamnitsky (1998).
The attitudes of a practitioner toward the level of detail in the logical model and in the input model are often quite different. The inclination to include unneeded detail in the logical model is pervasive, despite ubiquitous warnings in discussions of simulation modeling. The inclination to simplify the input model (via use of classical distributions, statistical independence, and time homogeneity) is equally pervasive. Warnings against such simplification are easily found: Bratley, Fox, and Schrage (1987), Kelton et al. (1990), Johnson (1987), and Wilson (1997); warnings are implicit in sensitivity analyses such as Gross and Juttijudata (1997), Gross and Masi (1998), and Reilly (1998).
Why the different attitudes? Certainly a reason is that the logical model is often visible and supported by commercial software while the input model is less visible in that it is hidden by its own essence: observations are random. In addition, there is the wide-spread sense thatθ depends only weakly on characteristics of the input model beyond the population mean E(X). Even in the simple case where the performance measure is a mean (i.e.,θ = E(Y ) = E(g(x)) ), typically E(g(X)) 6= g(E(X)); that is, model performance is not obtained by substituting mean behavior. For example, a serial production line with deterministic arrival and service times might need no buffers, but buffers are needed when stochastic behavior is present.
2.2 Stochastic and Subjective Uncertainty
Two types of uncertainty are represented in the input model. Helton (1996) refers to them as stochastic (or aleatory) and subjective (or epistemic). For example, suppose that for any particular simulated the time between arrivals is modeled as a Poisson process with a constant rate λ. In addition, suppose that for each day λ is chosen from a particular Weibull distribution. The arrival times are clearly stochastic uncertainty. The randomness ofλ, however, might be either stochastic or subjective. It would be stochastic if the arrival
rate depends upon the day’s weather; it would be subjective if the arrival rate is an unknown constant and the Weibull distribution is reflecting the modeler’s lack of knowledge about the true value. Notice that the simulation computation is identical in either case; only the interpretation differs. In the latter case, the simulation results correspond to no real-world system. Sometimes more appropriate is a sensitivity analysis that checks the change in ˆθ for a change in λ.
Barton and Schruben (1993) discuss explicit methods for incorporating uncertainty about the model in the simulation experiment. Here again, the interpretation of the simulation results must be carefully stated because there is no real-world system that corresponds to the simulated model.
2.3 Goodness-of-Fit Tests
As a final general comment, we mention the inappropriate use of goodness-of-fit tests to determine whether an input model is adequate (and, more generally, whether a simulation model is valid). The problem with such tests is that they deal with statistical significance while the relevant issue is practical significance. Remember that, except for some trivial situations, we know before any data are collected and/or analyzed that the typical simple null hypothesis of a goodness-of-fit test is false. The question is not whether the input model is absolutely correct; it is whether the input model is adequate for the analysis at hand. A particular model might be quite adequate for a quick study over the weekend and quite inadequate for a six-month study, even if the real-world system is the same. Similarly, an input model might be adequate for a simulation experiment with a small number of replications but inadequate when run with many replications. The fallacy of the goodness-of-fit test is made obvious when a large real-world data set it fitted to many classical distributions and all are rejected; all are rejected because the large sample size yields large power and the error in the model is indeed statistically significant. The tyranny of the goodness-of-fit test is such that many practitioners fear using a distribution that has been rejected, even after confirming visually and conceptually that it provides an adequate fit.
The preceding paragraph is not meant to criticize the use of goodness-of-fit statistics for heuristic ranking of competing models (for example, Cheng et al. 1996). It should be remembered, however, that different statistics can provide quite different rankings. But if comparing alternative input models is the goal, then a Bayesian approach (e.g. Chick 1997) has substantial advantages.
3 UNIVARIATE DISTRIBUTIONS
input-modeling programs mentioned earlier automate this task well. But what to do when the classical distributions are not adequate? Schmeiser (1977), in an input-modeling tutorial before commercial input-modeling software, em-phasized classical four-parameter families of distributions such as the Pearson and Johnson, both elegant in that they provide one and only one distribution for any first four moments and inelegant in that they use more than one functional form. Also discussed were two relatively new four-parameter family designed explicitly for use in simula-tion experiments (Ramberg and Schmeiser 1974, Schmeiser and Deutsch 1977), elegant in that they used only one func-tional form and are relatively easy to fit, but inelegant in that the one-to-one relationship with moments is lost. These dis-tributions are all inadequate for data sets having anomalies as simple as being bimodal with tails.
Later models (Avramidis and Wilson 1994 and Wagner and Wilson 1995, 1996a, 1996b), based on polynomials in various forms, allowed distributions to be formed on the fly with an arbitrarily large number of parameters. The B´ezier models of Wagner and Wilson are particularly elegant in that they combine direct tractable visual distribution fitting with an arbitrary number of parameters. The B´ezier transformation of the polynomial to a function that can be dragged on a computer screen is fundamental to the idea; without the visualization, a direct polynomial fit is unwieldy because polynomials are not automatically monotonically increasing from zero to one. That arbitrarily difficult data sets can be modeled interactively and visually using only one functional form is a significant advance.
4 MODELING DEPENDENCY
Two fundamental forms of statistical dependency arise: Ran-dom vectors with dependent components and time series with autodependence. We comment briefly on each.
Other than the multivariate normal distribution, few random-vector models are tractable and general, though many multivariate distributions are well documented (John-son 1987). The early approach to creating flexible multivari-ate models was to assume a particular marginal distribution, as done, for example, by Schmeiser and Lal (1980, 1982) and Lewis and colleagues (e.g., Lewis and Orav 1989). Wagner and Wilson (1995) discuss a generalization having B´ezier marginal distributions and Stanfield et al. (1996) discuss a generalization having Johnson marginals.
A more-general idea is to transform the multivariate normal distribution to a multivariate uniform distribution. The components of the multivariate uniform distribution then are placed in the inverse transformations of the desired marginal distributions to obtain pseudorandom vectors. The resulting model has the desired marginal distributions and desired rank correlation, as discussed, e.g., in Clemen and Reilly (1999) in the context of decision-analysis modeling.
For the harder problem of providing the desired Pearson correlation, see Cario and Nelson (1997b) and Chen (1999). When the multivariate properties of the model are par-ticularly difficult, a simple method is to “bootstrap” from a data set for real-world data. Customers might arrive accord-ing to a fitted input model but have characteristics assigned by choosing, with replacement, a real-world past customer. The advantage and disadvantage is that every customer in the simulation will be identical, except for arrival time, to a real-world customer.
Similar to random vectors, the most tractable time-series models have normal marginal distributions. Lewis and colleagues (for example, Lawrance and Lewis 1981) developed a variety of ingenious simple time-series mod-els having nonnormal marginal distributions, often gamma. More generally, and as with random vectors, transforma-tion from the normal model works well. See Cario (1996), Cario and Nelson (1997a), Song and Hsiao (1993) and Song, Hsiao, and Chen (1995). In principal, transformation from processes with other marginal distributions could be used; for example, various simple queueing models provide time series with exponential marginal distributions, as discussed in Schmeiser and Song (1989).
5 POISSON PROCESSES
Point processes model events that occur at points in time (such as arrivals) or in space (such as defects). The Poisson process is the special case when arrivals, say, are independent (that is, the numbers of arrivals in nonoverlapping increments of time are independent of each other). Fortunately, a Poisson process model is often appropriate and can be verified by the physics of the situation (and without data collection) simply by asking whether there is any mechanism by which arrivals can “see” each other.
If a Poisson process model is appropriate, the only input-modeling issue is to determine the appropriate arrival rate, which can be any nonnegative function of time. If the rate is assumed to be constant, estimation is trivially easy. Only a bit more difficult is to assume that the rate is piecewise constant over intervals of time. Substantially more generally, Wilson and colleagues (e.g., Kuhl, Damerdji and Wilson 1997, 1998, Kuhl, Wilson and Johnson 1997) have investigated an assortment of increasingly complex functional forms to model trends and seasonalities.
6 FINAL COMMENT
Despite the recent advances in generality, automation via software, and visualization, simulation practitioners are of-ten left to their own devices when the input model becomes complex (for example, Ware et al. 1998 and Pritsker et al. 1995). What to do, for example, when a nonnormal time series of random vectors is needed? What to do when non-Poisson defects in time and space are to be modeled? The ideas above are useful building blocks, but the state of the art is far from allowing novice practitioners to build complex input models in the way that they can build complex logical models with today’s commercial software.
REFERENCES
Arkin, B. and L. Leemis. 1999. Nonparametric estimation of the cumulative intensity function for a nonhomo-geneous Poisson process from overlapping intervals. Technical Report. Department of Mathematics, Col-lege of William and Mary.
Avramidis, A. and J. R. Wilson. 1994. A flexible method for estimating inverse distribution functions in simulation experiments. ORSA Journal on Computing 6, 342–355. Barton, R. R. and L. W. Schruben. 1993. Uniform and bootstrap resampling of empirical distributions. In Pro-ceedings of the 1993 Winter Simulation Conference, ed. G. W. Evans, M. Mollaghasemi, E. C. Russell and W. E. Biles, 503–508. Piscataway, New Jersey: Institute of Electrical and Electronics Engineers.
Bratley, P., B. L. Fox and L. E. Schrage. 1987. A Guide to Simulation. New York: Springer-Verlag.
Cario, M. C. 1996. Modeling and simulating time series input processes with ARTAFACTS and ARTAGEN. In Proceedings of the 1996 Winter Simulation Conference, ed. J. M. Charnes, D. J. Morrice, D. T. Brunner and J. J. Swain, 207–213. Piscataway, New Jersey: Institute of Electrical and Electronics Engineers.
Cario, M. C. and B. L. Nelson. 1997a. Numerical meth-ods for fitting and simulating autoregressive-to-anything processes. INFORMS Journal on Computing 10, 72– 81.
Cario, M. C. and B. L. Nelson. 1997b. Modeling and gen-erating random vectors with arbitrary marginal distribu-tions and correlation matrix. Working paper. Depart-ment of Industrial Engineering and ManageDepart-ment Sci-ences, Northwestern University (available at http://primal.iems.nwu.edu/ nelsonb/). Chen, H.-F. 1999. Random-vector generation with NORTA.
Working paper. Department of Industrial Engineering, Chung Yuan Christian University, Chung Li, 320, Tai-wan.
Cheng, R. C. H., W. Holland and N. A. Hughes. 1996. Selection of input models using bootstrap
goodness-of-fit. In Proceedings of the 1996 Winter Simulation Conference, ed. J. M. Charnes, D. J. Morrice, D. T. Brunner and J. J. Swain, 199–206. Piscataway, New Jersey: Institute of Electrical and Electronics Engineers. Chick, S. E. 1997. Bayesian analysis for simulation input and output. In Proceedings of the 1997 Winter Simula-tion Conference, ed. S. Andrad´ottir, K. J. Healy, D. H. Withers and B. L. Nelson, 253–260. Piscataway, New Jersey: Institute of Electrical and Electronics Engineers. Clemen, R. T. and T. Reilly. 1999. Correlations and copulas for decision and risk analysis. Management Science 45, 208–224.
Devroye, L. 1986. Non-Uniform Random Variate Genera-tion. New York: Spring-Verlag.
Gross, D. and M. Juttijudata. 1997. Sensitivity of output performance measures to input distributions in queue-ing simulation modelqueue-ing. In Proceedqueue-ings of the 1997 Winter Simulation Conference, ed. S. Andrad´ottir, K. J. Healy, D. H. Withers and B. L. Nelson, 296–302. Piscataway, New Jersey: Institute of Electrical and Electronics Engineers.
Gross, D. and D. M. B. Masi. 1998. Sensitivity of output performance measures to input distributions in queue-ing network modelqueue-ing. In Proceedqueue-ings of the 1998 Winter Simulation Conference, ed. D. J. Medeiros, E. F. Watson, J. S. Carson and M. S. Manivannan, 629– 635. Piscataway, New Jersey: Institute of Electrical and Electronics Engineers.
Helton, J. C. 1996. Computational structure of a perfor-mance assessment involving stochastic and subjective uncertainty. In Proceedings of the 1996 Winter Simu-lation Conference, ed. J. M. Charnes, D. J. Morrice, D. T. Brunner and J. J. Swain, 239–247. Piscataway, New Jersey: Institute of Electrical and Electronics Engineers. Jankauskas, L. and S. McLafferty. 1996. BESTFIT, distribu-tion-fitting software by Palisade Corporation. In Pro-ceedings of the 1996 Winter Simulation Conference, ed. J. M. Charnes, D. J. Morrice, D. T. Brunner and J. J. Swain, 551–555. Piscataway, New Jersey: Institute of Electrical and Electronics Engineers.
Johnson, M. E. 1987. Multivariate Statistical Simulation. New York: John Wiley & Sons.
Kelton, W. D., B. L. Fox, M. E. Johnson, A. M. Law, B. W. Schmeiser and J. R. Wilson. 1990. Alterna-tive approaches for specifying input distributions and processes. In Proceedings of the 1990 Winter Simula-tion Conference, ed. O. Balci, R. P. Sadowski, R. E. Nance, 382–386. Piscataway, New Jersey: Institute of Electrical and Electronics Engineers.
Kelton, W. D., R. P. Sadowski and D. A. Sadowski. 1998. Simulation with ARENA. Boston: McGraw Hill. Klein, R. W. and S. D. Roberts. 1984. A time-varying
Kuhl, M. E., H. Damerdji and J. R. Wilson. 1997. Esti-mating and simulating Poisson processes with trends or asymmetric cyclic effects. In Proceedings of the 1997 Winter Simulation Conference, ed. S. Andrad´ottir, K. J. Healy, D. H. Withers and B. L. Nelson, 287–295. Piscataway, New Jersey: Institute of Electrical and Electronics Engineers.
Kuhl, M. E., H. Damerdji and J. R. Wilson. 1998. Least squares estimation of nonhomogeneous Poisson pro-cesses. In Proceedings of the 1998 Winter Simulation Conference, ed. D. J. Medeiros, E. F. Watson, J. S. Carson and M. S. Manivannan, 637–645. Piscataway, New Jersey: Institute of Electrical and Electronics En-gineers.
Kuhl, M. E., J. R. Wilson and M. A. Johnson. 1997. Estimating and simulating Poisson processes having trends or multiple periodicities. IIE Transactions 29, 201–211.
Law, A. M. and M. G. McComas. 1998. EXPERTFIT: Total support for simulation input modeling. In Pro-ceedings of the 1998 Winter Simulation Conference, ed. D. J. Medeiros, E. F. Watson, J. S. Carson and M. S. Manivannan, 303–308. Piscataway, New Jersey: Institute of Electrical and Electronics Engineers. Lawrence, A. J. and P. A. W. Lewis. 1981. A new
au-toregressive time series model in exponential variables. Advances in Applied Probability 13, 826–845. Leemis, L. 1991. Nonparametric estimation of the
cumula-tive intensity function for a nonhomogeneous Poisson process. Management Science 37, 886–900.
Leemis, L. M. 1996. Discrete-event simulation input process modeling. In Proceedings of the 1996 Winter Simulation Conference, ed. J. M. Charnes, D. J. Morrice, D. T. Brunner and J. J. Swain, 39–46. Piscataway, New Jersey: Institute of Electrical and Electronics Engineers. Leemis, L. 1997. Seven habits of highly successful input modelers. In Proceedings of the 1997 Winter Simulation Conference, ed. S. Andrad´ottir, K. J. Healy, D. H. Withers and B. L. Nelson, 39–46. Piscataway, New Jersey: Institute of Electrical and Electronics Engineers. Leemis, L. 1998. Input modeling. In Proceedings of the 1998 Winter Simulation Conference, ed. D. J. Medeiros, E. F. Watson, J. S. Carson and M. S. Manivannan, 15– 22. Piscataway, New Jersey: Institute of Electrical and Electronics Engineers.
Leemis, L. 1999. Simulation input modeling. In this volume.
Lewis, P. A. W. and E. J. Orav. 1989. Simulation Method-ology for Statisticians, Operations Analysts, and Engi-neers, Volume 1. Pacific Grove, California: Wadsworth & Brooks/Cole.
Nelson, B. L. and B. Schmeiser. 1986. Decomposition of some well-known variance reduction techniques.
Jour-nal of Statistical Computation and Simulation 23, 183– 209.
Nelson, B. L. and M. Yamnitsky. 1998. Input modeling tools for complex problems. In Proceedings of the 1998 Winter Simulation Conference, ed. D. J. Medeiros, E. F. Watson, J. S. Carson and M. S. Manivannan, 105– 112. Piscataway, New Jersey: Institute of Electrical and Electronics Engineers.
Pritsker, A. A. B., D. L. Martin, J. S. Reust, M. A. F. Wag-ner, J. R. Wilson, M. E. Kuhl, J. P. Roberts, O. P. Daily, A. M. Harper, E. B. Edwards, L. Bennett, J. F. Burdick and M. D. Allen. 1995. Organ transplantation policy evaluation. In Proceedings of the 1995 Winter Simu-lation Conference, ed. C. Alexopoulos, K. Kang, W. Lilegdon and D. Goldsman, 1314–1323. Piscataway, New Jersey: Institute of Electrical and Electronics En-gineers.
Ramberg, J. and B. Schmeiser. 1974. An approximate method for generating asymmetric random variables. Communications of the ACM 17, 78–82.
Reilly, C. H. 1998. Properties of synthetic optimization problems. In Proceedings of the 1998 Winter Simula-tion Conference, ed. D. J. Medeiros, E. F. Watson, J. S. Carson and M. S. Manivannan, 617–621. Piscataway, New Jersey: Institute of Electrical and Electronics En-gineers.
Schmeiser, B. 1977. Methods for modeling and generating probabilistic components in digital computer simulation when the standard distributions are not adequate: A survey. In Proceedings of the 1977 Winter Simulation Conference, ed. H. J. Highland, R. G. Sargent and J. W. Schmidt, 51–57. (Reprinted in Simuletter 10, Fall 1978/Winter 1979, 38–43, 72.)
Schmeiser, B. 1990. Simulation experiments. Chapter 7 in Handbooks in Operations Research and Management Science, Volume 2: Stochastic Models, ed. D.P. Heyman and M.J. Sobel, 295–330. Amsterdam: North-Holland. Schmeiser, B. and S. Deutsch. 1977. A versatile four-parameter family of probability distributions, suitable for simulation. AIIE Transactions 9, 176–182. Schmeiser, B. and R. Lal. 1980. Multivariate modeling
in simulation: A survey. 33rd Annual Technical Con-ference Transactions, 252–261, American Society for Quality Control.
Schmeiser, B. and R. Lal. 1982. Bivariate gamma random vectors. Operations Research 30, 355–374.
Schmeiser, B. and W. T. Song. 1989. Inverse-transformation algorithms for some common stochastic processes. Pro-ceedings of the 1989 Winter Simulation Conference, ed. E. A. MacNair, K. J. Musselman and P. Heidelberger, 490–496. Piscataway, New Jersey: Institute of Elec-trical and Electronics Engineers.
distribution. In Proceedings of the 1993 Winter Simu-lation Conference, ed. G. W. Evans, M. Mollaghasemi, E. C. Russell and W. E. Biles, 374–377. Piscataway, New Jersey: Institute of Electrical and Electronics En-gineers.
Song, W. T., L. Hsiao and Y. Chen. 1995. Generation of pseudorandom time series with specified marginal dis-tributions. European Journal of Operational Research 93, 194–202.
Stanfield, P. M., J. R. Wilson, G. A. Mirka, N. F. Glasscock, J. P. Psihogios and J. R. Davis. 1996. Multivariate input modeling with Johnson distributions. In Proceed-ings of the 1996 Winter Simulation Conference, ed. J. M. Charnes, D. J. Morrice, D. T. Brunner and J. J. Swain, 1457–1464. Piscataway, New Jersey: Institute of Electrical and Electronics Engineers.
Wagner, M. A. F. and J. R. Wilson. 1995. Graphical interactive simulation input modeling with bivariate B´ezier distributions. ACM Transactions on Modeling and Computer Simulation 5, 163–189.
Wagner, M. A. F. and J. R. Wilson. 1996a. Using univariate Bezier distributions to model simulation input processes. IIE Transactions 28, 699–712.
Wagner, M. A. F. and J. R. Wilson. 1996b. Recent devel-opments in input modeling with B´ezier distributions. In Proceedings of the 1996 Winter Simulation Confer-ence, ed. J. M. Charnes, D. J. Morrice, D. T. Brunner and J. J. Swain, 1448–1456. Piscataway, New Jersey: Institute of Electrical and Electronics Engineers. Ware, P. P., T. W. Page Jr. and B. L. Nelson. 1998.
Automatic modeling file system workloads using two-level arrival processes. ACM Transactions on Modeling and Computer Simulation 8, 305–330.
Wilson, J. R. 1997. Modeling dependencies in stochastic simulation inputs. In Proceedings of the 1997 Winter Simulation Conference, ed. S. Andrad´ottir, K. J. Healy, D. H. Withers and B. L. Nelson, 47–52. Piscataway, New Jersey: Institute of Electrical and Electronics En-gineers.
AUTHOR BIOGRAPHY
