V ALIDATION
“Truth comes out of error more easily than out of confusion.”
—Francis Bacon
8.1 Introduction
Building a simulation model is much like developing an architectural plan for a house. A good architect will review the plan and specifications with the client or owner of the house to ensure that the design meets the client’s expectations.
The architect will also carefully check all dimensions and other specifications shown on the plan for accuracy. Once the architect is reasonably satisfied that the right information has been accurately represented, a contractor can be given the plan to begin building the house. In a similar way the simulation analyst should examine the validity and correctness of the model before using it to make implementation decisions.
In this chapter we cover the importance and challenges associated with model verification and validation. We also present techniques for verifying and validat- ing models. Balci (1997) provides a taxonomy of more than 77 techniques for model verification and validation. In this chapter we give only a few of the more common and practical methods used. The greatest problem with verification and validation is not one of failing to use the right techniques but failing to use any technique. Questions addressed in this chapter include the following:
• What are model verification and validation?
• What are obstacles to model verification and validation?
• What techniques are used to verify and validate a model?
• How are model verification and validation maintained?
Two case studies are presented at the end of the chapter showing how verifi- cation and validation techniques have been used in actual simulation projects.
203
8.2 Importance of Model Verification and Validation
Model building is, by nature, very error prone. The modeler must translate the real-world system into a conceptual model that, in turn, must be translated into a simulation model. This translation process is iterative as the modeler’s conception of the real system changes and the modeling approach continues to be refined (see Figure 8.1). In this translation process, there is plenty of room for making errors. Verification and validation processes have been developed to reduce and ideally eliminate these errors.
Model verification is the process of determining whether the simulation model correctly reflects the conceptual model. Model validation is the process of determining whether the conceptual model correctly reflects the real system.
Model verification and validation are critical to the success of a simulation project. Important decisions may be based on the outcome of the simulation ex-periment, and, therefore, demonstrable evidence should exist for the validity of the model. For simulations that are used on an ongoing basis for operational deci- sions such as production scheduling, model validation is even more important. The Department of Defense (DoD) has recognized the importance of model validation and has established a defined set of criteria for determining model accreditation. Accreditation, as defined by DoD (http://www.dmso.mil/docslib/ mspolicy/glossary/9801glss.doc), is “The official certification that a model or simulation is acceptable for use for a specific purpose.”
8.2.1 Reasons for Neglect
Despite their importance, seldom is adequate attention given to verification and validation when doing simulation. The primary reasons for neglecting this FIGURE 8.1
Two-step
translation process to convert a real-world system to a simulation model.
System
Concept
Model
important activity are
• Time and budget pressures.
• Laziness.
• Overconfidence.
• Ignorance.
Time and budget pressures can be overcome through better planning and in-creased proficiency in the validation process. Because validation doesn’t have to be performed to complete a simulation study, it is the activity that is most likely to get shortchanged when pressure is being felt to complete the project on time or within budget. Not only does the modeler become pressed for time and resources, but often others on whom the modeler relies for feedback on model validity also become too busy to get involved the way they should.
Laziness is a bit more difficult to deal with because it is characteristic of human nature and is not easily overcome. Discipline and patience must be devel- oped before one is willing to painstakingly go back over a model once it has been built and is running. Model building is a creative activity that can actually be quite fun. Model verification and validation, on the other hand, are a laborious effort that is too often dreaded.
The problem of overconfidence can be dealt with only by developing a more critical and even skeptical attitude toward one’s own work. Too often it is assumed that if the simulation runs, it must be okay. A surprising number of deci- sions are based on invalid simulations simply because the model runs and pro- duces results. Computer output can give an aura of validity to results that may be completely in error. This false sense of security in computer output is a bit like saying, “It must be true, it came out of the computer.” Having such a naive men- tality can be dangerously misleading.
The final problem is simply a lack of knowledge of verification and valida-tion procedures. This is a common problem particularly among newcomers to simulation. This seems to be one of the last areas in which formal training is re-ceived because it is presumed to be nonessential to model building and output analysis. Another misconception of validation is that it is just another phase in a simulation project rather than a continuous activity that should be performed throughout the entire project. The intent of this chapter is to dispel common mis- conceptions, help shake attitudes of indifference, and provide insight into this im- portant activity.
8.2.2 Practices That Facilitate Verification and Validation
A major problem in verifying and validating models occurs because of poor mod- eling practices. Beginners often build models with little or no thought about being able to verify or validate the model. Often these models contain spaghetti code that is difficult for anyone, including the modeler, to follow.
This problem be- comes even more acute as models grow in complexity. When the original model builder moves on and others inherit these models, the desperate efforts to unravel
the tangled mess and figure out what the model creator had in mind become almost futile. It is especially discouraging when attempting to use a poorly con-structed model for future experimentation. Trying to figure out what changes need to be made to model new scenarios becomes difficult if not impossible.
The solution to creating models that ease the difficulty of verification and val- idation is to first reduce the amount of complexity of the model.
Frequently the most complex models are built by amateurs who do not have sense enough to know how to abstract system information. They code way too much detail into the model. Once a model has been simplified as much as possible, it needs to be coded so it is easily readable and understandable. Using object-oriented techniques such as encapsulation can help organize model data.
The right simulation software can also help keep model data organized and readable by providing table entries and intelligent, parameterized constructs rather than requiring lots of low-level pro- gramming. Finally, model data and logic code should be thoroughly and clearly documented. This means that every subroutine used should have an explanation of what it does, where it is invoked, what the parameters represent, and how to change the subroutine for modifications that may be anticipated.
8.3 Model Verification
Verification is the process of determining whether the model operates as intended. It doesn’t necessarily mean that the model is valid, only that it runs correctly. According to Banks, et al. (2001), “Verification is concerned with building the model right. It is utilized in the comparison of the conceptual model to the com- puter representation that implements that conception.”
Model verification in- volves the modeler more than the customer. During the verification process, the modeler tries to detect unintended errors in the model data and logic and remove them. In essence, verification is the process of debugging the model. A verified model is a bug-free model.
Errors or bugs in a simulation model are of two types: syntax errors and se-mantic errors. Syntax errors are like grammatical errors and include the uninten-tional addition, omission, or misplacement of notation that either prevents the model from running or causes it to run incorrectly. The omission or misplacement of a decimal point in a number or parentheses in an expression can dramatically impact the outcome of a simulation. Most simulation software has built-in error detection that can identify inconsistencies and omissions.
This feature is much like spelling and grammar checking in a word processor.
As with a word proces- sor, however, it still remains the responsibility of the user to ensure that the in- tended meaning has been conveyed.
Semantic errors are associated with the meaning or intention of the modeler and are therefore difficult to detect. Often they are logical errors that cause behav- ior that is different than what was intended. For example, an if–then test on a vari- able may branch to the wrong logic simply because the condition was incorrectly specified or the branching was not set up correctly. Or numerical values may