4.2 Building trust
4.2.2 Verification and validation
Verification and validation10are the general names given to a host of methods used
for increasing reliability in scientific models and computer software. Understanding their role, then, turns out to be essential for the assessment, credibility and power to elicit the results of computers. Now, when the interest is on computer simulations, verification and validation methods are tailored to specific tasks. In the following, I’ll first discuss the generalities of verification and validation methods and discuss later in what respect they increase the reliability of computer simulations.
In order to account for the reliability of the computer simulations as well as to sanction the correctness of the results, researchers have at their disposal formal pro- cedures (e.g., for confirming the correct implementation of the specification in the computer) and benchmarks (i.e., accurate reference values) against which to com- pare the results of the computation (e.g., against other sources of data). In verifi- cation, formal methods are at the center for the reliability of computer software, whereas in validation, benchmarking is responsible for confirmation of the out- comes (Oberkampf and Roy 2010, Preface). In other words, in verification methods, the relationship of interest is between the specification – including the model – and the computer software, whereas in validation methods the relationship of interest is between the computation and the empirical world (e.g., experimental data obtained by measuring and observing methods) (Oberkampf, Trucano, and Hirsch 2003).11
Here are two definitions that are largely accepted and used by the community of researchers:
Verification: the process of determining that a computational model accurately represents the underlying mathematical model and its solution.
10Also known as ‘internal validity’ and ‘external validity,’ respectively.
11 To be more fair with Oberkampf’s, Truncano’s, Roy’s, and Hirsch’s general proposal, I must
also mention the analysis on uncertainty and how it propagates throughout the process of designing, programming, and running computer simulations. For more philosophical treatment on verification and validation, as well as concrete examples see (Oreskes, Shrader-Frechette, and Belitz 1994; K¨uppers and Lenhard 2005; Hasse and Lenhard 2017).
Validation: the process of determining the degree to which a model is an accurate represen- tation of the real world from the perspective of the intended uses of the model. (Oberkampf, Trucano, and Hirsch 2003)
Now, this way of depicting verification and validation is widely accepted and used, specially in many philosophical treatments of computer simulations. Eric Winsberg, for instance, takes it that “verification, [...] is the process of determin- ing whether or not the output of the simulation approximates the true solutions to the differential equations of the original model. Validation, on the other hand, is the process of determining whether or not the chosen model is a good representation of the real-world system for the purpose of the simulation” (Winsberg 2010, 19-20). Another example of a philosopher discussing verification and validation for com- puter simulations is Margaret Morrison. Although she takes a wider definition of verification and validation methods, and even thinks that these two methods are not always clearly divisible, she nevertheless downplays the need for verification meth- ods claiming that validation is more crucial of a method for assessing the reliability of computer simulation (Morrison 2009, 43).
The scientific and engineering communities, on the other hand, have a wider and more diverse set of definitions to offer, all tailored to the specificities of the systems under study.12Let us now discuss them separately and point out what is so specific
about computer simulations. 4.2.2.1 Verification
Verification in computer simulations consists in making sure that the specification for a given simulation is correctly implemented as a simulation model. The litera- ture provides several verification methods suitable for computer software in general, but there are two methods specifically important for computer simulations, namely, code verification and calculation verification13. Their importance lies in that both
methods focus on the correctness of the discretization, a key element for implement- ing mathematical models as computer simulations.
Code verification is defined as the process of determining that the numerical al- gorithms are correctly implemented in the computer code, as well as of identifying potential errors in the software (Oberkampf, Trucano, and Hirsch 2003, 32). In this respect, code verification provides a framework for developing and maintaining re- liable computer simulation code.
William Oberkampf and Timothy Trucano have argued that it is useful to further segregate code verification into two activities, namely, numerical algorithm verifi- cation and software quality engineering. The purpose of numerical algorithm veri- fication is to address the mathematical correctness of the implementation of all the
12See (Oberkampf and Roy 2010, 21-29) for an analysis on the diversity of concepts. Also, see
(Salari and Kambiz 2003; Sargent 2007; Naylor et al. 1967; Naylor, Wallace, and Sasser 1967).
13 Also referred to as solution verification in (Oberkampf and Roy 2010, 26), and as numerical
4.2 Building trust 111
numerical algorithms that affect the numerical accuracy of the results of the sim- ulation. The goal of this verification method is to demonstrate that the numerical algorithms implemented as part of the simulation model are correctly implemented and functioning as intended (William L. Oberkampf and Timothy G. Trucano 2002, 720).
Software quality engineering, instead, puts the emphasis on determining whether the simulation model produces the correct results – or approximately correct. The purpose of software quality engineering is to verify the simulation model and the re- sults of the simulation on a specific computer hardware, in a specified software envi- ronment – including compilers, libraries, I/O, etc. These verification procedures are primarily in use during the development, testing, and maintenance of the simulation model (721).
On the other hand, calculation verification is defined as the method that prevents three kinds of errors: human error in the preparation of the code, human error in the analysis of the results, and numerical errors resulting from computing the dis- cretized solution of the simulation model. A definition for calculation verification is the following:
Calculation verification: the process of determining the correctness of the input data, the numerical accuracy of the solution obtained, and the correctness of the output data for a particular simulation. (Oberkampf, Trucano, and Hirsch 2003, 34)
Thus understood, calculation verification is the empirical side of verification. It is based on the comparison between the results of the simulation against highly ac- curate solutions of the scientific model. In a sense, calculation verification is similar to validation assessment insofar as both compare estimated results with correct re- sults. It most commonly controls spatial and temporal convergence rates, iterative convergence, independence of solutions to coordinate transformations, and similar other processes (26).
4.2.2.2 Validation
The process of validation (also known as testing) consist in showing that the re- sults of the simulation correspond, more or less accurately and precisely, to those obtained by measurement and observation of the target system. Oberkampf and Tru- cano highlight three key aspects of validation:
i) quantification of the accuracy of the computational model by comparing its responses with experimentally measured responses,
ii) interpolation or extrapolation of the computational model to conditions corresponding to the intended use of the model, and
iii) determination if the estimated accuracy of the computational model, for the conditions of the intended use, satisfies the accuracy requirements specified. (W L Oberkampf and T G Trucano 2008, 724)
Although validation methods come naturally to many experimenters, as they ex- pect to reproduce – as opposed to represent or imitate – a piece of the world,14they
are rather a complex matter in the context of computer simulations. Here is when some concerns appear regarding the dependability of validation.
A chief problem stems from the fact that most validation methods are inductive, and thus must be expected to face typical problems of induction. The issue here is that the method only allows validation of a model up to a certain point, and therefore complete validation is absolutely impossible due to the large number of comparisons needed – not to mention the improbability of having all the possible results at hand. This is a reason why validation is known, mostly among computer scientists, as a method for detecting the presence of errors, but not designed for establishing their absence.15
Another problem is that validation depends on the capacity of comparing com- puter simulation results with empirical data. Such a dual relation obviously requires the presence of both, the results of the simulation and data gathered from an em- pirical source. This leads to the exclusion of the many computer simulations for which there are no counterpart empirical data. In this sense, validation is only a suitable concept for those cases in which a computer simulation is representing an actual system, and not a possible or conceivable system (e.g., a simulation that vi- olates a constant of nature, such as simulation with a gravitational force equal to G = 1mkg 1s 2).
Having said this, it is important to mention that, with the introduction of com- puter simulations in experimental contexts, validation does not exclusively depends on contrasting results against empirical data. Ajelli and team show how it is possi- ble to run different computer simulations and use their results to assert the reliability for each one – in this case, there is not mere convergence of results, but also of key variables (Ajelli et al. 2010).16
Figure 4.1 shows in a flow diagram how verification (both code verification and calculation verification) and validation methods are put into effect in standard scien- tific practice. The conceptual model here is the product of analyzing and observing the physical system of interest (i.e., what we called the scientific model). In key applications of computational physics (such as computational fluid dynamics, com- putational solid mechanics, structural dynamics, shock wave physics, and computa- tional chemistry), the conceptual model is dominated by the set of PDEs used for representing physical quantities.
Two further types of model can be identified: a mathematical model, from which the computational or simulation model is created, and a physical model which, for simplicity, we shall identify with an experiment (recall our treatment of experimen- tation in section 3). The computational model, in our own terminology, is an op- erational computer program that implements the simulation model as a computer simulation.
14Whereas this is a valid claim for some form of experimentations in science, it is not so for others
such as economics and psychology.
15This claim is widely attributed to Edsger Dijkstra.
4.2 Building trust 113
Fig. 4.1 Phases of modeling and simulation, and the role of verification and validation (ASME 2006, 5).
Let us note that the figure also shows that code verification deals with the fi- delity between the conceptual model and the mathematical model, whereas calcu- lation verification deals with the agreement between the simulation results and the expected results of the computational model. Validation, on the other hand, is a quantitative adequacy between the solutions of the simulation and the experimen- tal measurements or observations. This adequacy, again, can be determined by a comparison that provides an acceptable agreement between the solutions of the two models involved.17
Verifications and validations are two fundamental pillars for asserting the reli- ability of computer simulations (Dur´an and Formanek 2018). This means that re- searchers are on good grounds for claiming that their computer simulations are re- liable processes, and thus that they are justified in believing that the results of the simulation are correct.
This outcome is important because, as we shall argue in the next chapter, relia- bility of computer simulations support the claim that researchers are able to claim for explanation, prediction, and other epistemic activities. In other words, reliable computer simulations also allow researchers to claim for understanding of the re- sults.
A final and very short comment on verification and validation before introducing the next topic. Although neither software nor hardware can be fully verified or vali- dated, researchers are still developing methods that reduce the occurrence of errors and increases the credibility of the simulation. This interest works as evidence for the importance of both methods for the general reliability of computer simulations.