What’s the Di ff erence between Experiment and Simulation?

To underline a point which should be clear by now: When we are talking about broad- sense simulation and comparing experiments to physical simulations (studies of physical models), there is no interesting or important difference between experiment and

simulation. I have discussed cases which we might think of as either experiments or broad sense simulations. It does not ultimately matter, for the purpose of evaluating the

inferences in question, whether we call these experiments or simulations. The same is true for any other study where researchers are intervening in a physical system (their object) for the purpose of making some inference about some other physical system (their target). Other than in very rare cases like the ones mentioned in Section 4.5, scientists’ objects of study are always stand-ins for their target. They almost never study their targets directly. The important question for evaluating what we can learn from a given study of a physical system is not “Is it a simulation or an experiment?” The important question is, how is this object of study being used to generate or justify inferences about the target of inquiry in

question. I agree with Winsberg (2009, p. 591) when he says that “[h]ow trustworthy or reliable an experiment or simulation is depends on the quality of the background

knowledge, and the skill with which it is put to use, and not on which kind it belongs to.” There is no epistemic or methodological distinction between experiments and physical simulations. Focusing on which kind a case of scientific inquiry belongs to focuses us on the wrong issues.

That was all about experiments versus physical simulations. When we’re talking about experiments versus computer simulations rather than broad-sense simulations, there is an important methodological difference at play, namely, the difference between studying a physical system and studying a computer simulation. This matters for pragmatic reasons. Most often, doing an experiment will be more costly than doing a simulation. The supplies, reagents, and person hours needed to run a laboratory experiment tend to cost significantly more than running a model on a computer. (I say tend to because even here there are exceptions: running most middle-school chemistry experiments costs significantly less than running meteorologists’ climate models).

Simulations can allow one to observe an object of study’s dynamics over time much more quickly than doing so in a real-time experimental system.

This pragmatic advantage can come with epistemic costs. Many people have the intuition that it always comes with epistemic costs; this is an important part of the

intuition which the materiality thesis tries to explain: The idea is that studying a model as opposed to a material system involves sacrificing realism, and sacrificing realism reduces epistemic value. Again, this is a good intuition in contexts where we know relatively little about the features of our target of inquiry relevant to designing a good experimental system or model. But science is not always operating in such contexts. One example of a

context in which there is no such epistemic cost associated with simulation is the study of molecular bond angles in chemistry. We know enough about chemical bonding to answer questions via mathematical modeling and computer simulation about how atomic

substitutions will affect the bond angle in a given molecule; for example, swapping atoms of phosphorus for atoms of arsenic in the molecular backbone of DNA (as in Denning & MacKerell 2011). For answering questions about straightforward atomic substitutions in familiar molecules, we would not be in a better epistemic situation were we to carry out the relevant experimental manipulation (and it would certainly be far more pragmatically costly). So again, the point about the epistemic costs of simulation is a point that holds in many contexts. But it is not an in-principle epistemic difference between experiment and simulation.

The methodological difference between experiment and simulation is not purely pragmatic. It matters for making judgments about epistemic value—but only in a context- sensitive way. All of science is about engaging with some object of study to learn about some target of inquiry, and very rarely are the object and target identical. We should not look to the experiment/simulation distinction alone to tell us anything in principle about the epistemic value of cases of scientific inquiry.

A final point about the difference between studying experimental systems and computer models: Even regarding this version of the experiment/simulation distinction, people have been hasty in drawing sharp methodological lines (and using these, in turn, as a basis for conclusions about the epistemic superiority of experiments). While it was once common for individual scientists, laboratories, or even entire subfields to focus on one or the other, experimental and computational methods are now increasingly

identity of the object of study, the methodological overlap between experiment and computer simulation in another sense is often significant. The views on experiment versus simulation cited above often talk as if researchers choose to do one or the other, as their main program of research or even within the context of a given study. But this is

increasingly not the case. Computer simulations are often a key part of the experimental process. Of course, one way computation is used in biology is to solve mathematical models, like population- and Mendelian-based models of evolutionary genetics. But simulations can play plenty of other roles in the experimental process as well. For example, they are used as initial steps in LNS experiments to help figure out which variables to fix (Roff & Fairbarin 2009), or to explore theoretical questions about spatial structure in experimental microbial communities in tandem with studying those communities themselves (Kerr et al. 2002).

I have argued that we should not look to the experiment/simulation distinction to tell us anything in principle about epistemic value. I have shows that two senses in which experiments are commonly thought to have epistemic privilege over simulations— inferential power and capacity to generate surprises—do not generalize across science. Studying a material system as opposed to a computer model does not automatically entail better inferences. In Chapter 5 I will begin to develop an account of where we should look instead.

5. Conclusion: Evaluating Inferences about the Natural