There are no standard approaches for deterministic computer experiments as there is no variation in the results, if you ran the model 20 times with the same factor set up you would get exactly the same answer each time.
Stochastic computer experiments have the variation modeled in, a lot of the time this will have been through a mix of physical trials and/or SME judgment.
Again, as with the previous section I only discuss a few standard designs in addition to physical design methods.
Latin hypercube sample (LHS) is a commonly used design. The factors are scaled from 0 to 1 and are scaled based on the minimum and maximum val-ues for each one. If the computer experiment has time to run to completion this is a recommended method as it will thoroughly cover the system space.
However if the experiment has to be stopped early then there may be spaces with no information due to the randomization of placing the points.
Low discrepancy sequence also has its factors scaled from 0 to 1 by the mini-mum and maximini-mum values. However the advantage of this over the LHS is if the simulation may be stopped early as this method pushes the points around the design space. The algorithm works by trying to fill the empty spaces so if it does have to finish early there is still a good spread of information around the design space. The disadvantage though is that it takes longer to run than the LHS. Figure 1-17 shows an example of the difference between the LHS and the low discrepancy sequence if stopped after 10 runs instead of 25 runs.
Chapter 1 | Design of Experiments
26
Uniform designs are useful if you are only interested in an average, which may not be ideal if the response is complex and has regions of extreme change.
This design is another that scales the factors from 0 to 1, but it then puts points on a neat grid rather than randomly scattered through the space.
An example where a uniform design wouldn’t be appropriate is using a simu-lated motorcycle accident, as data shown in Figure 1-18.
Figure 1-17. LHS and low discrepancy sequence designs stopped after 10 runs
Translating Statistics to Make Decisions 27
The measurements recorded are head acceleration across time after impact.
It is clear that this is a complex relationship as the head jerks back and forth in an accident. Figure 1-19 shows the difference between using a uniform design compared to a LHS design for investigating the relationship between the variables.
Figure 1-18. Relationship between variables from a motorcycle accident
Chapter 1 | Design of Experiments
28
If the uniform design had been used a lot of important information would be missing and we wouldn’t have seen the pattern discovered through using the LHS design.
When there are very complex models that take days to run iterations of the simulations, meta-models can be developed to give a quick response. A sample will be required to train the meta-model, and it should include uncertainty about the predictions. Once trained it should be a good representation of the complex model that will produce much quicker results. The tradeoff is that the response will be less precise, but it’s a good indicator before the complex model results are completed.
At the observation points of the meta-model there is little or no variation, however between the points the variance increases, and once beyond the limits of the outer points the variance increases exponentially, so care should be taken with the estimates made outside the recorded range.
Surveys
The other primary way in which data is collected is through surveys. Some of the advice also can be used in questionnaires given to participants during a designed experiment. However survey design as a data collection method generally refers to the case where the participants are not known or are assigned beforehand.
Figure 1-20 shows the suggested process to follow when thinking about designing any survey.
Figure 1-19. Uniform design compared to a LHS design to model a motorcycle accident
Translating Statistics to Make Decisions 29
Once you have assessed who the population includes and the population size, a representative sample size will need to be calculated. However this will need to be tweaked due to nonresponses. Response rates for surveys can be quite low, so you will need to send out extra surveys to account for this. Additional thought should be given to the reasons for nonresponses and how this may cause bias in the results and affect the scope of the study.
When designing the survey itself and this applies to designed experiments questionnaires as well, there is quite a lot to think about. Personally I consult a human factors specialist as it’s not my area of expertise, but the following are general areas for consideration.
Wording of the questions can be quite tricky, especially if a suitable survey design doesn’t already exist. Keep the language simple but precise as to what you are asking, keep the questions short, avoid leading questions, determine whether the question is multiple choice, and don’t always be biased toward using positive statements.
Think about the question response type and the implications during analysis they may have. For example, if an open text box was the only response option available, which analysis methods could be used? However, it can be useful to provide an optional text box as it can help clarify why participants have selected a specific answer. Likert responses are a popular method for ques-tionnaires, but thought needs to be given to whether there is an even or odd number of categories, that is, do you allow an “on the fence” option. There are multiple opinions about this, but it’s ultimately a choice whether you want to force a subject to pick a positive or negative answer when they may not want to or allow a neutral answer that may not be informative in the analysis.
Figure 1-20. Survey design thought process
Chapter 1 | Design of Experiments
30
Demographics also should be collected whether it’s a key focus of the sur-vey or not, as they can help inform results during the analysis. Again there is discussion as to whether this should be placed at the beginning or the end of the questionnaire.
It’s always recommended to run a pilot study first as this will highlight any ambiguity in the questions, suggest where closed questions should include more response options, show where a question would be better placed in a different format, and also indicate whether the survey is a manageable length.
Once the experimental design has been completed, the next item to do is think about is collecting the data and formatting it.
Summary
This chapter delved into the thought that needs to go into designing experi-ments and showed a suggested process to follow to make sure all the relevant information is collected. It was split into four main sections corresponding to the four steps shown in the design of experiment process in Figure 1-1.
The first of which showed how to form the study question that will then be turned into a hypothesis. It explained briefly how hypotheses are formed and also described the type of questions that should be asked of the customer.
The second section looked at power and sample size calculations and showed some common misunderstandings involved with the “how many” question.
Within this section the first subsection explained the information needed to run the calculations, such as risk and other values dependent on data type, and how to translate that to get it from the customer. The second subsection then delved into how to actually conduct the calculations using R and how to interpret the results.
The third section discussed defining the scope of the study including thinking about the applicability of results in other scenarios and also the assumptions that need to be justified and recorded.
The last section was the largest and moved onto how to design the experi-ments themselves. This was split into three subsections, variables, designed experiments, and surveys.
The first subsection described some common terminology linked to vari-ables, and then went on to explain interactions and confounding effects with examples.
The second subsection probed into some of the available designs for both physical and computer experiments. This included factorial and optimal designs, adaptive designs, and some of the other popular designs for physical
Translating Statistics to Make Decisions 31
experiments, as well as some off the common computer experiments designs including Latin hypercube samples, low discrepancy sequences, and uniform designs.
The final subsection briefly investigated survey design including a suggested process of what needs to be thought about along with some further details of each stage.
Chapter 2 moves to actually collecting the data and what considerations should be given before jumping straight into carrying out the experimental design created. It also looks at the formatting side of things once the data is being/has been collected.
© Victoria Cox 2017
V. Cox, Translating Statistics to Make Decisions, DOI 10.1007/978-1-4842-2256-0_2
C H A P T E R