Experiment Design and Specification

The third step in the design of an experiment is to establish the appropriate range over which variables should be varied. This is the parameterisation of variables and a fundamental step in establishing the experimental design. The design is translated into a specification by mapping the combination of parameter variations into treatments and blocks.

6.1.5.1 Treatments and Blocks

Treatments refer to variations of a specified parameter, whilst blocks refer to the conditions under which the parameter is varied. The design framework identified two treatment dimensions: permitted agent behaviours and market conditions. This leads naturally to the design of two related experiment designs: 1) an experiment where market structure (treatment) is varied in specified permitted behavioural contexts (blocks); and 2) an experiment where permitted agent behaviours are varied (treatment) in specified market conditions (blocks). This fundamental structure for the design of the experiments will reveal data that permits a complete investigation of how the variables describing both market structure and permitted agent behaviours of collaboration and price competition impact open and closed network disruption responses to normal operations. Table 6-2 summarises the 32 configurations of market structure, and Table 6-3: the eight configurations of permitted behaviours.

Table 6-2: Market structure configurations Treatment /Blocks Manufacturing Price Variation Manufacturing Mean Margin Retail Price Variation Wholesale Price Variation Mean Wholesale Margin

1 High High High High High

2 High High High High High

3 High High High High High

4 High High High High High

5 High High High High High

6 High High High High High

7 High High High High High

8 High High High High High

9 High High High High High

10 High High High High High

11 High High High High High

12 High High High High High

13 High High High High High

14 High High High High High

15 High High High High High

16 High High High High High

17 High High High High High

18 High High High High High

19 High High High High High

20 High High High High High

21 High High High High High

22 High High High High High

23 High High High High High

24 High High High High High

25 High High High High High

26 High High High High High

27 High High High High High

28 High High High High High

29 High High High High High

30 High High High High High

31 High High High High High

Table 6-3: Permitted behaviours configurations

Treatments

/blocks _{Collaboration Price} Permitted Behaviours

competition

Open Market Closed Market

1 No No No Yes

2 Yes No No Yes

3 No Yes No Yes

4 Yes Yes No Yes

5 No Yes Yes No

6 Yes Yes Yes No

7 Yes No Yes No

8 No No Yes No

6.1.5.2 Replications

Having specified the levels of the logically argued treatments and blocks the final component of the experimental design is the number of replications that are required to control for the nuisance variables.

In noncomplex systems this generally involves using statistical methods to define sampling and repetition regimes. However, as already established, CASs are non-deterministic in that they are non-linear, path dependent and emergent. Levin (2002) eloquently summarised these characteristics as:

“The study of complex adaptive systems is the study of systems limited in their predictability. Because complex adaptive systems are systems in which microscopic interactions and evolutionary processes give rise to macroscopic phenomena through nonlinear interactions, these systems are subject to path dependence, with implications for the likelihood of multiple stable states, chaotic dynamics and frozen accidents.”

The characteristics of a CAS therefore precludes any statistical basis for determining the number of repetitions required to satisfy the prescribed confidence limits. However, the observations made by Bak (1999) and the limits of practicality regarding how many simulation runs can be made with the available computational resource can be combined to determine ex

post whether or not the number of repetitions provides sufficient data to

develop a good fit model for the cumulative distribution of disruption events.

Give the length of time it takes to run a single simulation of 1000 days and the number of treatments and blocks, the initial level of replication was set at five. The replication level was tested using the most complex case (maximum diversity and all agent behaviours permitted), and the results were then formally tested against power and exponential cumulative distribution curves.

The results of this experiment to establish whether or not the level of repetition is adequate are summarised in Table 6-4 and Figure 6—3.

Table 6-4: Determination of level of repetition

Model Summary and Parameter Estimates Dependent Variable: days>magnitude

Equation Model Summary Parameter Estimates

R Square F df1 df2 Sig. Constant b1

Power 0.729 4997.436 1 1855 0.000 199729.068 -0.716

Exponential 0.978 81465.309 1 1855 0.000 1564.144 0.000

Figure 6—3: Cumulative distribution of disruption events in determining the level of repetition

Parameters: maximum diversity, maximum range of permitted behaviours and five repetitions.

Examination of Table 6-4 and Figure 6—3 shows the cumulative distribution of disruption events can be described as being exponential with an exponent of 1564.144, which explains 98% of the observed variation. This level of fit was deemed acceptable as it provided a high

level of explaining power and was affordable in terms of the computing power available. As a consequence the number of repetitions was set at five which would require just over 1300 hours of simulation.

6.2 Summary

The experiment design is structured as two experiments which alternately take market structure and permitted behaviour as treatments and blocks. Experiment

1 is configured as a five factor (market structure), two level factorial experiment with eight permitted behaviour blocks. Experiment 2 is configured as a three factor (permitted behaviours), two level factorial experiment with 32 market structure blocks.

Each experiment generates 256 cases, each being described by a unique combination of market structure and permitted behaviours and repeated with 5 different random seeds. These experiments permit data to be collected for 1000 simulated days. Each case is controlled for the nuisance variables of location and price by repeating the simulations with different random number seeds. The following chapter will describe the data collected from these experiments and their subsequent analysis with the intent of directly answering the research questions developed in consideration of the extant literature summarised in Chapter 3.

7 Findings

The previous chapters have addressed the underpinning theories, operationalisation of normal operations, model specification, and experiment design. This chapter will present the results from the experiment through a lens of appropriate analysis which allows answers to be developed regarding the research questions specified in Chapter 2.

The chapter will begin with a generalised description of the results which will be used to frame the subsequent statistical analysis. The statistical analysis will identify cases framed by the experiment design which are statistically different before applying post hoc tests to develop explanations for the differences observed. The chapter concludes by synthesising the results into coherent answers to the questions posed.