How good is the empirical validation?

Lack of multiple simulation replications also undermines any attempt to evaluate Baker and Quinn’s results in the light of their empirical data on real organisations. Focussing on the three most developed simulation models, several results in table 1 lie outside the range of values observed in the five real energising and de-energising networks: in particular, “Normalised Indegree” and “Graph centralisation” in the Energising Network, and “Null”, “Asymmetric”, “Negative Mutual” and “Mixed Mutual” in the dyads counts.

Dyad census, which here does not fit very well, is not the most demanding of tests. Counting triads and higher-order network features or motifs is becoming increasingly common in the analysis of empirical network data (Carrington et al, 2005, chapter 10; Milo et al, 2002; 2004). Triad census was described in a standard social network analysis textbook in 1994 (Wassermann & Faust) and there are free tools for calculating them and other motifs (for examples, Siena (Snijders, n.d.) and mfinder

(Alon, n.d.)).

Fitting data on node degrees (numbers of links per node) and network degree centralisation is relatively trivial when one can control a parameter like those used to initialise the model (see Figure 6 for examples). As well as assumptions and choices of function, there are four parameters for populating the initial energy and relational energy values. Another four are used to control the decay rates for energy and relational energy. For comparing output, on the other hand, we have degree and centralisation figures for two types of network, and six dyad types - ten data points if

we treat them as independent of each other (which they are not – for example, there are a constant total number of dyads). With so many parameters and so few outputs, we are entitled to ask how surprising it is that the model achieves the output it does.

0 2 4 6 8 10 12 0 0.2 0.4 0.6 0.8 1 Initial Agent Energy Parameter

R e p li cat io n M ean Average of ENetMean LB UB 0 100 200 300 400 500 600 700 800 900 1000 0 0.2 0.4 0.6 0.8 1 Initial Agent Energy Parameter

R e pl ic a ti on M e a n Average of NullDyad LB UB

(a) Energising Network Indegree (b) Mutual Dyads

Figure 6 Illustrative aggregate statistics of the Baker-Quinn Hierarchy Model

Illustrative aggregate statistics of 20 replications of our reconstruction of the Baker-Quinn Hierarchy Model for a range of different values of the energy initialisation parameter (the mean of a normal distribution). Baker and Quinn’s results were obtained by setting it to 0.5. 95% confidence intervals for the replication means are shown as well. Clearly a wide range of values can be obtained by changing the input parameter. Notice that the mean of “Mutual Dyads” plateaus well below 974, the value cited by Baker & Quinn in their table 1!

Further problems arise when we examine the details of a typical run of the Hierarchy version of the BQ Model. The idea behind this model is that there are now two types of agent, with their type determining the probability of interaction partners being of a particular type. Five of the 50 agents are intended to represent “Managers”. Figure 7

shows the matrix of relational energy after a typical run, with energising and de- energising relations shaded. The system has converged to a state where nearly all relations involving one of the five managers (agents 1 to 5) - either as ego or as alter

or both - are energising. (On the fewer occasions a run converges on a system with mostly de-energising relations, the picture is the same, but with de-energising relations for the managers in place of these energising ones. A key driver in determining which state the system converges on seems to be whether the Managers had predominantly high or low energy at initialisation.) That is, in these model organisations the managers are all wonderful energisers (except for those few organisations where they are the most dismal of de-energisers!) But Cross and Parker (2004b, e.g. p.50) cite cases of real organisations they studied where most of the de- energisers were the managers. Clearly the energised state in Figure 7 is not a good representation of one of these.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 1 0 1 1 1 1 0.8 1 1 1 0.9 1 -0.1 1 0.9 1 0.9 1 1 1 0.9 0.9 1 1 0.8 0.9 1 1 0.8 1 1 1 1 0.8 0.8 1 1 0.9 2 1 0 1 1 1 1 1 1 1 1 1 1 1 -0 1 1 1 -0.2 1 1 1 1 1 1 1 1 1 1 1 1 1 -0 1 1 1 1 1 3 1 1 0 1 0.9 1 0.8 1 0.9 0.8 1 1 1 1 0.9 1 0.8 1 1 0.9 1 0.9 1 0.8 0.9 0.9 1 1 0.9 1 0.9 1 0.9 1 1 1 1 4 1 1 1 0 1 1 1 1 1 0.9 0.9 1 1 0.9 1 0.9 1 1 1 0.9 1 0.9 1 0.9 0.9 1 1 1 1 1 1 1 1 0.9 1 0.9 1 5 1 1 1 1 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 6 0 0 0.8 0.6 0.6 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 -0 0 0 7 0.6 0.2 0.1 0.9 0.7 0 0 0 0 0 0 0 0 -0.2 0 0 0 0.1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 8 0.8 0.8 0.8 0.9 0.8 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0.1 0 0 0 0 0 0 0 9 1 0.3 0.3 0.7 0.9 0 0 0 0 0 0 -0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 10 0.2 0.4 0.5 0.4 0.9 0 0 0 0 0 0 0 0.2 0 0 0 0 0 0 0.2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 11 1 1 1 0.4 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 12 -0.1 1 0.6 0.7 0.9 0 0 0 -0.1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 -0 0 0 0 0 0 0 0 0 0 0 0 0 0 13 0.8 0 0.8 1 1 0 0 0 0 0 0 0 0 0 0 0.3 0 0 0 0 0 0 -0.1 0 0 -0 0 0.1 0 0 0 0 0 0 0 0 0 14 0.3 -0.3 0.5 0.5 0.6 0 -0.1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0.2 0 0 0.2 0 0 0 0 0 15 0.9 1 0.5 0.5 1 0.1 0 0 0 0.1 0 0 0 0 0 0 0 0 -0 0 0 0 0 0 0 0.2 0 0 0 0 0 0 0 0 -0 0 0 16 0.6 0.4 1 0 0.9 0 0 0 0 0 0 0 0.3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0.5 0 0 0 17 0.9 1 0.2 0.9 0.9 0 0 0 0 0.1 0 0 0 0 0 0 0 0 0 0 0 0.1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 18 0.9 -0.2 0.7 0.8 0.9 0 0.3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0.1 0 0 0.1 0 0 0 0 0 19 0.8 0.3 0.4 0.8 0.9 0 0.1 0 0 0 0 0 0 0 -0.1 0 0 0 0 0 0 0 0.1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 20 0.2 0.6 0.3 0.3 0.5 0 0 0 0.3 0.3 0 0 0 0 0 0 0 0 0 0 0 0 0 -0 0 0 0 0 0 0 0 0 0 0 0 0 0 21 0 0.9 0.7 0.7 0.8 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0.1 0.1 22 0.7 1 0.5 0.5 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 23 1 0.7 0.7 0.8 0.7 0 0 0 0 0 0 0 -0.1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0.1 0 0 0 0 0 0 0 24 0 0.8 0.1 0.4 0.8 0 0 0.1 0 0 0 -0.1 0 0 0 0 0 0 0 -0.1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 25 0.3 0.3 0.3 0.2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0.2 0 0 0 0.1 26 1 1 0.3 0.6 0.2 0 0 0 0 0 0 0 -0 0 0.3 0 0 0.1 0 0 0 0 0 0 0 0 0 0 0.3 0 0 0 0 0 0 0 0 27 0.8 1 0.6 0.8 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 28 0 0.6 0.5 1 0.5 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 29 0.8 0.2 0.3 0.8 0.9 0 0 0 0 0 0 0 0 0.1 0 0 0 0.1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 30 0.8 1 0.8 0.8 1 0 0 0.1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 31 1 0 0.3 0.4 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 32 0.4 -0.3 0.6 0.8 0.9 0 0 0 0 0 0 0 0 0 0 0 0 0.1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 33 0 0.9 0.2 0.5 0.3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 34 0 1 0.5 0 0.4 0 0 0 0 0 0 0 0 0 0 0.4 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 35 0.7 0.8 0.6 0.7 0.7 -0.2 0 0 0 0 0 0.3 0 0 -0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 36 0.8 0.2 0.6 0 0.7 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0.1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 37 0.3 0.2 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0.3 0 0 0 0 0 0 0 0 0 0 0 0 0.3 0 0 0 38 0.8 -0.3 1 0.8 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0.1 0 0 0 0 0 0 0 -0.2 0 0 39 0.7 0.8 0.5 0.3 0.8 0 0 0 0 0 0 0 0 0 0 0 0.2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 40 0.7 0.9 0.6 0 0.5 0 0 0 0 0 0 0 0 0 0 0 0.3 0 0 0 0 0 0 0 0 0 0 0 0.2 0.2 0 0 0.2 0 0 0 0 41 0.8 0.6 0.8 0.6 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0.1 0 0 0 0 0 0 0 42 0 1 0.8 0.4 1 0 0 0 0 0 0 0 0 0 0 0 0.3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 43 0.9 1 -0 1 1 0 0 -0 0 0 0 0 0 0 0 0 0.2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 44 0.8 0.8 0 0 0.9 0 0 0 0 0 0 0 0 0 0 0.3 0 -0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 45 0.6 0.8 0 0 0.6 0 0 0 0 0 0 0 0 0 0 0 0 0 0.1 0 0 0 0 0 0.2 0 0 0 0 -0 0 0 0.1 0 0 0 0 46 0 0.9 1 0.8 0.6 0 -0.2 0 0 0 0 0 0 0 0 0.1 1 0 0.3 0 0 0 0 0 0 0 0 0.2 0 0.3 0 0 0 0.3 0 0 0 47 0.9 0.9 0.9 0.6 0.7 0 -0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 -0.6 0 0 0 0 0 0.1 0 48 0.8 0.5 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0.2 0 0 0 0.3 0 0 0 0 0 49 0.9 1 0.6 0.2 0.6 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0.2 0 0 50 0.1 1 0.7 0.9 1 0 0 0 0 0 0 0 0.1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0.1

Figure 7 The matrix of relational energy values

The matrix of relational energy values (“Zi|j”) at the end of a typical run of Baker and Quinn’s Hierarchy model. Cells shaded blue represent energising relations; pink ones represent de-energising ones. As can clearly be seen, the five agents belonging to the top level of the hierarchy (agents 1 to 5) are almost entirely energising and energised.

But a de-energised state will produce the wrong statistics to resemble those of the empirical networks in table 1. So we must ask whether the real organisations studied for table 1 had these miracle managers, and a qualitative divide between worker types. If so, it would be worth mentioning this. If not, then there seems a risk of readers inferring erroneously a degree of empirical validation from the resemblance between the “representative” Hierarchy model statistics and those of the empirical organisations.

The fact that the BQ Model can be relied on to produce a small set of de-energised final states given sufficient simulation runs suggests we should be asking whether there is not an empirical analogue. Baker and Quinn mention this minority of cases - they seem to have run 20 replications for each test of each model (Baker & Quinn, 2007, p.28, 33, 35, 37) – but do not propose any real-world interpretation of it.

Baker and Quinn do not claim the model outputs in table 1 are statistically reliable, or that the empirical data have validated any versions of the model. But there is a risk that the positioning of single-run simulation outputs next to empirical data will create an impression of more empirical validation than is actually there. The paper would benefit from highlighting the fact that the outputs are from single runs and have been specially selected by the authors from at least 20 replications, some of which (the de- energising states) would have looked completely different.