Research Findings 157

In this section we compare the results of our unaided analysis in chapter 7 with our network aided analysis. By comparing our findings using statistical techniques we analyse:

1. the relationship between important nodes identified by an analyst and a node’s eigenvector centrality ranking;

2. the relationship between the complexity of nodes as ranked by an analyst and a node’s feedback degree ranking.

8.5.1 Findings from Network Aided Analysis

To test whether eigenvector centrality is a good indicator of element importance we firstly categorised the nodes into two groups – those that we believed to be ‘most important’ (See Table 8.1) and the rest we labelled ‘least important’. The ‘most important nodes’ were those that we believe contributed most to creating and sustaining the problematic system according to our unaided analysis. The diagram below (Figure 8.4) provides a reminder of these findings from chapter 7, and thus what elements were considered to be ‘most important’ by the analyst.

Figure 8.4 – Socio-Technical Interactions Identified in Chapter 7

The reader may observe that the elements in Table 8.1 are those that the analyst perceived to contribute most to the vicious circles. For instance, node ‘3 Domestication Responsibility’ supports the domestication failure element and thus is present in Table 8.1. Node ‘6 Practice Culture’ supports the practice culture element and therefore is present in Table 8.1. The same may be said of nodes ‘7 EM_Outcome’ and ‘8 EM_Outcome’, which sustain the EM-PM responsibility conflict. Node ‘9 PM_Outcome’ supports both the practice culture and exemption culture elements represented. Node ‘10 Eng-Outcome’ is a supporter of EDM usability issues and EDM domestication failure represented in Figure 8.4. Nodes 21-26 are supporters of the EM-PM responsibility conflict. Node 28 is a supporter of EDM domestication failure and practice culture. And finally Node 29 is a supporter of the EDM domestication failure and of the exemption culture.

7. EM_Outcome : Encouraging the use of EDM in a standardised manner

8. EM_Outcome : Pursuing an improvements strategy based on process standardisation 9. PM_Outcome : Each PM encourages the use of EDM according to their own practices 10. Eng_Outcome : EDM is perceived to be a source of frustration

21. EM_Responsibility : Implementing change / process improvements

22. PM_Responsibility : Delivering product on time, on budget in accordance to contractual obligations 23. PM_Responsibility : Meeting customer expectations

25. Eng_Responsibility : Design Systems / components

26. Eng_Responsibility : Meeting time, budget, and quality pressures 28. Eng_Outcome : Extent and use of EDM varies on project by project basis

29. PM_Outcome : Each PM implements and follows data management plans according to their own practices

Secondly, having identified the ‘most important’ nodes we generated eigenvector centrality scores and visually inspected the results to explore the relationships. We observed that the eight most highly ranked nodes (in Table 8.2) are members of the ‘most important’ group (Table 8.1). In other words there appeared to be a good correspondence between the ‘most important’ nodes as identified during unaided analysis (See Table 8.1) and those highly ranked according to eigenvector centrality (See Table 8.2).

Table 8.2 - Ranking of Node Importance using Eigenvector Centrality

# Node ID and Description

EV Centrality

1 6. Practice Culture : Tendency to follow norms rather than procedures 1.00E+000

2 28. Eng_Outcome : Extent and use of EDM varies on project by project basis 9.54E-001

3 7. EM_Outcome : Encouraging the use of EDM in a standardised manner 8.51E-001

4 9. PM_Outcome : Each PM encourages the use of EDM according to their own practices 8.35E-001

5 3. Domestication Responsibility : Domestication responsibility has fallen between the cracks 6.19E-001

6 10. Eng_Outcome : EDM is perceived to be a source of frustration 4.38E-001

7 8. EM_Outcome : Pursuing an improvements strategy based on process standardisation 3.53E-001

7 29. PM_Outcome : Each PM implements and follows data management plans according to own practices 3.35E-001

9 30. EM_Objective : Pursue an improvements strategy based on process standardisation and Lean thinking 2.76E-001

10 16. PM_Objective : Meet customer expectations 1.07E-001

11 19. Eng_Objective : Meet programme managers schedule, cost and quality expectations 8.85E-002

12 15. PM_Objective : Meet contractual obligations 8.01E-002

13 31. Eng_Objective : Meet engineering managers process expectations 2.86E-002

14 23. PM_Responsibility : Meeting customer expectations 3.04E-003

15 18. Eng_Objective : Design high quality component or system 1.74E-003

16 22. PM_Responsibility : Delivering product on time, on budget in accordance to contractual obligations 1.29E-003

17 26. Eng_Responsibility : Meeting time, budget, and quality pressures 2.75E-004

17 12. EM_Objective : Improve quality 2.75E-004

17 13. EM_Objective : Improve safety 2.75E-004

17 14. EM_Objective : Improve efficiency 2.75E-004

17 21. EM_Responsibility : Implementing change / process improvements 2.75E-004

17 11. EM_Objective : Improve delivery time 2.75E-004

24 24. PM_Responsibility : Internal reporting 0.00E+000

24 25. Eng_Responsibility : Design Systems / components 0.00E+000

24 1. EDM : Document Management System 0.00E+000

24 20. EM_Responsibility : Improving delivery time, quality, safety, efficiency, repeatability 0.00E+000

24 4. Shared Folders : Document Management System 0.00E+000

24 27. Eng_Responsibility : Following process 0.00E+000

24 2. Matrix Structure : Engineers managed by Programme management & Engineering Manager 0.00E+000

Statistical Test of Hypothesis 1

In order to test hypothesis 1 we performed an Independent-Samples Mann- Whitney U test using PASW. The results of the Mann-Whitney U test indicated to reject the null hypothesis with a p = 0.01. The distribution of eigenvector centrality scores was not the same across the population of ‘most important’ and ‘least important’ elements.

Table 8.3 - Descriptive Statistics of 'Most Important' and 'Least Important'

Statistics

Most Important Least Important

N Valid 13 17 Mean .4146062 .0343126 Median .3530000 .0002750 Std. Deviation .39882271 .07188513 Minimum .00000 .00000 Maximum 1.00000 .27600

Via inspection of each population’s descriptive statistics and histograms (See Table 8.3, Figure 8.5) one can corroborate the findings of the Mann-Whitney U test by observing that the standard deviation, mean, median, minimum and maximum values of the least important and most important populations are very different. The most important population had a mean of .41, a median of .35, a standard deviation of .40 and a minimum of 0 and a maximum of 1. In contrast the least important population had a small standard deviation and a mean and median close to zero with a minimum of 0 and maximum of 0.28.

The corroborated Mann-Whitney U test enables us to conclude that elements with a large eigenvector centrality are more likely to be a member of the ‘most important’ population than the least important population14

. This confirms that ranking elements by eigenvector centrality is a reasonable indicator of the importance of an element in a problematic system. These findings thus indicate that computers may be used to aid the analysis of large problematic socio- technical systems by identifying elements that are most important to sustaining the current problematic system.

Statistical Test of Hypothesis 2

In order to test whether feedback degree is a good indicator of ‘complexity’ (hypothesis 2) we ranked (without the aid of network analysis) the ‘complexity’ of nodes in three subsections of the overall problematic system. The reason we used subsections is because the ranking of every node in the whole problematic system (with over 30 nodes and over 60 interactions) may have been unreliable due to human error. Therefore we opted for three chunks of the problematic system comprising 10 or fewer nodes. The subsections we selected were responsibilities and their interactions (see Figure 8.6 and Table 8.4), outcomes and their interactions (see Figure 8.7) and objectives and their interactions (see Figure 8.8). We selected these sub graphs as a convenience sample since they each had 10 or fewer nodes.

Figure 8.6 - Directed Graph of Responsibilities Table 8.4 - Ranking of Complexity of Responsibility Nodes

Id FD Rank

22. PM_Responsibility : Delivering product on time, on budget in

accordance to contractual obligations 0.4 1 23. PM_Responsibility : Meeting customer expectations 0.3 2 26. Eng_Responsibility : Meeting time, budget, and quality

pressures 0.3 3

20. EM_Responsibility : Improving delivery time, quality, safety,

efficiency, repeatability 0.2 4

25. Eng_Responsibility : Design Systems / components 0.2 4

Our rationale for ranking the complexity of responsibility elements (Figure 8.6) is as follows. One may observe that node 22 is involved in the most complex interactions in comparison to its peers. This may be observed by the fact that it is influenced by three nodes and influences another node. It helps the PM fulfil his responsibility to meet customer expectations (node 23) and is helped by the EM responsibility to improve delivery time, quality (and so on) (node 20) and the engineers responsibility to design components (node 25) and systems and to meet time, budget and quality pressures (node 26).

The second most highly ranked node is node 23 as it is influenced by three others nodes but does not influence any node. This may be observed by the fact that the fulfilment of the responsibility to meet customers expectations is helped by the engineers responsibility to meet time, budget and quality pressures (node 26), the PM’s responsibility to deliver product on time on budget in accordance to contractual obligations (node 22), and engineers responsibility to design systems and components.

The third most highly ranked node is node 26 as it is influenced by one other node and influences two other nodes. The engineer’s responsibility to meet time, budget and quality pressures is helped by EM responsibility to improve delivery time, quality, safety, efficiency, repeatability (node 20). Node 26 helps the PM’s responsibility to delivery product on time and on budget in accordance with contractual obligations, and it also helps the PM’s responsibility to meet customer expectations. The four most complex nodes are node 20 and 25. They both exhibit the least complex behaviour as they are not influenced by any other node but both influence two other nodes. A similar rationale was used for ranking the complexity of nodes in outcome (Figure 8.7) and objectives (Figure 8.8) sub graphs.

Figure 8.8 - Directed Graph of Objectives

In order to test hypothesis 2, we performed Spearman’s correlation test on the data from these three sub graphs using PASW. For all sub graphs tested we were able to detect statistically significant correlations between feedback degree (FD) and complexity as judged by an analyst. For the responsibility sub graph it may be observed that FD has a correlation of -0.973, which is statistically significant at the 0.01 level (Table 8.5). For the outcome sub graph it may be observed that FD has a correlation of -0.997, which is statistically significant at the 0.01 level (See Table 8.6). For the objectives sub graph it may be observed that FD has a correlation of -.975, which is statistically significant at the 0.01 level (See Table 8.7). This enables us to reject the null hypothesis. These findings confirm hypothesis 2 as the correlations are statistically significant account for a significant proportion of the variance e.g. r>0.83. These findings thus indicate that computers may be used to aid the analysis of large problematic socio- technical systems by identifying elements that display the most complex behaviour.

Table 8.5 Spearman's Rho for Responsibility Sub Graph

Correlations

Degree Loops Feedback Degree

Correlation Coefficient -.973** _{. -.973}**

Sig. (2-tailed) .005 . .005

Spearman's rho Rank

N 5 5 5

**. Correlation is significant at the 0.01 level (2-tailed).

Table 8.6 - Spearman's Rho for Objectives Sub Graph

Correlations

Degree Loops Feedback Degree

Correlation Coefficient -.768** _-.701* _-0.997**

Sig. (2-tailed) .010 .024 .010

Spearman's rho Rank

N 10 10 10

**. Correlation is significant at the 0.01 level (2-tailed).

*. Correlation is significant at the 0.05 level (2-tailed).

Table 8.7 - Spearman's Rho for Outcome Sub Graph

Correlations

Degree Loops Feedback Degree

Correlation Coefficient -.866 -0.975** _-0.975**

Sig. (2-tailed) .058 .005 .005

Spearman's rho Rank

N 5 5 5