Accuracy of Stepwise Methods for Increasing Numbers of Nodes

and backward elimination to be maintained, or compromised, as the number of nodes increases. As it is not feasible to run an exhaustive search on large networks, we consider smaller networks, specifically subnetworks of the 15 node ‘safe’ dataset with 6, 8, 10 and 12 nodes (as detailed in Table 3.4). We compared the performance of the stepwise approaches (without the termination instruction) on these smaller networks with the performance on the 15 node networks. Results are shown in Figures 3.7 - 3.9.

Subnetwork Nodes

6 VMPFC, OFC-L, OFC-R, aMCC, DLPFC-L, DLPFC-R

8 Subnetwork 6, Amyg-L, Amyg-R

10 Subnetwork 8, AntIns-L, AntIns-R

12 Subnetwork 10, Post-Ins-L, Post-Ins-R

Table 3.4: Brain regions included in the subnetworks of the 15 node ‘safe’ dataset.

Figure 3.7 shows the accuracy for forward selection and backward elimination, both individually and combined, where, as in Figures 3.2a, 3.3a and 3.5a, accuracy is defined as the number of edges correctly identified as present or absent when the stepwise

networks are compared with the networks found in an exhaustive search. There is a small but noticeable reduction in accuracy as the number of nodes increases: in the 8 node subnetwork, the combined forward selection and backward elimination method reproduced the exhaustive networks with 100 % accuracy for all 32 subjects whereas for the 10 node subnetworks, there were 3 subjects with 2 or 3 incorrect edges. For 12 nodes, the median is still 100 %, falling to 98.6 % for the 15 node network. While this reduction in accuracy is small, it emphasises that we should not assume the accuracies observed with small numbers of nodes will necessarily be maintained for larger networks. As the size of the model space increases exponentially, the number of ‘incorrect’ models also increases exponentially. However, as previously discussed, assessing ‘correctness’ in terms of a single model (the model with the highest Log Predictive Likelihood) may not be the most appropriate method. As we showed in Figures 3.4 and 3.6, the reduction in accuracy can be mitigated by relaxing our definition of correctness to allow models where there is insufficient evidence for a difference i.e. where LPL[P aˆ (r)]−

LPL[P aˆ (r)step] <1.

Figure 3.8 shows that for increasing numbers of nodes, the loge Bayes factor when comparing the ‘best’ model to the second best (i.e. the parent set with the second highest LPL) tends towards lower values. As the size of the model space increases, we might expect the number of models with equivalent evidence to increase also. This is confirmed in Figure 3.9a, which shows the number of ‘equivalent’ models using logeBF <1 and logeBF <3. It is interesting to compare Figure 3.9a with Figure 3.9b, which shows the number of models with equivalent evidence as a percentage of the total number of models (the size of the model space). As the number of nodes increases, the number of equivalent models increases but decreases as a percentage of the size of the model space. This suggests that, within a very large model space, there will be a relatively small number of models which provide a good fit to the data. Further discussion will be provided in Chapter 5.

85 90 95 100 Number of nodes P

ercentage of edges identified correctly

6 8 10 12 15

FS BE FS + BE

Figure 3.7: The accuracy of the stepwise approaches decreases as the number of nodes increases. Boxes show the accuracy of forward selection, backward elimination and combined forward selection, backward elimination for subnetworks of the 15 node resting-state (‘safe’) data. Accuracy is expressed as the percentage of edges correctly identified over the whole network for each subject.

0 10 20 30 40 50 Number of nodes log e Ba y es f actor 6 8 10 12 15

Figure 3.8: The loge Bayes factor for the highest scoring vs. the next highest scoring model decreases as the number of nodes increases. Boxes show the logeBayes

factor comparing the highest scoring with the second highest across all subjects and nodes for subnetworks of the 15 node resting-state (‘safe’) data. For easier visualisation, outliers are not shown.

0 5 10 15 20 25 30 Number of nodes

Number of models with equiv

alent e vidence 6 8 10 12 15 loge BF < 1 loge BF < 3 (a) 0 2 4 6 8 10 12 Number of nodes P

ercentage of the model space

6 8 10 12 15

loge BF < 1

loge BF < 3

(b)

Figure 3.9: The number of models with equivalent evidence increases as the number of nodes increases, but decreases as a percentage of the model space. (a) For subnetworks of the 15 node resting-state (‘safe’) data, we found the number of models with a logeBayes factor of less than 1 (green) and less than 3 (purple) compared to the highest scoring

model, across all subjects and nodes. (b)As (a)but expressed as a percentage of the size of the model space. For easier visualisation, outliers are not shown.

3.5 Discussion

In this chapter, we have exploited the fact that the Log Predictive Likelihood of the MDM-DGM factors by node in order to replace an exhaustive model search with a stepwise one. For a resting-state network with 15 nodes, stepwise regression methods can successfully reproduce the networks estimated by an exhaustive search over the

model space. We have shown that forward selection and backward elimination may be used in isolation, but combining the results of both allows for greater accuracy, as much as 100 % in some cases. The reduction in the size of the model space is so dramatic that these methods may readily be combined without compromising the massive reduction in computation time that they offer.

While these methods may readily be applied to networks of with 50 or even hundreds of nodes, it should be noted that there is no guarantee the performance of the stepwise algorithms would be replicated on larger networks. As mentioned in Chapter 1, section 1.5, for networks with numbers of nodes close to the number of time points require a regularisation term to ensure the stability of the partial correlation matrix. As we showed in the previous chapter, the MDM-DGM tends to detect multiple edges with low connectivity strengths which may be potentially spurious. The following chapter considers how we might introduce a penalty term to ensure more robust edge detection, while potential extensions to the model selection algorithms will be discussed in Chapter 5. For the moment, we conclude that stepwise methods allow fast reconstruction of small networks, and, if interpreted with due caution, may allow the MDM-DGM to be applied to much larger networks than have previously been feasible.