Comparisons to Other Methods

Using the simulation framework, the SSS has been assessed under a variety of conditions. These investigations showed that the SSS can reveal plausible networks if the impetus of the data is sufficiently large. This is an important observation, but which is somewhat dimensionless, since the impetus has not been used before in order to specify the dependence of other techniques on it. It would thus be desirable to see how other network inference methods (e.g. those in Table 2.1) perform under similar conditions. This would allow for a direct comparison to the new SSS; however, within the scope of this work not all techniques can be tested, but one has been chosen: cross-correlation [Perkel et al., 1967], which results in comparable computational costs. Details on how this technique has been applied are given next.

Cross-correlation is calculated between two time-series; in order to account for different time-lags, the correlation between any two channels A andB has been evaluated for time-shifted data: shifting B’s data forward (relative toA)

Table 7.1: Quality comparison of networks inferred with the SSS and cross- correlation for different ranges of impetus. The data of the first simulation series (Fig. 7.2a) were analysed with both techniques. Resulting networks were categorised according to the impetus of the data (low, medium, high) before evaluating them; for each category values (recovery rate, precision, P-value) were averaged over all data-lengths and rounded.

recovery-

impetusi rate precision P-valuea SSS cor SSS cor SSS cor low (5%≤i≤20%) 16% 12% 37% 44% 0.14 0.26 medium (25%≤i≤35%) 23% 13% 53% 53% 0.02 0.16 high (75%≤i≤100%) 31% 16% 74% 58% 10−4 _0.09 a _{Note that a comparison regarding precision does not render the comparison of P-}

values superfluous: Precision corresponds to percentage of recovered links that are plausible, but the P-value also takes into account the total number of recovered links. Learning all possible links can yield a moderate precision, if many plausible links exist, but the corresponding P-value one will then indicate the irrelevance of this achievement.

by 1,2, or 3 time-bins. The maximal correlation between channels was then assigned to the corresponding link A → B. Links with maximum correlation equal or above a thresholdαarelearned; all remaining ones are not. In practice, the thresholdαwould have to be chosen by the user, which is not possible for the large number of simulations performed here. Instead, for each analysed data-set, the threshold has been chosen to yield optimal performance: According to the

Neyman-Pearson Lemma (Neyman and Pearson [1933] or [Dayan and Abbott,

2005, pp.119]), no better choice forαexists than the one which yields the highest

likelihood-ratio (i.e. recovery rate / [100 - precision]). The threshold αis set

according to this optimal trade-off and thus yields the best performance that can be reached with this technique.

The cross-correlation has been calculated for the same data the SSS has been applied to in the first series (Fig. 7.2a). Like for the SSS, learned networks were assessed regarding their plausibility. In order to compare both approaches, the average performance of each has been determined for three ranges of impetus (Table 7.1), which show that results of the SSS are generally better. Only in two cases, for low and medium impetus, the precision of the cross-correlation reaches or exceeds that of the SSS.

For practical application it is of interest how the performance of either tech- nique depends on the amount of data that are available for analysis. To show this, the results have been clustered according to the length of the spike trains, but also regarding the impetus of the data (Fig. 7.11). Both more data and

a 0 5 10 15 20 25 30 35

5sec 10sec 30sec 1min 5min 10min

SSS - low SSS - medium SSS - high cor - low cor - medium cor - high recovery rate b 0 10 20 30 40 50 60 70 80 90 100

5sec 10sec 30sec 1min 5min 10min

SSS - low SSS - medium SSS - high cor - low cor - medium cor - high precision c 1e-07 1e-06 1e-05 0.0001 0.001 0.01 0.1 1

5sec 10sec 30sec 1min 5min 10min

SSS - low SSS - medium SSS - high cor - low cor - medium cor - high P-value data length

Fig. 7.11 legend: Quality comparison of networks inferred with the SSS and cross-correlation for data of the first simulation series (Fig. 7.2a). Re- sulting networks were categorised according to the length and impetus of the data before evaluating them; for each category values (recovery rate, preci- sion, P-value) were averaged. (Impetus ranges low, medium, and high de- fined according to table 7.1.) The curves illustrate the performance of both methods for different data-lengths and impetus: a recovery rates, bpreci- sion, and c P-values of precision shown in (b) on logarithmic scale. (See text for interpretation.)

higher impetus generally benefit either technique; however, there are two ex- ceptions concerning the recovery rate of the cross-correlation method: For both low and medium impetuses an increase of data length does not improve but decrease recovery rates. For medium impetus data this seems to be a transient drop, since rates increase as data length reaches the order of several minutes, but no such recovery is observed at low impetus. This might be an effect of spurious correlations, which average out as amount of data exceeds a critical length. The significant improvement in recovery rate at high impetus (between 1 and 5 minutes) supports this hypothesis, which, in order to be tested, would require much longer data-sets; however, these experiments have not been per- formed. In contrast to cross-correlation, the SSS shows a steady improvement in performance over the explored parameter range and it seems to be less affected by spurious correlation in short data-sets.

In conclusion, the SSS yields comparatively good results, especially since cross-correlation was applied with optimal threshold α: Knowledge used in or- der to determine this threshold is generally not available in practical applica- tion; performance of the cross-correlation method is thus likely to be worse than shown here. However, the SSS can successfully compete against this technique under best case performance. The reason for this is the score’s interpretation of the data (Section 5.2.2), which focuses on excitatory relationships, as simulated here. This practically demonstrates the advantage of data-specific adaptation for network inference, as discussed earlier (Chapter 5).

This ends the neural simulation series and their discussion. Corresponding re- sults are now briefly summarised and analysed with respect to their implication on real data analysis.