In this section, we highlight the tools and techniques used for showing the consistency and discriminatory property of signatures. These include thresholding, neural computation path analysis, and correlation measures (i.e. Pearson and Spearman correlation coefficients).
5.5.1
Thresholding
Thresholding is one of the key tools that was used in the analysis of the computational signatures. It is a standard technique used in various analyses including the study of data from graph theoretical analyses of the brain [96]. It is valuable in revealing relatively notable features by filtering out some that are below the threshold.
In our work thresholding was done to filter out the values that were below the mean value to reveal information that could be of interest. The mean (µ) was chosen because it is relative to the range of values being considered. Thus, it makes it unlikely to filter out some information that might be important or not filter out signatures that are not useful.
We also used another adjustment parameter (α), which can be used to increase the threshold expressed as in Eq. 5.4.
T = µ + α (5.4)
Here T is the threshold, µ is the mean of the given data (e.g. coexistence likelihood matrix), and α is the adjustment parameter.
5.5.2
Neural Computation Path Analysis
Though computational signatures present a holistic view of the underlying strategies being used for a problem by various models using their meta-features (i.e. likely neural computa- tion paths and their connection strengths), it is hard to examine the most prevalent building blocks common to these models for specific problems. There is a need for a method that combines the results of some of these signatures to present a simplistic and informative representation of the signatures. As such, we use the neural computational paths found in the higher-order signature to reconstruct portions of these strategies. We define neu- ral computational paths as the connections between neurons (i.e. transfer functions) in the
neural network, which transforms data using these paths. Thus, a path between two transfer functions fxand fycould be expressed as ( fx, fy).
This approach involves the use of a graph-based analysis of the neural computational paths from the resulting higher-order signatures (i.e. coexistence likelihood and connection strength). The analysis involved using the neuron (transfer function) with the most connec- tion density from the signatures as a reference for recreating the most likely portion of the strategy by visualizing the transfer functions that were most likely to connect to it. This was reproduced in the form of a graph, with the edges representing the connections, and the intensity of the edges representing the connection strength of the edges. Only connections with the most likelihood to connect to the reference neuron (i.e. with the most connection density) relative to the other transfer functions connecting to it were used in the recreation. This was done by using the average (µ) as the threshold, which allows for a more adaptive form of thresholding; thus, showing only those that were above average.
The steps of the neural computational path analysis were as follows:
• Find the transfer function fterminalthat had the highest connection density. • Add it to a directed graph (di-graph) G.
• Trace back the transfer functions connected to it Tin = { fi... fn}, where Tin refers to
set of transfer functions sending inputs to the terminal transfer function (i.e. Tin).
• Find the mean connection strength for all the transfer functions from the set mconnStrength.
• Remove transfer functions from the set Tin with connection strengths lower than
mconnStrength
• Add the transfer functions to the graph G and connect them with the fterminalnode/vertex. The resulting graph is a reconstruction of the most likely neural computational paths connecting to the neuron with the highest connection density, which gives an empirical expectation of the primary building blocks for the neural computation strategy with the fittest score for the problem. These building blocks further reveal hints of the underlying computational strategy predisposed to the problem.
In the context of showing that the signatures are consistent for problems, and indicating the differences between signatures for different problems, path analysis is especially use- ful. This is because demonstrating that the primary building blocks for the signatures are consistent regardless of changes to factors such as the size of the population used for signa- ture extraction, or the number of independent samples collected (i.e. runs R), conclusively
shows that they are consistent under those conditions. Similarly, if different problems ex- hibit different building blocks, then they can be conclusively shown to be discriminatory for problems.
5.5.3
Measuring Differences between Signatures
Measuring the similarity between signatures was done using correlation measures such as the Pearson correlation coefficient, and the Spearman correlation coefficient built into the Python SciPy library [57]. This is especially useful for evaluating the differences in correlation between signatures and was used to conclusively verified with results from the neural computational path analyses. This has also been used to verify the findings of the discriminatory ability of higher-order problem signatures in conjunction with thresholding.
The Pearson correlation coefficient is expressed as follows:
r=
∑
i∑
j (Aij− E[A])(Bij− E[B]) "∑
i∑
j (Aij−E[A])2 # "∑
i∑
j (Bij− E[B])2 # (5.5)Where A and B are m × n matrices representing higher-order signatures or second-order signatures such as the transfer function likelihood. Expectation of A (E[A]) and B (E[B]) are defined as:
E[A] =
E[A1,1] E[A1,2] E[A1,3] . . . E[A1,n]
E[A2,1] E[A2,2] E[A2,3] . . . E[A2,n]
..
. ... ... . .. ...
E[Am,1] E[Am,2] E[Am,3] . . . E[Am,n]
(5.6) and E[B] =
E[B1,1] E[B1,2] E[B1,3] . . . E[B1,n]
E[B2,2] E[B2,2] E[B2,3] . . . E[B2,n]
..
. ... ... . .. ...
E[Bm,1] E[Bm,2] E[Bm,3] . . . E[Bm,n]
(5.7)
E[Ai j] - the expectation of a neural computation path i j, where i and j are both transfer