Tool Evaluation for Accuracy

The developed tool can be used to overcome the challenges of textual descriptions of OpenSim function interrelationships. However, no matter how usable or simple the tool is, it has to accurately depict the true nature and the significance of the functions. Therefore, the complex network topology of OpenSim functions was analysed for its accuracy with suitable statistical measures.

5.4.1 Evaluation Using Eigen Vector Centrality

It was decided to use Eigenvector Centrality [216], which is defined as the principal eigenvector of the adjacency matrix defining the network, as the centrality measure to examine the network interrelationships. It has been successfully used for similar purposes in previous studies for evaluating the node interrelationships in a complex network [217].

108

The Eigenvector (EV) metric has two important properties: first, it captures the fact that a node (a function) that influences many other nodes is influential (has a higher value). Secondly, the property that makes the EV metric unique: the EV centrality measure considers that a node that affects many highly influential nodes is more influential than a node which affects many weakly influential nodes [217]. EV measure follows the approach that all nodes influence their neighbours, without necessarily being confined to the shortest path node connectivity [218]. This is an appropriate measure as our functional network has multi-path interlinks that must be accounted for in an accurate analysis.

The built-in analyser in Gephi was used to obtain the Eigen Vector Centrality (EVC) values for each OpenSim function. The functions were then ranked using these EVC values and assigned the corresponding EV Rank in the ordered list of the functions.

In parallel to this analysis, the list of the OpenSim functions used in the network topology were grouped into 5 categories of ordinal measures (with encoding values) as Very High = 5, High = 4, Moderate = 3, Low = 2 and Very Low = 1 based on the Perceived Level of Significance (PLoS). The PLoS category of each function was obtained as a subjective measure from an expert analyst and further viewed by another analyst in order to eliminate significant anomalies. The grouping of functions on PLoS was performed in mutual exclusion with the EVC values so that the encoded PLoS values for the functions were treated as independent measures based on system studies and expert views.

The two data sets were treated as independent samples of data. A comparison of EVC and PLoS scores for a selected set of 20 functions is shown in Table 5.4, (ordered according to EV Rank).

Spearman’s correlation coefficient between EVC and PLoS was 0.847, (N=211). It indicates a strong positive correlation that explains over 70% of the variance in the values tested, with significance (p<0.01).

EV Rank Function EVC PLoS

1 Content Management 1.000 Very High (5)

2 Teleport 0.844 Very High (5)

3 Avatar Activity 0.768 Very High (5) 4 Avatar Terraform 0.695 Very High (5) 5 Create Content Objects 0.666 Very High (5) 6 Permission Settings on an Object 0.619 Very High (5)

7 Avatar Fly 0.496 High (4)

8 Edit Content Objects 0.451 Very High (5) 9 Allow Create Objects 0.443 Very High (5) 15 Manage parcel access list 0.370 Very High (5) 18 Near field spatial management 0.331 Very High (5) 19 Region Estate Management 0.330 Very High (5) 34 Bypass Permissions 0.289 High (4) 37 Group Management 0.286 High (4) 38 Force permission ON/OFF 0.285 High (4) 87 Allow Script Run 0.209 Moderate (3) 164 Alert massages to users 0.031 Low (2) 170 Set Parcel Name 0.027 Low (2) 180 Set Music URL 0.024 Low (2) 205 Group visibility change 0.003 Very Low (1)

109

Although the PLoS categories were derived in 5 scale ordinal distribution to increase the accuracy in capturing subjective scores from human analysts, in practice Eigen values are more effective in indicating two contrasting samples: highly important and less important items. Therefore, to evaluate the accurate representation of the functional importance in the network topology the PLoS values were further categorised into two sub samples as Importance-High PLoS (≥ 3) and Importance-Low PLoS (< 3). The analysis could have even been done by selecting only the two PLoS samples at the extreme ends (i.e., Very High -5 and Very Low -1); but the entire sample of functions was used to increase the analysis accuracy where the entire topology is accounted for the analysis.

The corresponding EVC measures were grouped into two sets in accordance with the two PLoS samples derived. The descriptive statistics of the EVC values in the two PLoS samples (from PSAW), are shown in Table 5.5. The descriptive statistics show some indication that the high PLoS functions also show a relatively higher EVC values when comparing the means of the two samples. However, this was tested statistically for the accuracy of the topology. For that, the following examinable hypotheses were used (as required by the test statistics).

Null Hypothesis: There is no difference between the distribution of corresponding Eigenvector Centrality (EVC) values, across the Importance-High and Importance-Low PLoS categories

Alternative Hypothesis: There is a difference of the distribution of Corresponding EVC values across the Importance-High and Importance-Low PLoS categories.

Importance_High (PLoS) Importance_Low (PLoS)

N (number of functions) 137 74 Mean .221 .092 Std. Error of Mean .0138 .0027 Median .158 .085 Std. Deviation .1612 .0231 Minimum .0000 .0000 Maximum 1.0000 .2013

Table 5.5: Descriptive Statistics for the two groups of PLoS

Independent Samples Kruskal-Wallis test was employed using PASW (18.0). The result, (Fig. 5.7) suggests rejection of the Null Hypothesis with significance (p<0.05). Having rejected the Null Hypothesis, we can say that high EVC values are more probable to be a member of the high important PLoS population, than the low important population of PLoS. This indicates the accurateness of EVC values to represent the importance of OpenSim functions with their interrelationship behaviours.

110

Figure 5.7: Hypothesis test output (PASW)

The EVC values can also be used as a measure to prioritise different OpenSim management considerations. It can be a significant challenge for a non-expert to decide which functions to focus on or prioritise, let alone figuring out the complex interrelationships between those functions. EVC values can help users to identify critical functions, which have a high influential nature, so that a reasonable consideration can be made for managing those. The study findings help users to rely on a statistically verified ranking mechanism, in case of ambiguity or doubtfulness on which functionalities should be prioritised for their policy implementations.

5.4.2 Evaluation Using Betweenness Centrality

EVC values do not take into account the holistic nature of the function network. That is, the analysis using EVC considers the functional significance with interrelationships, but not how the functions are actually interconnected in the structure. For example, a user may consider high EVC functions with high priority to manage, but may not have a clear idea on how the functions are implemented in functional categories and the structural interconnections irrespective of the importance; this can result in difficulties in locating the function’s ego network and the relative position. This is a well-known property in complex networks where EVC values cannot be used. Betweenness Centrality (BWC) [219] was considered as a measure to evaluate the network accuracy in depicting the structural interconnections of the functions. The built-in analyser in Gephi provides BWC measures as well. Table 5.6 shows a sample of 15 functions with their BWC values selected to represent the range of the distribution.

The Pearson correlation coefficient for EVC and BWC R = 0.179 (R2_{=0.032) indicates a negligible} relationship on variances as the two measures are effectively representing different properties of the function network. This indicates the use of BWC to complement the previous analysisbasedon EVC. A ranking mechanism for BWC was not used. BVC values do not provide a meaningful metaphor in the 3D learning environment context since they merely depend upon the number of functional linkages and relative positions among each other. In fact, it would be an imprecise notion to categorise BWC values. Therefore, it was treated as a qualitative study on function architecture; it was performed through observations.

111

Rank Function Betweenness Centrality

1 Group Management 912.30

2 Near Field Spatial Management 784.23 3 Region Estate Management 721.83

4 Avatar Activity 642.14

5 Content Management 611.87

7 Avatar Behaviour 529.38

10 Region Estate - Estate Management 386.09 12 Near Field Spatial Management- General 331.07 15 Region Estate Debug Management 202.68 21 Group Parcel Access Management 151.31 50 Group Object Management 63.46

100 Parcel set passes 11.03

147 Set Landing Point 3.49

195 Invite users 0.00

207 HUD attachment 0.00

Table 5.6: A selected sample of Betweenness Centrality measures of functions

If we consider the top BWC values, they represent the major areas of OpenSim functions: Group Management, Near Field Spatial Management, etc. The mid-ranged BWC values associate with Group Object Management, Group Parcel Access Management, etc. The smaller BWC values represent: HUD attachment, Invite Users, etc. The functions with high BWC values represent the common categories (sub communities) of functions. The mid-range BWC values represent sub categories of functions that cluster several derived solo functions; these are within the major communities, however. Finally, the individual functions such as HUD attachment, Instant Messaging, Invite users, etc. represent the perimeter functions in the functional network; with low values of BWC. Therefore, the distribution of BWC indicates the hierarchical relationship among functions, irrespective of their importance. As a result, users can benefit from the visual representation of this function network for their management needs. Moreover, this validates our approach of depicting the OpenSim functions as a network, since the network properties match with the actual function implementations and their relationships.

5.4.3 Analysis Overview

The statistical and qualitative analysis of the user guidance tool suggests that the required guidance on OpenSim functions can be given by portraying the functions as a network of functional interrelationships. The analyst ranking mechanism used in PLoS was a subjective measure in view of the operational significance of the considered function. Human analysts can have slightly different ranking opinions on functions that give ties on importance as per their observed behaviour, which can affect the analysis. However, these errors are distributed between the two groups: Importance High and Low.

112

The management policies mainly depend on the academic preference and contextual properties of the learning activity; these contextual policies on managing learning activities in OpenSim can be supported using the tool. The guidance network topology is prepared ‘as is’ with respect to the OpenSim system implementations. A given OpenSim function can be used to implement multiple policies [one to many association: i.e., 1:*] depending on the contextual norms of learning in these MULEs, which depends on user preference. In that respect the function network tool contribution appears to be multidisciplinary in nature; it can probably be used in most of the domains that use OpenSim; however, further studies are recommended before directly using it in a context other than MULEs with OpenSim .