Experiment One

This experiment was designed to assess the usefulness of the quality metrics identified by this research in the process of evaluating and selecting ontologies for reuse. It also helped in

38 _{http://knoesis.org/sweto}

determining the effectiveness of those factors in predicting the reusability of ontologies. The following steps were taken in this experiment: first, a group of 20 ontologies were selected from BioPortal, based on the number of times they had been reused. They were then divided into two groups: (1) those that had never been reused, and (2) those which had previously been reused (11 times or more). Finally, they were evaluated using the metrics identified in this study, with the findings presented below.

 Usage Information (Factor 1). In BioPortal, there is no information on why an ontology has previously been reused. Therefore, it was very difficult to quantify this metric and to determine if there is a relationship between the usage related information provided for each ontology and the number of times it has been reused.

 Documentation, Metadata and Standardization (Factor 2). It was very interesting to see that a link to external documentation was provided for nine (out of 10) ontologies that had previously been reused. However, there was no external documentation or additional information provided for almost all of the ontologies that had not been reused. There was also a very clear link between being a part of a standard, e.g., OBO Foundry, with the number of times an ontology had been reused. Most of the ontologies (8 out of 10) in the first group were not a part of any recognised standard. In contrast, ontologies that were reused more often had mostly followed the principles proposed by OBO Foundry. Moreover, most of the ontologies that had been reused more often had web pages, which had provided different additional information and browsing facilities.  Other Metadata (Factor 3). The comparison between the 10 most reused ontologies

on BioPortal with the 10 least reused ones in this repository did not confirm the link between reusability of an ontology and its development language; only 4 (out of 10) ontologies that were reused more often were developed in OWL, whereas, most of the ontologies in the first group were built using OWL. Usage information in the BioPortal website did not indicate any meaningful relationship between the size of an ontology and the number of times it has been reused. For instance, SNOMED CT41 _{and Ontology}

for Biomedical Investigations (OBI)42 _{as two of the most reused ontology in this}

repository had 347,358 and 3,380 classes, respectively. In contrast, Surgical Secondary

Events43_{, as an example of a small ontology, had not been previously reused by}

BioPortal users.

 Popularity (Factor 4). The results of comparing the acceptance score of the selected sample suggest that 60% of ontologies that were frequently reused had an acceptance score of 80 or above (out of 100). In contrast, the maximum acceptance score for the ontologies in the first group was 26.1. Due to the limitations of the current selection systems, this experiment was unable to measure other aspects of popularity identified by this research, namely, how popular the developer team or organisation responsible for an ontology were.

 Maintenance and Responsiveness (Factor 5). In this experiment, ontologies that had been reused more often were the ones that were frequently updated (e.g. monthly). Moreover, some of the ontologies in this group had mailing lists and/or GitHub pages that users could refer to, in case they had any queries about the ontology or needed any alteration to it. In contrast, it was observed that ontologies in the first group were updated less frequently; some had not been updated since 2013. Contact information, however, was provided for all of the 20 ontologies in this experiment. Overall, it can be concluded that responsiveness of the developer team or organisation responsible for ontologies is likely to affect how they are being reused.

 _{Community, Reputation and Trust (Factor 6). Calculating and quantifying the level} of reputation and trust amongst the community of ontologists and knowledge engineers is hardly possible, as none of the current selection systems for ontologies collect and provide relevant information for these factors. However, by combining popularity related metrics with different information about the organisation that some of the ontologies had been built at, it is likely that a connection exists between the reputation of the team and/or organisation responsible for ontology development and the number of times an ontology gets reused. SNOMED CT, as one of the very well-known and reused ontologies, is developed by NHS and College of American Pathologists (CAP). Overall, it can be concluded that most of the factors identified in this study can be used to judge the reusability of an ontology. They can also be used as a guideline for developing reusable ontologies. In other words, ontologists and knowledge engineers can make the ontologies they

are building more reusable by following and providing different sets of additional information, as suggested by this research. These include documentation, metadata, and also maintenance of ontologies.

Due to the limitations of the current selection systems and lack of data concerning social aspects of ontologies, e.g., community ratings and reviews, measuring some of the proposed metrics, such as popularity and reputation and trust, was not easy. Notwithstanding these limitations, these findings have important implications for the ontology selection systems and the kinds of facilities they should provide for their users.