The primary concern of Ontology evaluation methodologies is to examine whether require-ments for a given ontology have been met. Five main requirerequire-ments in ontology design and ontology evaluation are presented by Gruber [1995]: i) Clarity means that an ontology should
“communicate the intended meaning of the defined terms”, ii) Coherence means the axioms used in the ontology should be “consistent with the ontology definitions and concepts”, iii) Extensibility refers to an ontology with the possibility of using “a shared vocabulary”, iv) Minimal encoding bias means minimising the conceptualisation at the knowledge level without “depending on a particular symbol-level encoding”, and v) Minimal ontological com-mitment means that an ontology should make “a few claims as possible about the world being modelled, allowing other parties to instantiated if needed”.
In addition, there are other goals in ontology evaluation, such as ontology selection and tracing progress in ontology development. In this section, we present two ontology evalua-tion methods, OntoClean and OntoMetric, that have been influential in ontology evaluaevalua-tion techniques to date.
2.8.1 OntoClean
OntoClean is presented by Guarino and Welty [2002] which is domain independent aiming at simplifying modelling assumption to validate taxonomies. Applying OntoClean results in ontologies to meet requirement of correctness, that is whether the entities of the world being modelled are correctly represented in the ontology. OntoClean introduces a set of meta-properties such as rigidity, identity and unity, which capture various characteristics of classes to assess the correct usage of the subsumption relations determining whether they should be modelled as properties, as a subclass of another concept or even a separate class on its own [Guarino and Welty, 2002]. There are some limitations with OntoClean. One of the constraints is that each class in the ontology needs to be manually classified and tagged with the proposed meta-properties, which is a tedious and time-consuming activity. In addition, the classification task is subject to the ontology engineer’s interpretation of representing concepts in the ontology. Therefore the evaluation results may vary from one engineer to
Ontology Evaluation Methodologies 33
another. Finally, OntoClean only evaluates the correctness of the subsumption relations of an ontology and it fails to validate other aspects of ontology such as completeness and soundness.
2.8.2 OntoMetric
OntoMetric is a criteria-based ontology evaluation methodology presented by Lozano-Tello and G´omez-P´erez [2004] that is used for ontology selection based on a set of characteristics. It differentiates ontologies using an adapted version of the Analytic Hierarchy Process (AHP) method that is a decision-making process based on multiple criteria [Saaty, 1990]. The OntoMetric process is a multi-level tree (or a taxonomy) of characteristics (MTC). The top level of this taxonomy contains five dimensions that are used to construct an ontology including: content, language, methodology, tools and the cost. A set of factors are associated with each dimension in the middle level of the hierarchy. And a set of characteristics obtained from existing work are also associated with each factor at the bottom level. The steps of utilising OntoMetric are:
1. Analyse project aims.
2. Obtain a customised MTC.
3. Weigh up each characteristic against each other.
4. Assign linguistic score for each characteristic of a candidate ontology.
5. Select the most suitable ontology.
However, there are several limitations associated with OntoMetric. First, specifying the customized MTC for ontology selection is subjective to the user’s manual specification and as a result it is inconsistent. Second, the set of characteristics for evaluating content is limited.
In addition, there is a lack of specific measures for scaling linguistic scores and it depends on the user to assign values for the characteristics of the candidate ontology. The third and perhaps the most important drawback of OntoMetric is that this method can be only used for decision-making tasks for the most suitable ontology from a set of candidates ontologies.
Ontology Evaluation Methodologies 34
Figure 2.3: ROMEO Methodology Adapted from [Yu et al., 2009]
Finally, in the case of a new ontology being developed, OntoMetric no longer can be used.
The latter limitation is our main reason not to select this methodology for the ontology evaluation method in this research because we plan to extend the presented model in the future, therefore we seek for an ontology evaluation technique that meet this extendibility requirement.
2.8.3 ROMEO
Requirements Oriented Methodology for Evaluating Ontologies ROMEO presented by Yu et al.
[2009] is a tasked-based ontology evaluation method, that is used for evaluating OSIS.
As illustrated in Figure 2.3, the evaluation tasks of ROMEO starts by establishing a set of ontology roles to clarify the needs of a given ontology. According to Yu et al. [2009], eliciting the roles of an ontology is important to understand how the ontology is used in the context of an application and it also helps to determine a set of appropriate ontology requirements.
Next, these roles are linked to a corresponding set of ontology requirements. Yu et al. [2009]
specify a series of generic requirements, such as “competency”, “capability”, “functionality”
and “standardized”. Ontology requirements must reflect a set of needs from a suitable ontology for the given application. These in turn are mapped to a set of criteria-related questions. The ROMEO approach stipulates that a set of questions is administered for each
Ontologies used in Sustainability and Sustainability Reporting Indicator Sets 35
of the requirements. Such questions explore various aspects of a given requirement providing a deeper understanding of the ontology. In addition, criteria questions have lead to appropriate measures which are critical in an ontology evaluation process. Yu et al. [2009] propose a list of criteria questions for a variety of ontology requirements. Ultimately, measures are then developed to answer these questions, and hence, whether a given requirement has been met.
Various ontology criteria measures have been proposed in the ontology evaluation literature that are summarised in a previous survey conducted by Brank et al. [2005].
We use this approach to evaluate the ontology to develop for sustainability indicator sets.
The details are given in Section 3.1.5.
2.9 Ontologies used in Sustainability and Sustainability Reporting Indicator