As described in Section 3.1.5, among several ontology evaluation approaches, we choose ROMEO to evaluate OSIS. Our main motivation for choosing this method is that ROMEO focuses on the ontology evaluation requirements as various tasks. In other words, it breaks down a complex process into approachable and easy-implemented steps. In what follows, we step through ROMEO methodology containing five steps. The process starts by defining several frames of reference.
— 1st Level 2nd Level 3rd Level Indicators
GRI: 6 33 - 78
OECD: 5 16 28 47
Table 4.3: Number of Sub-category and Indicators per Category for GRI and OECD Taxonomies
4.4.1 Frames of Reference
A frame of reference describes documents in a requirements specification, controlled vo-cabulary or structured dataset. Yu et al. [2009] define a frame of reference F as, F =
< Fc, Fi, Fr >, where Fc is a set of concepts, Fi is a set of instances and Fr is a set of relationships, which is the union between the set of relations between concepts Fcr and the set of relations between instances Fir. Here we interpret “frame of reference” to be the sorts of knowledge sources solicited during the Knowledge Acquisition activity – the indicator systems themselves.
Since OSIS is designed to support two key requirements of intuitiveness and reusability, we choose three indicator sets to evaluate the ontology design candidates against each other.
The first two – the GRI and the OECD given in Section 3.1.2 – are the sources used to construct the initial design, which are referred as the “seen” frames of reference. The third one – taken from the UN – is used to evaluate the ontology candidates of OSIS, which is described as the “unseen” frame of reference since neither candidate has any explicit entities drawn from it.
Each system reflects subtle yet distinctive features of how sustainability is conceptualised by their respective organisations. Hence using 2+1 frames of reference allows us to triangulate
OSIS Post-Design Process: Ontology Evaluation 87
the key domain concepts to at least the first degree of approximation. Table 4.3 shows the number of indicators and nodes per category used in these organisations.
4.4.2 Establishing the Ontology Role
In defining the role of OSIS, we investigate how this ontology will be used in a sustainabil-ity reporting framework in future. One of fundamental aspects in sustainabilsustainabil-ity reporting practices is to suggest relevant indicators for a particular issue. We define the role of OSIS to enhance the suggestion process of indicators by solving representation issues of semantic heterogeneity of indicator sets discussed previously. The role of OSIS is shown in Table 4.4.
Application: An integration framework for systematic reporting on sus-tainability indicators.
Ontology: A formal ontology which contains several concepts and richly defined relations in the context of sustainability.
Role: Enhance effectiveness of query expansion module in suggest-ing indicators for query tasks by solvsuggest-ing representation is-sues of semantic heterogeneity of indicator sets
Table 4.4: OSIS Role
4.4.3 Ontology Requirements
Ontology requirements reflect a specific competency or quality of the ontology that can be obtained from existing ontology requirements or application requirements. For that, we use the same requirements defined during the Specification activity given in Table 4.1, which are also associated with the ontology role and purpose described in the previous section. As shown in Table 4.5, the first requirement is that the ontology candidate provides a precise and intuitive representation for the indicator systems. The second requirement is that its reusability allows other indicator systems to apply it easily and extendedly. Other conditions required to be addressed in this phase are also described in Table 4.5.
4.4.4 Criteria Questions
We specify two questions for the OSIS ontology candidates and ensure each question is an-swered with respect to both the seen (GRI and OECD) and unseen (UN) frames of reference.
OSIS Post-Design Process: Ontology Evaluation 88
Requirement OR1: Does the ontology provide a precise and intuitive represen-tation of the indicator systems it represents?
Requirement OR2: Does the ontology allow for other indicator systems to be easily incorporated using existing concepts, properties and relations?
Analyze: OSIS
For the purpose of: Querying an ontology
With respect to: Having rich definition of concepts and correctness of rela-tions between concepts
From the viewpoint of: Expert Panel of Social Scientists in Sustainability In the context of: Sustainability Indicators
Motivation: A suitable ontology for sustainability indicators sets (OSIS) needs to make correct suggestions of indicators to the user queries. To address this goal, adequate level of coverage and correctness of the components of the ontology (concepts, relations and instances) are essential.
Table 4.5: OSIS Requirements 1 and 2
The criteria questions are listed below.
1. Do the ontology components (concepts, instances and relations) adequately cover the terms of the given domain?
2. Do the ontology components capture the terms of the given domain correctly?
The first question examines the coverage criteria and the second question determines the correctness feature of the ontology. In ROMEO [Yu et al., 2009] correctness is measured whether or not the right concepts, relations and instances have been modelled correctly according to the frame of reference. Similarly, coverage is assessed by whether or not the ontology sufficiently captures key concepts in the given domain.
4.4.5 Measures
In the final stage, Yu et al. [2009] suggested adopting a set of measures that are compatible with the ontology requirements which allowed us to answer the criteria questions. We adopt precision to measure to the correctness criterion, by determining the percentage of overlap-ping terms in an ontology O that overlaps with the set of terms from a frame of reference F (Equation 4.1). Additionally, recall is used to measure the coverage criterion, referring to
Experiments 89
the percentage of overlap between a set of terms from the ontology and the frame of reference (Equation 4.2).
precision(O, F ) = |F ∩ O|
|O| (4.1)
recall(O, F ) = |F ∩ O|
|F | (4.2)
We also apply the F -measure to ontology evaluation in the context of indicating appro-priate coverage of concepts in the relevant frame of reference (Equation 4.3).
F-measure(O,F) = 2
1
recall(O,F ) +precision(O,F )1
(4.3) A discussion of the origin of above measures is provided in Section 3.1.5.