Table 12.1 summarizes the characteristics of the 20 test ontologies and match tasks of the contest. Among the ontologies, five are real-world ontologies, BibTex, Food, MIT, UMBC, and INRIA, which were independently developed by different organizations and are available on the web. The remaining ones are generated from the BibTex ontology by systematically modifying some particular ontology features. Except for the Food ontol- ogy (102), which is from a completely different domain, all ontologies are from the Bib- liography domain and represent different classifications of bibliographical references, such as books, journals, articles, etc. The ontologies are of medium size, with around 35 classes and 60 properties. Class instances are also included in several ontologies. How- ever, COMA++ does not have instance-based matchers yet, so that we omit instance sta- tistics in Table 12.1.
The test ontologies are available in the OWL-DL format at the contest website [44]. We imported them into our internal graph representation using an OWL parser of our own development. Classes and class properties, the main elements of an ontology, are repre-
Table 12.1 Characteristics of the test ontologies and match tasks
Task
No Target Ontology Description Type #Classes/Properties #Corresp to BibTex Onto Sim to BibTex
101 BibTex Reference ontology OWL-
DL 35 / 59 91 0.97 102 Food Unrelated ontology 64 / 4 0 0 103 Variations of the refer- ence ontol- ogy BibTex Language generalization 35 / 59 91 0.97 104 Language restriction 35 / 59 91 0.97 201 Random names 36 / 59 91 0.96 202 Random names, no comment 49 / 59 91 0.90 204 Name conventions 35 / 59 91 0.97 205 Name synonyms 36 / 58 91 0.96 206 French translation 37 / 58 91 0.96 221 No class hierarchy 35 / 59 91 0.97 222 Reduced hierarchy 31 / 59 91 0.96 223 Expanded hierarchy 70 / 59 91 0.82 224 No instances 35 / 59 91 0.97 225 No restrictions 35 / 59 33 0.97 228 No properties 35 / 0 75 0.51 230 Flattening 27 / 52 61 0.86
301 MIT Real-world ontologies 15 / 40 48 0.75
302 UMBC 15 / 30 47 0.60
303 Karlsruhe 45 / 69 76 0.41
304 INRIA 39 / 49 91 0.82
sented as nodes in the graph. Relationships between super- and subclasses, between super- and sub-properties, and containment relationships between classes and properties yield structural relationships connecting the nodes with each other. We observe that classes and class properties are unique in the ontologies and there are no shared elements in the sense that one element is a building block of other ones. This is a major difference to the test schemas in our previous evaluations, which are to a large extent built of shared elements and thus require context-dependent matching to deal with.
Table 12.2 Statistics of the test series
Series Tasks Avg Source Nodes Avg Target Nodes Avg Corresp Avg Global 1:1 Corresp Avg Ontology Sim
1xx 3 94 94 91 91 / 100.00% 0.97 2xx 12 94 91.75 84.83 83.83 / 98.86% 0.90 3xx 4 94 77.75 58 47.25 / 79.97% 0.65 123 19 94 89.16 80.16 77.26 / 95.06% 0.86
The ontologies are organized in 20 match tasks, which are in turn grouped into three dis- joint series of tasks, aiming at characterizing the behavior of a match method with regard to different ontology features. All ontologies are to be matched against the reference ontology BibTex. Note that several tasks, in particular, 203, 226, 227, 229, 231, were proposed by the contest, but are not yet specified with test ontologies and real results, so that no experiments with these tasks are possible. We describe the single series in more detail in the following:
12.2.EX P E R I M E N T DE S I G N 1 3 3
• 1xx: This series includes the tests from 101 to 104 performing simple tasks, i.e., com- paring the reference ontology with itself (101), with another irrelevant ontology (102), or with the same ontology in its restriction to OWL-Lite syntax (103, 104).
• 2xx: This series comprises the tests from 201 to 230. In this tests, the reference ontol- ogy Bibtex is matched against a perturbed version of it. In particular, some features of the initial ontology are systematically discarded or modified while leaving the remain- der untouched. As indicated in Table 12.2, the considered features include names, comments, hierarchy, instances, relations, etc.
• 3xx: This series matches the reference ontology against four real-world ontologies of bibliographic references found on the web, in particular, from Massachusetts Institute of Technology (301), University of Maryland (302), Karlsruhe University (303), and INRIA (304).
• 123: In this series, we simply consider all the 20 tests defined in the single series above to estimate the average quality over the whole contest.
All the tasks were manually solved by the contest organizers and the obtained real map- pings are made available on the contest website [44]. They specify the correspondences to be identified between classes and properties of the test ontologies, i.e., between nodes in our graph representation. A few of the real results contains correspondences with semantic relation (e.g., inclusion), which are regarded as similarity correspondences in our evaluation. Table 12.1 also shows the number of required correspondences and the ontology similarity for each match task. Table 12.2 characterizes the series by showing the number of involved match tasks, the average number of source and target nodes indi- cating the search space, the average number of the required correspondences, the average number and ratio of global 1:1 correspondences, and the average ontology similarity. The real result for task 102, which matches two completely unrelated ontologies, does not contain any correspondences. Therefore, we omit it from this table.
In contrast to the match tasks in our previous evaluations, we observe that most of the required correspondences are unique 1:1 match relationships and only few in the 3xx series are involved in m:n match relationships. Like schema similarity, ontology similar- ity was computed by applying the Dice strategy (the ratio of the matching nodes over all nodes in both input ontologies - see Section 7.3) on the real mappings. We observe high similarity, >0.90, between the ontologies in the 1xx14 and 2xx series, indicating that most elements have a matching candidate. This is to be explained with the fact that the target ontology is a slightly modified version of the source ontology. Ontology similarity drops to 0.65 for the 3xx series involving the real-world ontologies. Over the entire con- test, we have to deal with match problems of a moderate size of 94*90 nodes with a high similarity of the input ontologies, 0.86.
12.2 Experiment Design
Like in our previous evaluations, we used COMA++ only in automatic mode in this eval- uation, i.e., we did not consider possible improvements by user feedback or manual
14. The ontology similarity shown in Table 12.1 for task 101 matching BibTex against itself is not 1.0 as expected because the real mapping does not cover all elements of the ontology but excludes some for- eign classes imported from other ontologies.
refinements. The result returned by the automatic match operation was directly com- pared against the provided expected match result to compute four quality measures, Pre- cision, Recall, Fmeasure and Overall. For each series, i.e., 1xx, 2xx, and 3xx, a number of experiments were performed, in each of which a different configuration of the match operation was applied to solve the match tasks of the series. The quality measures were first determined for the single tasks and then averaged over all tasks in the experiment, i.e., average Precision, average Fmeasure, etc. To compare and rank the quality of differ- ent configurations, we mostly use the combined measure Fmeasure. Note that the real result for task 102 does not contain any correspondences, making it impossible to com- pute quality measures for this task. Therefore, we omit it from the quality evaluation and only verbally discuss the result.
Table 12.3 Test configuration for ontology matching
Series Strategy Matcher Similarity Combination
1xx 2xx 3xx
NoContext+Schema 8 single (NamePath excluded) 247 combinations
Average, Both, MaxN(1), Average
Sum: 3 1 255 1
We performed test experiments with all three series, 1xx, 2xx, and 3xx, while the aver- age quality for the 123 series was derived from the quality computed for all match tasks. Based on the insights from the previous evaluations, we focused on the following param- eters most likely to yield the best quality:
• We applied the NoContext match strategy to match complete ontologies, i.e., NoCon- text+Schema. The ontologies do not exhibit shared elements like the schemas in our previous evaluations and we only need to derive node correspondences, making con- text-dependent strategies not necessary. Furthermore, they are of small size so that it is not worth to evaluate with fragment-based strategies.
• We tested with 8 combined matchers, Name, NameType, NameStat, Comment, Children, Leaves, Parents, and Siblings, and all possible combinations of them. NamePath was not considered as we do not have to match paths but nodes, for which NamePath performs exactly like Name. Altogether, we have 8 single matchers and 247 different combina- tions of 2, 3, ..., up to 8 matchers, resulting in 255 alternatives.
• For similarity combination, we employed the default strategies identified in the previ- ous evaluations, in particular, Average, Both, and Average for aggregation, direction, and computing combined similarity, respectively. As all elements, i.e., classes and properties, are unique in an ontology and mostly 1:1 matches are to be identified, we tested with the selection approach MaxN(1) (or Max1 for short) to select the single best match candidate for each element.
Several prototypes participating in the contest did not consider auxiliary information. Hence, to obtain the most objective results for comparison, we also omitted using any kinds of auxiliary information, like synonyms or abbreviations, from our experiments. That is, our quality reported here is based only on comparing names, data types, com- ments, and structures provided by the ontologies. Like in our previous evaluations, all time measurements were performed on a Linux machine equipped with a 2.4 GHz Intel Xeon processor and 1GB RAM.
12.3.QU A L I T Y A N D EX E C U T I O N TI M E 1 3 5