In the following, we first discuss the best quality observed for the single tasks as required by the contest. We then examine how the quality varies between different combinations of matchers. Finally, we discuss the average quality and execution time for the single series.
Task Quality
Figure 12.1 shows the best quality (with the highest Fmeasure value) observed for all 19 tasks of the contest. In 10 tasks, 101, 103, 104, 221, 222, 223, 224, 225, 228, and 230, the target ontology retains many classes and properties of the reference ontology. As name similarity is considered in many of our matchers, we can achieve absolute or nearly abso- lute quality in such cases. The quality decreases in the remaining tasks, which involve name diversity tests or real-world ontologies. In the following, we discuss the quality for the single match tasks.
Figure 12.1 Best task quality
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 101 103 104 201 202 204 205 206 221 222 223 224 225 228 230 301 302 303 304 Precision Recall Fmeasure Overall
• 101: It is trivial to identify all required correspondences. However, the real result does not include foreign classes imported from other ontologies, such as Organization, Per-
son from FOAF15, leading to some wrong correspondences in our match result.
• 102: The input ontologies, BibTex and Food, are from completely different domains. The matchers compute very low similarity between their classes and properties. How- ever, the Max1 strategy still returns some correspondences for this task, which can be easily discarded by applying a low Threshold value, such as 0.5.
• 103 and 104: These tasks test for restrictions enforced by OWL-Lite syntax. Unavail- able constraints are replaced by the more general available or discarded, leading to small changes in the structure of the reference ontology. Like for the 101 task, we achieve absolute Recall, while some correspondences for foreign classes not consid- ered in the real result are also returned.
• 201 and 202: In task 201, all class and property names of BibTex were replaced by random strings. Still our matchers can exploit data type information, structure, and comments, which remain the same between the input ontologies. While missing 25%
of the required correspondences, we could achieve almost absolute Precision. In task 202, all comments were suppressed in addition to scrambling the names. Being left with data type and structure information, we only achieve a moderate quality with a best Fmeasure of 0.55 (Precision 0.67 and Recall 0.47).
• 204: This task tests for different naming conventions, such as uppercasing, using underscores or dashes, etc. Our matchers are robust enough to detect all such changes. In particular, we achieve the same, almost absolute, quality as for matching the same ontologies in the 101 task.
• 205: A large portion of class and property names was replaced by their synonyms. Furthermore, all comments are suppressed. Without the knowledge of synonyms (see Section 12.2, Experiment Design), our matchers only examine data types and ontology structure. The quality is comparable to that of task 201.
• 206: BibTex was matched against its french translation, in which all text fields, such as names and comments, are affected. Apparently, many words are similar between the two languages. We achieve high quality with 0.97 Precision and 0.82 Recall, which is slightly higher than that of task 205.
• 221, 222, and 223: The class hierarchy of BibTex was perturbed by removing all super-subclass relationships (221), removing a large portion of such relationships (222), and adding numerous intermediate classes (223). In all cases, structure is still preserved between classes and their properties. Furthermore, classes and properties still retain their name and comment, leading to a high quality for these tasks.
• 224: All instances were suppressed from BibTex to obtain the target ontology. As our matchers do not exploit instance data, the quality remains the same as for the 101 task. • 225 and 228: BibTex was perturbed by removing locally declared properties (restric- tions) of a class (225) or all class properties (228). However, as class names still remain the same in both cases, we can achieve absolute quality as for the 101 task. • 230: Some classes in BibTex were replaced by their components in the class structure
(e.g., class date by their year, month, day attributes) to obtain the target ontology. This change confuses some of our structural matchers, leading to a small decrease of 3% in Fmeasure compared to the 101 task.
• 301, 302, 303, and 304: In these tasks matching BibTex against four other real-world ontologies, we obtain very promising quality with Fmeasure ranging from 0.76 to 0.96. Furthermore, we observe that match quality correlates well with ontology simi- larity in real-world match tasks. In particular, the best quality is achieved for the 304 task with the highest ontology similarity, while the worst quality is observed for the tasks 302 and 303 involving highly different ontologies.
Impact of Matchers
We now examine how the choice of the matchers affect the match quality. Like in our last evaluations, we determine and analyze the value range, i.e., the minimum, maxi- mum, and average, of the quality achieved by all tested 255 matchers/matcher combina- tions. Figure 12.2 shows the value range of average Precision, average Recall, average Fmeasure, and average Overall for the single series. Despite the wide variation range, the average values are in general close to the maximum, indicating that most matcher combi- nations achieve good quality and only few are outliers with bad quality. An examples for
Figure 12.2 Quality variation of matcher combinations 123 Series -0.1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Preci sion Recall Fmeas ure Ove rall Min Max Avg 3xx Series -0.1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Preci sion Recall Fmeas ure Ove rall Min Max Avg 2xx Series -0.1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Preci sion Recall Fmeas ure Ove rall Min Max Avg 1xx Series -0.1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Prec ision Recal l Fmeas ure Over all Min Max Avg 12.3.QU A L I T Y A N D EX E C U T I O N TI M E 1 3 7
such outliers is the single matcher Parents, which returns high similarity for all children of two matching element.
Next, we compare the quality of the three important matcher combinations, Default, our default combination, All, the most expensive combination, and Name, the least expensive combination, with Best, the matcher combination achieving the best average Fmeasure in a series. As NamePath is equivalent to Name in matching nodes (instead of paths), the Default combination involves Name, Leaves, and Parents, while the All combination origi- nally consists of 7 no-reuse combined matchers (without NamePath - see Section 11.4). As we also tested with the Comment matcher in this evaluation, we determine the quality for each matcher combination first without and then with Comment added. Figure 12.3 shows the average Fmeasure achieved by Best, Default, All, and Name, each with and with- out the Comment matcher added, in the four series.
Figure 12.3 Quality of Best, Default, All, Name without/with Comment added
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Best Defa ult All Name Be st Defa ult All Name Be st Defa ult All Name Be st Defa ult All Name Without COMMENT With COMMENT
1xx Series 2xx Series 3xx Series 123 Series
We found out that Default and All in general perform closely to Best. In the 1xx and 2xx series, the single Name matcher shows the worst quality among the four alternatives because of many homonyms in BibTex (e.g., class Publisher and property publisher of class Reference), and many tasks with name variations in the 2xx series. However, in the 3xx series, Name achieves a comparable quality to Default and All, indicating that consid- ering additional information, such as data type and structure, does not further improve match quality. This is due to two reasons. First, class properties in these real-world ontol- ogies are mostly of type string. Second, the ontologies are structurally quite different to
BibTex, which is indicated by the lower ontology similarity compared to the other series (see Table 12.1).
We observe that adding the Comment matcher to the corresponding matcher combina- tions generally improves their quality in the 1xx and 2xx series. This is because the com- ments, if not suppressed, remain the same between BibTex and its variations. However, the improvement yielded by Comment is only around 5-6%, indicating that we can still achieve high quality for these series without considering comments. On the other hand, Comment is not very helpful for the 3xx series, for which some matcher combinations, such as Best and Default, perform even better without Comment. After a closer examina- tion, we found out that only the MIT and INRIA ontologies provides comments on classes and properties, while UMBC and Karlsruhe do not.
Average Performance
Figure 12.4a shows the best average quality observed for the single series. In particular, we achieve almost absolute quality for the simple tasks of the 1xx series, in which the reference ontology BibTex are changed only slightly by altering or removing OWL-DL- specific constructs to conform to the OWL-Lite syntax. In the 2xx series, BibTex is matched against a systematically perturbed version of it. For these match tasks, the best average Fmeasure observed is 0.90. The 3xx series contains the most challenging tasks of the contest by matching real-world ontologies. Although match quality further decreases as compared to the first two series, we still achieve high quality with an aver- age Fmeasure of 0.80. Over all 19 tasks (i.e., the 123 series), COMA++ achieves the best average quality with average Precision of 0.93, average Recall of 0.85, average Fmea- sure of 0.88, and average Overall of 0.79, which is very promising considering the diver- sity of the match tasks.
Figure 12.4 Quality and execution time for test series
A) Best Average Quality
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 1xx 2xx 3xx 123
Precision Recall Fmeasure Overall
B) Average Execution Time (Seconds) 0 0.5 1 1.5 2 2.5 3 1xx 2xx 3xx 123 Name All
Figure 12.4b shows the average execution time for the single series. Like in our previous evaluations, we also measured the execution time for using only one matcher, Name, and using the combination of all 8 combined matchers, All. As shown in Table 12.2, the aver- age size of the target ontology slightly decreases from the 1xx, to the 2xx, and further to the 3xx series. Accordingly, we also observe some reduction in execution time. In all, the NoContext match strategy performs very fast, requiring at most only 3 seconds on our test machine for the most expensive configuration, i.e., utilizing all 8 matchers to solve the largest match tasks of the 1xx series.
12.4.SU M M A R Y 1 3 9
12.4 Summary
We performed a comprehensive evaluation of COMA++ to match ontologies written in OWL. The test ontologies and real mappings were taken from the EON Ontology Align- ment Contest. Although we could not submit our results to the contest, which took place in November 2004, we strictly followed the rules as published on the contest website to ensure the best comparability with other participants. We did not perform any specific optimization or tuning for ontology matching. The only effort required was for develop- ing the OWL parser. Furthermore, we excluded the use of auxiliary information, such as synonyms and abbreviations, to obtain the most objective results.
Considering the problem size and the similarity of the test schemas/ontologies, the match tasks of the contest are comparable to that of the Small series in our previous evaluations (see Section 11.1). We also observe similar behavior in quality and execution time between the two cases. COMA++ has shown high quality for most tasks of the contest largely using the default configuration identified in the previous evaluations. In particu- lar, we obtained almost absolute quality for the 1xx series, average Fmeasure of about 0.9 and 0.8 for the more complex 2xx and 3xx series, respectively. The best average Fmeasure over all 19 match tasks was 0.88, which is comparable to that of the best con- test participants (see Section 13.4). We also measured the execution time required for all match tasks, which took at most 3 seconds for the largest match task with all matchers involved. The high quality and fast execution time observed again prove the feasibility of our generic solution for different application domains.
In general, we can observe some technical changes between matching common schemas and matching ontologies. While ontologies are represented in the same directed graph representation as schemas, we do not need to pay much attention to shared elements in ontologies. This is because classes and properties in an ontology are by nature unique and their relationships (e.g., super-subclass) do not constrain the instantiation of a class like containment relationships between an element and its subelements in a schema. Hence, context-dependent matching is not necessary for ontologies. With the absence of shared elements, the likelihood for global m:n correspondences is also reduced. Mostly, 1:1 correspondences are required, which can be effectively identified with the Max1 selection strategy as shown by our evaluation.
C
H A P T E RCHAPTER 13
O
THER
E
VALUATIONS
AND
C
OMPARISON
As demonstrated in the last two chapters, schema matching evaluation needs to be care- fully designed and may take a long time to accomplish due to many complex tasks, e.g., determination of the real match results, systematic execution of test experiments, and quality analysis and presentation. So far many evaluations have been published in the lit- erature. While most of them were done for an individual prototype, such as [7, 8, 16, 23, 29, 34, 35, 42, 62, 71, 84, 88, 134], we also observe several efforts to perform a compar- ative evaluation comparing the own approach with that of others [3, 39, 53, 54, 87, 88, 92, 131]. Unfortunately, the evaluations were mostly conducted in diverse ways making it difficult to assess the effectiveness of each single system and to compare their effec- tiveness. Even in comparative evaluations, the results still depend much on the subjectiv- ity of the authors in selecting the match tasks, configuring the single prototypes, and designing a test methodology. So far, the only effort to uniformly compare multiple sys- tems on a benchmark basis was the Ontology Alignment Contest organized in 2004 at the 3rd Evaluation of Ontology-based Tools (EON) Workshop [44, 48].
To obtain a better overview about the current state of the art in evaluating schema match- ing approaches, we review in this chapter various schema matching evaluations pub- lished in the literature. In the next section, we focus on individual evaluations conducted for single prototypes, while comparative evaluations are discussed in Section 13.2. In Section 13.3, we compare several representative evaluations and our evaluations done for COMA++. In Section 13.4, we discuss the evaluations done for the EON Ontology Alignment Contest. Unlike comparative evaluations, which are performed by one author for several prototypes, the contest requests the authors to perform individual evaluations on a uniform test base. In this case, we can largely omit the details on the execution of the single evaluations and focus more on comparing their results. Finally, Section 13.5 summarizes the chapter and points to the issues to be addressed in future evaluations.