Work objectives
3.2. Contributions to the state of the art
This thesis aims at giving solutions to the previous open research problems.
Chapter 4 describes the solutions proposed for the first objective (the devel-opment of a benchmarking methodology for Semantic Web technologies) and chapters 5 and 6 present the solutions related to the second one (the bench-marking of the interoperability of Semantic Web technologies using RDF(S) and OWL as interchange languages).
With regard to the first objective, the thesis presents new advances in the following aspects:
C1. A benchmarking methodology for Semantic Web technologies, grounded on existing benchmarking methodologies and on practices in other areas, as general and open as possible so the methodology can cover the broad range of Semantic Web technologies. This methodology describes the benchmarking process with its sequential tasks, actors, in-puts, and outputs. Such methodology has been validated by checking that it meets the necessary and sufficient conditions that every methodology should satisfy. Moreover, the methodology has been applied with suc-cessful results to different types of Semantic Web technologies in different scenarios and with different evaluation criteria.
The second objective of this thesis is to apply the previous benchmarking methodology with the purpose of benchmarking the interoperability of Semantic
Web technologies. The thesis here presented has contributed to the advance of the current state of the art by providing the following:
C2. A method for benchmarking the interoperability of Semantic Web technologies using RDF(S) and OWL as interchange lan-guages. This method has been defined by instantiating the benchmarking methodology for Semantic Web technologies mentioned above and pro-vides a framework for comparing the interoperability results of different types of Semantic Web tools. The method has been used in the two bench-marking activities performed in this thesis, namely, the RDF(S) and the OWL Interoperability Benchmarkings.
C3. The UPM Framework for Benchmarking Interoperability1 (UPM-FBI) that includes all the resources needed for benchmarking the interop-erability of Semantic Web technologies using RDF(S) and OWL as inter-change languages.
The UPM-FBI offers, as figure 3.1 shows, four benchmark suites that contain ontologies to be used in interoperability evaluations and two ap-proaches for performing interoperability experiments (one manual and an-other automatic), each of them containing different tools that support the execution of the experiments and the analysis of the results.
Figure 3.1: The UPM Framework for Benchmarking Interoperability.
The UPM-FBI provides the following mechanisms:
C3.1. Four benchmark suites for evaluating and benchmarking the interoperability of Semantic Web technologies using RDF(S) and OWL as interchange languages. These benchmark suites have been used as input for the experiments in the UPM-FBI. Three of them were defined to evaluate the interoperability using RDF(S) as the interchange language, namely, the RDF(S) Import Benchmark
1http://knowledgeweb.semanticweb.org/benchmarking_interoperability/
Suite, the RDF(S) Export Benchmark Suite, and the RDF(S) Inter-operability Benchmark Suite; and the fourth one was defined to eval-uate the interoperability using OWL as the interchange language, the OWL Lite Import Benchmark Suite.
C3.2. A manual and an automatic approach for evaluating and benchmarking the interoperability of Semantic Web tech-nologies using RDF(S) and OWL as interchange languages.
In the manual approach, experiments are performed by accessing the tools manually, whereas in the automatic approach, the execution of the experiments and the analysis of the results are performed over the tools automatically.
C3.3. Software tools for evaluating and benchmarking the inter-operability of Semantic Web technologies using RDF(S) and OWL as interchange languages. These software tools are needed to carry out the experiments in the UPM-FBI and have been de-veloped having reusability in mind and, thus, they can be used in other evaluations. Two different tools support the manual approach, namely, the rdfsbs tool, which automates part of the experiments execution, and the IRIBA2 web application, which provides an easy way of analysing the results3. One tool supports the automatic ap-proach, the IBSE4tool, which automates the experiments execution and the analysis of the results.
C4. A clear picture of the interoperability between different types of Semantic Web tools. Benchmarking the interoperability of Semantic Web technologies using RDF(S) and OWL as interchange languages has provided us with detailed information about the current interoperability of the tools participating in the benchmarking activities.
The verification of the goals stated in this thesis has taken place in the following environments:
Most of the work performed in this thesis has been funded by the Knowl-edge Web5European Network of Excellence (FP6-507482). Within Knowl-edge Web, benchmarking had a main role. The candidate was responsible for organizing and leading the benchmarking activities related to the in-teroperability of Semantic Web technologies, defining the experiments to perform, developing the benchmark suites and the software to use, and analysing the results of all the tools. Other participants executed the experiments over the tools participating in the benchmarking.
2http://knowledgeweb.semanticweb.org/iriba/
3This thesis does not deal with the IRIBA application as this is currently being developed
by one undergraduate student.
4http://knowledgeweb.semanticweb.org/benchmarking_interoperability/ibse/
5http://knowledgeweb.semanticweb.org
The CICYT project Infraestructura tecnol´ogica de servicios sem´anticos para la web sem´antica6(TIN2004-02660) has funded the benchmarking of the performance, scalability and interoperability of the ontology develop-ment tool WebODE, with the aim of learning from its current performance and pushing the transfer of this tool to industry. This project has also funded the writing of this thesis.