Abstract—Finding an appropriate tool to develop ontology is the first step towards ontology development. A lot of ontology development tools are available in the market, however, some are free and some are commercial. Which tools are mostly used by users? Are there any drawbacks using the tools? And if any, then what? The answers of these questions are highlighted in this paper. These topics are investigated and discussed based on the findings of an online survey concerned with current process of ontology development, the users of the tools, and their domain of working.
Index Terms— Ontology development tools, ontology editors, protégé
I. INTRODUCTION
Ontology development is a complex and largely domain-oriented process that can be benefited from tool support. In the recent years, researchers have developed a lot of tools for developing ontology, for example, protégé [1], SWOOP [2], Top Braid composer [7], OilED [3], WebODE [4], Ontolingua [5], Internet Business Logic [8], OntoTrack [9], and IHMC Cmap Ontology Editor [10].
As defined by N. F. Noy and M. A. Musen [6], ontology development tools allow users to define new concepts, relations and instances. Besides the capability of importing and extending emerging ontologies, development tools may contain some additional features such as graphical browsing, search and constraint checking capabilities.
In order to design and use an effective tool, we need to understand the relationship among users, tools, task and the process. For instance, who are the users of the tools? Why do they need to develop ontology and for which domains? Do they use the currently available tools and if so why do they use them? And, do these tools meet their needs? To answer these questions, a survey ´has been done to get feedback information from the ontology development community and analysing them. To our knowledge, this survey is the first of its kind getting feedback from the ontology development community and analysing them The information gained from this survey should be valuable to both tool and ontology developers.
Manuscript received on February 22, 2010.
M. Rahamatullah Khondoker is with the Integrated Communication Systems, University of Kaiserslautern, Germany (corresponding author to
provide phone: 49-631-205-2643; fax: 49-631-205-3056; email:
Prof. Paul Mueller is with the Integrated Communication Systems, University of Kaiserslautern, Germany (phone: 49-631-205-2263; fax: 49-631-205-3056; email: [email protected])
II. RELATEDWORK
Researchers have been spending their time to build new ontology development tools and trying to evaluate their tools as the best one. As a consequence, enormous numbers of tools are available in the web. Until today, the number of semantic web tools is seventy [11]. In [12], ontology development tools are compared based on certain features such as modeling features/limitations, base language, web support and use, import/export format, graph view, consistency checks, multi-user support, merging, lexical support, and information extraction. Comparing development tools based on user-experience is a scarce attention for the research community.
III. METHODOLOGY
The survey questionnaire was constructed based on four usability components; Tools, Task, Environment, and the User, as suggested in [13], and referenced by [14]. Online surveying was identified as the most suitable method for the research proposed in this paper, on account of its benefits, as identified by [15]: due to the narrow time frame quick delivery and easy return was required.
We reach a large number of ontology developers, spread all over the world through the internet. It was possible to integrate multiple question formats, structured in an elaborate manner by the use of filters, capture data directly in a database, and it enabled data quality checking. Furthermore confidentiality could be ensured easily.
As suggested in [16], the number of questions were limited (taking into account the branches of filtered questions) to reduce the likelihood of abandonment.
A list of questions was compiled, each item belonging to one of the four usability categories stated above. Furthermore a few general questions were compiled, to gain background information on our participants.
A. Pilot Study
As part of the pre-testing phase, some questionnaires were handed out as a softcopy to a small sample of the respondent population; cf. [17]. They were asked to go through the questionnaire, paying attention to wording, consistency, understandability, redundancy, as well as matters of overall appearance. Additionally, every individual question was evaluated with respect to its understandability on a scale from one to five. The questionnaire was refined according to the given feedback.
B. Equipment and Software
The reworked questionnaire was then transferred into Globalpark Enterprise Feedback Suite 5.2 (EFS Survey), a web based software solution by Globalpark GmbH [18] that
Comparing Ontology Development Tools Based
on an Online Survey
provides all functions to organize, implement, and evaluate online questionnaires.
C. Reliability of our survey
After one month of intense preparation work, the questionnaire was ready to be published. The link was sent only to various ontology development forums taking into consideration of getting response from only ontology developers.
D. Motivational quality
After the initial low return rate, several measures were taken to encourage the developers to participate. Some reminder emails were sent to the forums, pointing out the deadline of the survey.
IV. RESULTS
The first question was “which ontology development tool did you try most”. Only single response was allowed for this question. Among 32 respondents, 24 participants use protégé. SWOOP, Internet Business Logic, and Top Braid Composer are used by 2 participants each. Others use Onto track and IHMC Cmap Ontology Editor, each one by a single participant, are shown in fig.1.
[image:2.595.306.548.61.180.2][image:2.595.48.291.340.460.2]
Fig. 1. Number of participated users of different tools.
Fig. 2. Experienced-level of the participated users.
As shown in fig. 2, all experienced level ontology developers ranging from beginners to more than 3 years experienced one participated in the survey.
The next question was task oriented stated, “what is the domain of your ontology?” Open and Multiple responses were allowed in this question. A variety of domains were responded by the participants. Majority of the participants 41.94% (n = 31) use the ontology for Information-system design. Other domains include Biomedical (7), Media (1), Linguistics (2), Business (6), Travel (2), Web services (1), and Logic puzzles (1), Engineering (2), Education (2), Construction (1), Entertainment (1), Government (1) and Homeland Security (1) as shown in the following fig. 3.
Fig. 3. Participants domain of working.
A simple task was given to the developer. The task was, “Create ontology with three classes and two object properties. The Classes are: Person, Book and Library and the object properties are: hasBook (Domain: Person Range: Book) and boughtFrom (Domain: Book Range: Library). Assign an individual such a way that the person named Rahim bought a book named Graph Theory from the Oxford library. The subsequent questions were asked based on this task.
A. Protégé
[image:2.595.305.548.441.558.2]Total 24 protégé users participated in the survey. They have experienced level from less than 6 months to more than 3 years in ontology development using protégé. Among these 24 participants, 7 out of 23 (one participant didn’t show his/her experience) showed experience level between 7 to 11 months. Others have experience level less than 6 months (5), 1-2 years (3), 2-3 years (6), and more than 3 years (2) as shown in fig. 4.
Fig. 4. Experienced-level of protégé users.
[image:2.595.47.290.473.610.2]Fig. 5. Protégé users domain of working.
[image:3.595.48.289.295.420.2]The question was asked about their feeling in developing ontology using protégé with three different criteria: good, interesting and easy. The good attitude showed by the ontology developer using protégé is: Very positive (3), positive (7), neutral (2), negative (4) and very negative (2). So, the majority of the participants showed positive attitudes (10 out of 18).
Fig. 6. Protégé users’ attitude toward developing ontology.
Majority of the participants (10 out of 17) think that developing ontology using protégé is interesting: very positive (4), simple positive (6). Only 4 participants showed negative attitude, they think that it’s simply boring, but not so much 3 participants are neutral in this case as shown in fig. 6. How can you explain the available help for this tool? This question was asked with two different criteria: sufficiency and
user-friendliness. The majority of the participants (10 out of 17) think that available help is sufficient: very positive (3), simply positive (7). Some participants (5 out of 17) think that it’s not so sufficient: negative (4), very negative (1). 2 participants are neutral for this criterion as shown in fig 7.
Fig. 7. Protégé users attitude regarding available help
The majority of the participants (6 out of 16) think that the available help of protégé is not user-friendly: negative (5), very negative (1). In contrast, some people (5 out of 16) think that the available help is user-friendly. In the opinion of 5 participants, the available help is neither user-friendly nor user-unfriendly.
The next question was, “How long did you take to learn this tool as an ontology developer?” Majority of them (11 out of 17) learnt the tool within 1 month. Some participants (4) took from 1 to 2 months to learn protégé. Rest of them were said to
take even more time (2-3 months) to learn protégé.
The majority of the participated (11 out of 15) ontology developers using protégé completed the task 100%. One participant failed to do the task. The rest completed 70%, 80%, 90% sequentially by each participant.
6 out of 14 took only 5 min to complete the task. 3 participants completed the task even shorter time. They took only 3 min. 10, 12, and 15 min were consumed 3, 1 and 1 developer respectively. In average, each participant took 4 min to complete the task.
Majority of the participants (12 out of 14) didn’t encounter any problem to complete the task. The rest faced problem to complete the task. Unfortunately, they did not explain the problem.
[image:3.595.308.547.440.549.2]The next question was, “How can you describe your satisfaction using this tool?” Majority of the polled participants (10 out of 14) showed positive attitudes toward satisfaction using protégé: very positive (5), simply positive (5). Some participants (3) showed negative attitudes: negative (2), very negative (1). 1 participant is neutral in this regard as shown in fig 8.
Fig. 8. Protégé users satisfaction level.
The next question was, “Why do you think so?” to have some feedback from the ontology developers using protégé. These feedbacks are shown in the following table 1.
Table 1. Comments from protégé users. Experience Feedback
Less than 6 months
[image:3.595.49.290.615.745.2] [image:3.595.304.543.635.788.2]because of different versions, old documentation and a website which seems to be also involved in the development. Great supports are given via mailing list. It seems to be the standard tool.
7-11 months 1. I am satisfied because it allows me to quickly create classes and Individuals, and to link them with properties. 2. I am satisfied because it allows me to relatively easily develop ontologies, is popular so it has enough documentation on the web, is extensible. I am
somewhat unsatisfied because I have seen tools that are more intuitive and allow a cleaner overview.
1-2 years 1. Too many windows.
2. I am satisfied because the tool is very easy to use and has a very active mailing list that helps you out with all your questions. It also has some nice tutorials to get you started (even if you know very little about ontologies). It allows me to model almost all the knowledge I need.
2-3 years 1. Unfriendly too makes strange things in the back.
2. I am satisfied because the tool makes individual creation easy. Altogether Protege 4.0 provides an excellent navigation metaphor.
More than 3 years
No feedback
B. SWOOP:
Two participants use ontology development tool named SWOOP. Both of them are 7-11 months experienced in this field. They work in the domain of Information System Design and Construction. Both of them feel good (showed simply positive attitude) to develop ontology using SWOOP. One of them thinks that developing ontology using SWOOP is easy and interesting. Other participant is neutral describing the tool using the criteria easy and interesting.
Both of them think that the available help for SWOOP is sufficient. One of them also thinks that that it is user-friendly, but another participant is neutral in describing the available help for this tool as user-friendliness. Both of them took less than 1 month to learn this tool.
One participant of this tool completed 100% of the assigned task in 15 min. He didn’t face any problem to complete the task. Both of them showed simply positive attitude toward satisfaction of ontology development using SWOOP. In the opinion of one participant, not all functions of this tool are easily accessible.
C. Top Braid Composer:
Two participants develop ontology using Top Braid Composer. Both of them are experienced developer; one of them has been using this tool for 1-2 years and the rest one has been using this tool for more than 3 years. The main domain of their ontology is engineering. One of them also works in the domain of Information System Design, Travel and Business.
One of them feels very good to develop ontology using this tool. He also thinks that this tool is very easy to use and interesting (Very positive). Other participant is neutral in supporting those criteria in favour of this tool: good, interesting and easy.
Both of them think that the available help for this tool is very sufficient (very positive) and user-friendly (positive = 1, very positive = 1).
One of them learnt this tool very promptly less than 1 month. Other participant took 3-4 months to learn this tool.
One participant who is more than 3 years experienced could not proceed. Other participant completed 100% assigned task in 2 min because he didn’t face any problem to complete the task. His comment regarding this tool is, “I like drag and drop in Top Braid composer. Also the fact that everything is on a single screen (in contrast to Protege's tabs)”.
D. Internet Business Logic:
Two participants who develop ontology using Internet usiness Logic participated in the survey. They have been using this tool for more than 3 years. Their domain of working are supply chain management, financial application, billing and much more. Both of them feel very good to develop ontology using this tool (very positive). They also think that developing ontology using this tool is very interesting (very positive) and easy (simply positive).
Both of them think that the available help for this tool is very user-friendly (very positive) but they disagree in evaluating sufficiency. One of them showed very strong attitudes toward sufficiency and other showed simply positive attitude.
Both of them took less than 1 month to learn this tool. One of them failed to proceed. Other completed 100% of the assigned task in 5 min Moreover; he did not face any problem to complete the task. His comment for this tool is, “I am satisfied because the tool combines the semantics of English with the semantics of the ontology, and it provides English explanations of the results of reasoning.”
E. Onto Track:
One participant use Onto Track for developing ontology. He has been using this tool for more than 3 years. His domains of working are Biomedical, media, and business. He feels very good to develop ontology using this tool (very positive). In his opinion, this tool is very easy (very positive) and interesting (simply positive).
He feels good to learn this tool (simply positive) and he thinks that learning this tool is very easy (very positive).
According to him, available help for this tool is sufficient and user-friendly (simply positive). He took less than one month to learn this tool.
feedback and this tool contains adequate graphical representation.
F. IHMC Cmap Ontology Editor:
One participant use IHMC Cmap Ontology Editor for developing ontology. He has been using this tool for 2-3 years. His domains of ontology are information-system design, education and art. He feels simply good to develop ontology using this tool (simply positive). According to him, learning this tool is very easy. He feels very good to learn this tool (very positive). He dropped at this point.
V. DISCUSSION
Though there are enormous ontology development tools available for free on the web, our participants tried only six tools. As shown in the result section, the most dominant and domain-independent tool is protégé which is used by 75% respondent. One reason of such enormous number of developer tends toward protégé could be available online help by mailing list. 55.5% of protégé users feel good to develop ontology and 58.8% think that developing ontology using protégé is interesting.
Ontology development using protégé is easy or difficult? The answer of this question is very difficult to tell by analyzing the response of the participants. (6 out of 17) showed positive attitude: very positive (3), simply positive (3). Similarly, the same number of participants (6 out of 17) showed negative attitude: negative (3), very negative (3). But, 5 participants are neutral in judging this criterion.
Since the majority of the participant thinks that the available help for protégé is sufficient but not user-friendly, the developer of the tool could concentrate on making the sufficient tool user-friendly way.
Protégé can be learnt in 1 month as responded by 64.7% developer. Depending on the extensibility of learning and merits of the user, 2-3 months could be consumed. Developers who need to learn and develop ontology within short time could use protégé.
Based on the result of the given task, a table is constructed as shown in the table 2, where n is denoted as the number of users. 11 protégé users completed 100% task in average 4 min.
Table 2. Task result of various tools users.
Tools proté
gé
SWO OP
TopBraid IBL Onto Track 100% task
completed n = 11
n = 1 n = 1 n = 1 n = 0
Avg time taken (min)
4 15 2 5 2
ACKNOWLEDGMENT
We are very thankful to Prof. Dr. Andreas Breiter who permitted us to use his survey software. Without his help, it would be impossible for us to do this research.
REFERENCES
[1] Protégé. The Prot´eg´e project, http://protege.stanford.edu, (2002)
[2] Kalyanpur, Aditya., Parsia, Bijan., Hendler, James. : A Tool for
Working with Web Ontologies In: Proceedings of the International Journal on Semantic Web and Information Systems, Vol.1, No.1, Jan-Mar (2005)
[3] Bechhofer, S., Horrocks, I., Goble, C., Stevens, R.: OILEd: a
reasonable ontology editor for the semantic web In: KI2001, Joint German/Austrian conference on Artificial Intelligence,
volume LNAI Vol. 2174, pages 396-408, Vienna (2001)
[4] Arpírez, J.C., Corcho, O., Fernández-López, M., Gómez-Pérez, A.:
WebODE: a scalable worbench for ontological engineering. In: KCAP-01, Victoria, Canada (2001)
[5] Farquhar, A., Fikes, R., Rice, J.: The Ontolingua server: a tool for collaborative ontology construction. In: Tenth Knowledge Acquisition for Knowledge-Based Systems Workshop, Banff, Canada (1996).
[6] Noy, N.F., Musen, M.A.: Evaluating ontology-mapping tools:
Requirements and experience In: Proceeding of OntoWeb-SIG3 Workshop, pages 1-14 (2002)
[7] Top Braid Composer. http://www.topbraidcomposer.com
[8] Internet Business Logic.
http://www.semanticweb.org/wiki/Internet_Business_Logic.
[9] Liebig, Thorsen., Noppens, Olaf.: OntoTrack: Fast Browsing and Easy
Editing of Large Ontologies: In: Proceedings of the 2nd International Workshop on Evaluation of Ontologybased Tools (EON-2003) Sanibel Island, Florida, USA (2003)
[10] Hayes, Pat., Eskridge, C. Thomas., Reichherzer, Tomas., Saavedra,
Raul., Mehrotra, Mala., Bobrovnikoff, Dmitri. : COE: Tools for Collaborative Ontology Development and Reuse. In: Knowledge Capture Conference (K-CAP) (2005)
[11] Semantic Web: http://www.semanticweb.org/wiki/Tools
[12] Denny, Michael: Ontology Building: A survey of editing tool
http://www.xml.com/2002/11/06/Ontology_Editor_Survey.html (2002)
[13] Yen, P.-Y., Gorman, P.N.: Usability Testing of a Digital Pen and Paper
System in Nursing Documentation. In: AMIA 2005 Symposium Proceedings, pp. 884-848. (2005)
[14] Bennett, J.: Visual Display Terminals: Usability Issues and Health Concerns. Englewood Cliffs New Jersey: Prentice Hall (1984)
[15] Jansen, K. J., Corley, K. G., Jansen, B. J.: E-Survey Methodology. In: R
A Reynolds, R Woods, J D Backer,: Electronic Surveys and Measurements, pp. 1-8. (2007)
[16] Lumsden, J.: Online-Questionnaire Design Guidelines. In: R A
Reynolds, R Woods, J. D. Backer, Electronic Surveys and Measurements, pp. 44-64. (2007)
[17] Dillman, D. A.: Mail and internet surveys, the tailored design method.
Hobokon, NJ: Wiley. (2007)
[18] Globalpark. (n.d.). Unipark information.
