Data Collection

As was mentioned earlier, sequential data collection was used for the purpose of this research. Data can be collected using different primary and secondary data collection methods. Hox and Boeije (2005) defined primary data as “original data collected for a specific goal” and secondary data as “data originally collected for a different purpose and reused for another research question”. Observation, interviews, and surveys are among the most well-known primary data collection methods. Literature review and official data archives can be considered as the main sources of secondary data (ibid.).

3.10.1 Literature Review

Reviewing prior and relevant literature is one of the first and most important steps in conducting any research project. A literature review builds an understanding of theoretical concepts; it supports the identification of a research topic, gap, and the areas that are beneficial

to research (Rowley and Slack, 2004). Ontology evaluation and selection for reuse is a very complicated task and depends on many different factors. Thus, the following topics were reviewed:

 The general notion of ontology, ontology selection, and ontology reuse.  Different selection systems in the ontology domain.

 The evaluation methods, metrics, and frameworks used in the literature to address different challenges in the ontology selection domain.

3.10.2 First Phase: Qualitative Data Collection

A group of ontologists and knowledge engineers with different levels of expertise and backgrounds in building and reusing ontologies were contacted, and 15 of them accepted to participate in the first phase interviews. Participants in this study were working in domains; four of them had only worked on biomedical ontologies, five had some biomedical experience but had also worked in other fields, such as computer science; the rest of the interviewees were mostly involved in developing ontologies in manufacturing, smart cities, oil, and other non- biomedical domains.

The semi-structured interview protocol focused on how each individual (i) built, (ii) searched for, (iii) evaluated and (iv) reused ontologies. Interviews ranged from 20 to 60 minutes and all, except one of them, were done over Skype as the interviewees were based in different geographical locations. One face-to-face interview was also conducted with an expert in the UK. Interviewees were first informed about the purpose of the study and were asked if they could be recorded. After obtaining consent, interviews were recorded. The interviewer also took field notes during the interviews. Procedures suggested by Creswell and Plano Clark (2007) were followed to analyse the data, starting from data preparation (e.g., transcribing), data exploration, using (QSR International, 2015) for coding, and finally representing and discussing the identified themes or categories.

3.10.3 Second Phase: Quantitative Data Collection

A questionnaire was designed with the total number of 31 questions, broadly divided into four different sections. Each section consisted of different number of questions and aimed to explore and discover the opinion of ontologists and knowledge engineers regarding (1) the process of ontology development, (2) ontology reuse, (3) ontology evaluation and the quality metrics used

in that process, and (4) the role of community in ontology development, evaluation and selection for reuse. Screening questions were used throughout the survey to ensure that respondents are presented with the right set of questions and the answers are valid. None of the participants were discarded from the survey based on their answers to the screening questions, but they were presented with a different set of questions that best suited their previous experiences.

The first screening question was used to discover how often survey respondents build ontologies. This question aimed to make sure that all the respondents were involved in the process of ontology development. The second screening question checked how often respondents consider reusing ontologies. If the respondents had never reused an ontology, they would be presented with a question that would ask them whether they had ever evaluated an ontology. At the end of the survey, five demographic questions were asked to learn about the respondents’ job title, the type of organisation they worked for, how experienced they were, the main domains they had built or reused ontologies in and the primary language they used for ontology development.

The second part of the survey focused on ontology reuse and started by asking respondents about how often (never to always) they consider reusing ontologies. If they chose anything other than never, meaning that they had some experiences of reusing ontologies, they were presented with a list of search and selection systems for ontologies and a 5-point Likert scale, ranging from never to always; they were asked how often they use the suggested search and selection systems to find an ontology for reuse. A comment field was also provided for mentioning the other search engines or repositories participants would use. Respondents were also asked open-ended questions about (1) the main challenges they had faced while searching for a reusable ontology and (2) the best ontology they had reused and why they called it the best? If respondents selected never in answer to the question “How often do you consider reusing existing ontologies?”, they would be asked to share the reasons why.

The most important part of this survey aimed to investigate the process of ontology evaluation and the set of criteria that can be used in that process. In this section, respondents were first asked about the approaches and metrics they tend to consider while evaluating ontologies; this part aimed to explore the respondents’ views and opinions. They were then presented with four different sets of quality metrics, gathered both from the literature and the first phase of the data collection, and were asked how important they thought each of those metrics was, by offering

a 5-point Likert scale, ranging from “Not important” to “Very important”. Those four categories were:

1. Internal aspects of ontologies that can be used in the evaluation process, including their scope, content, and structure.

2. Metadata or different pieces of additional information that can be used for ontology evaluation like documentation, language, accessibility, and frequency of updates. 3. Community related metrics, such as community activeness and responsiveness, and

reputation of the ontology developer team and/or organisation.

4. Popularity related metrics, namely, the number of times an ontology has been reused and the popularity of ontology in the community.

The quality metrics presented in this section were defined and clarified using a brief description or some examples. The reason for proving these descriptions was because the researcher wanted to make sure that respondents knew what each of the metrics exactly meant and referred to. For instance, the metric “language that ontology is built in” had a short example “e.g., OWL”, which would mean that the question is asking about the programming language, and not the natural language that an ontology is built in.

The last section of the survey presented the respondents with some of the interesting statements mentioned in the first phase about the role of community and how it can affect the process of ontology evaluation and selection for reuse. A 5-point Likert scale ranging from “Strongly Disagree” to “Strongly Agree” was used in this section to collect the opinions of ontologists and knowledge engineers on this matter and to measure the level of agreement/disagreement amongst them.

There are many different ways of conducting an online survey, including embedding the survey questions in an email, sending a questionnaire or a survey program as an email attachment or emailing a link to the survey to the respondents (Gunter et al., 2002). In this research, a link to the survey was emailed to different potential respondents. The survey was designed and tested using three different online tools, but the one designed in “Qualtrics”17 _{was chosen at the end,}

as it would provide a better layout and would support different types of questions. There are many different advantages for conducting online surveys, such as, global reach, low cost, convenience, flexibility and ease of analysis (Evans and Mathur, 2005; McPeake, Bateson and

O’Neill, 2014). Global reach and convenience were two of the main reasons for conducting an online survey in this study.

3.10.4 Third Phase: Qualitative Data Collection

The third phase of the data collection aimed to validate the findings of the previous phases. It involved two different experiments. In the first one, the identified metrics were applied to a set of ontologies in the NCBO BioPortal to test if they can predict the number of times ontologies get (re)used. The second one was a user centred experiment (Mandran and Dupuy-Chessa, 2018); eight experts in the ontology domain were interviewed using Skype. This experiment aimed to determine how well the findings of this study can predict the ontology that knowledge engineers would likely select for reuse. This experiment also investigated whether ontologists and knowledge engineers find having information about the factors proposed in this study helpful in the selection process.