Qualitative and Quantitative Research Methods

All of the research questions concerning the derivation of an ontology selection and evaluation framework, (i.e., all RQs stemming from the research question RQ1.2 – What typifies and expert- grounded ontology selection and evaluation framework that can support a multi-disciplinary Antarctic science community using Web services ?), have been addressed through a combination of qualitative and quantitative research methods.

Qualitative survey techniques (Denzin, 1978; Dey, 1993; King, 2004; Miles and Huberman, 1994; Jansen, 2010) were applied in this thesis to a sample of expert ontology practitioners, in order to assess the diversity of approach to the practise of ontology selection and evaluation. All experts were, or had been, actively involved in building scientific data exchange infrastructure and had practised ontology re-use. As with all qualitative studies, sampling design, data collection and data analysis techniques had to be matched to the problem domain and the goals of the research (Neuman, 1999; Jansen, 2010).

A Screening Survey, essentially a type of pilot study (Maxwell, 1992) was created as an online survey tool with a mixture of open-ended and closed questions. This Survey was developed to both identify suitable study participants and to generate an understanding of how experts viewed the meaning of some commonly used concepts that would be recur throughout the study. Answers to these types of survey questions gave early valuable insight into some perspectives that informed an expert’s views on ontology selection and evaluation, expressed subsequently during in-depth interviews.

Having identified and recruited suitable study participants via the Screening Survey, each expert was interviewed for approximately one hour. Consideration was given to the extent to which the

interview should be pre-structured. Structured approaches help ensure the comparability of data across sources and are useful in answering questions that deal with differencesbetween things and the explanation for these differences (Maxwell, 2008). They can also target issues of interest. Cognisant of this it was decided that the same standardised, but open-ended questions would be asked in each interview. The purpose of these interviews were: to establish information about the expert’s background and experience; become familiar with the communities with which they were involved and the ontology projects they had, or were currently supporting; understand what methods they had used to select and evaluate ontologies; to elicit opinions they had about these

17

techniques and specifically to identify what evaluation criteria and measures they considered important/less important. Each interview was recorded; transcribed and then analysed through a process of coding and thematic (template) analysis (Crabtree and Miller 1992; Dey, 1993; Miles and Huberman 1994; King, 2004).

As a result of coding and thematic (template) analysis, a three-tiered hierarchical expert-grounded evaluation model was subsequently constructed comprising of five dimensions at the top tier (i.e., ‘Structure’, ‘Functional Relevance’, ‘Usability’, ‘Maintenance’ and ‘Governance’), which decompose into thirteen sub-categories in the second tier and forty-two individual evaluation criteria at the lowest level. This model then became the basis of a quantitative method, that used an AHP-based (Saaty, 1980) pair-wise comparison exercise. In this exercise, a second questionnaire elicited weights from individual experts, for each model element (pair-wise compared). In AHP, each criterion, sub- category and dimension is assigned a rating by an expert, during pair-wise comparison, from a scale of absolute numbers (i.e., 1 to 9). This particular type of numbering scale has been proven in practice and has been validated by physical and decision problem experiments (Saaty 1980, 1994). The resultant individual preferences are then converted into ratio scale weights, framed as matrices of preferences and an Eigenvalue equation operating on these comparison matrices is used to compute estimates of the relative importance of the various model decision criteria (Genest and Zhang, 1996). The AHP process is based on the well-defined mathematical structure of consistent matrices and their associated right eigenvector's ability to generate true or approximate weights (Merkin, 1979) and three axioms (Saaty, 1980). The first axiom, the reciprocal axiom, requires that, if PC(EA,EB) is a paired comparison of elements A and B with respect to their parent, element C, representing how many times more the element A possesses a property than does element B, then PC(EB,EA) = 1/ PC(EA,EB). For example, if A is 5 times larger than B, then B is one fifth as large as A. The second, or homogeneity axiom, states that the elements being compared should not differ by too much, else there will tend to be larger errors in judgment. The third axiom states that judgments about, or the priorities of, the elements in a hierarchy do not depend on lower level elements. This axiom is required for the principle of hierarchic composition to apply (Forman and Gass, 2001).

AHP is a multi-attribute decision analysis support tool and has been applied to a very wide range of decision-based problems. It has also been used previously to evaluate ontologies (Lozano-Tello and Gomez-Perez, 2004). The Ontometric technique, developed by Lozano-Tello (2002), is a formal application of AHP that uses 160 evaluation criteria for ontology assessment. This method inspired the use of AHP in this study, not for its capacity to act as an ontology evaluation method, but

18

because of its potential use as a tool for analysing expert preferences concerning individual ontology evaluation criteria.

Information provided during in-depth interviews was also used to identify practical metrics for performing evaluation criteria assessments and these were linked to criteria in the hierarchical evaluation model. The weighted evaluation model, associated evaluation measures and some suggested methods of application, taken together constitute the expert-grounded selection and evaluation framework delivered in this thesis. Data emerging from expert interviews also provided descriptions of practise relating to methodological issues and matters associated with ontology and community governance. These are discussed and presented in detail in Chapter 7.

A limitation of the qualitative method, as applied in this research, was the relatively small expert population sample size (14 experts initially dropping down to 8 for later facets of the study) that was eventually used. Ideally, a qualitative sample should represent the diversity of the phenomenon (in this case ontology selection and evaluation approaches) under study within the target population (Jansen 2010). Whilst diversity of approach was considered adequately captured by this sample (as evidenced by later triangulation with the literature), due to participant drop-out in later stages of the research, there was little replication of expertise within the scientific disciplines covered during the pair-wise comparison exercise. This lowered confidence in some conclusions reached about patterns found in the preference data (with respect to how experts rated the importance of ontology

evaluation criteria). This limitation is also discussed in detail in Chapter 7.