Design and development

Conceptually, the artefact we aim to develop is composed by two main parts. One aiming the assessment of the semantic interoperability performance and another aiming the determination of its relevance. To achieve the objectives of the first part, we have developed a set of indicators that can be used to assess the present, desired and possible semantic interoperability performance of an organization, in relation to the other organizations involved in the information exchange initiative and based on the common model used to exchange information among them. Then, to achieve the objectives of the second part, we have developed a way to determine the relevance of the information to be exchanged regarding a set of strategic objectives that the information exchange initiative aims to contribute to.

The element which brings both parts together is the common information model, since both are based on it. To obtain the necessary data to calculate the indicators and the relevance of the situation we have also developed a questionnaire, to be filled in by domain experts with technical and functional backgrounds. Finally, to help in calculating many of the components of the framework, we have developed a software application.

On the one hand, indicators are a suitable tool for assessing semantic interoperability, since they are the qualitative and/or quantitative information on an examined phenomenon which enables the analysis of its evolution, checking if quality targets are met, driving actions and decisions (UNI 11097, 2013). On the other hand, semantic interoperability is one of the characteristics of the process to exchange information among organizations. Therefore, the evaluation of the performance of semantic interoperability cannot be disconnected from the evaluation of the performance of information exchange, and so falls in the context of process performance assessment. Consequently, to design and develop the indicators, we have used a specific methodology (Franceschini et al., 2007) for defining and testing process performance indicators, which comprises the five activities depicted in the following figure.

To exemplify the indicators and the determination of the semantic interoperability performance, we have defined an information sharing scenario, which is depicted in the following figure and is described in detail in appendix 1. This scenario comprises the typical elements involved in the determination of the information exchange performance and a series of situations that allow exemplifying the usage of all indicators defined. This scenario is not representative of any real situation; on the contrary, it is quite small (the information model used has 100 times less information elements than the case we use to demonstrate our artefact. Moreover, in the scenario there are only 3 organizations involved, four times less the ones involved in the project used for the demonstration of the framework, and over 100 times less the ones involved at the wider European level, thus allowing us to see that, without an artefact like the one we propose, that supports the determination of the performance and relevance of the semantic interoperability in a semi-automatic way in large-scale initiatives, it would be extremely hard, if not impossible, to do it manually.

The central aspect of the methodology used to define the indicators is the process which performance will be measured. According to the ISO 9000:2000 standard (ISO, 2000), a process is “an integrated system of activities that uses resources to transform inputs into outputs”. Therefore, we have defined a generic process for information exchange among organizations, where semantic interoperability components are put into evidence. As such, the assessment of semantic interoperability is aligned with the overall assessment of the information exchange process and, in future work, other aspects of information exchange, such as the technical, legal and organizational interoperability can also be defined in this process and measured in an analogous way, and hence enable a complete assessment of the performance of information exchange among different organizations.

Figure 10: Process used to define and test the indicators

The relevance of the semantic interoperability performance is determined based on the relevance of the information elements to be exchanged regarding the strategic objectives defined. However, this relevance varies from stakeholder to stakeholder and, therefore, it is important to achieve their consensus in this matter. In this context, we use the questionnaire to obtain the opinion of each of the stakeholders and then use the Weighted Sum Model (WSM) (C. Fishburn, 1967) to calculate the relevance of each of the information elements considered. Then, we use the Delphi method (Linstone, Harold A. Turoff, 1975) to reach the consensus of the stakeholders.

Multi-Criteria Decision Methods (MCDM) aim to try to determine, via various procedures, a ranking of the decision alternatives that is optimal concerning several criteria. However, there are various possibilities, among the most commonly used MCDM (Chen et al., 1992; Hwang, 1987) such as the Weighted Sum Model (WSM), the Analytic Hierarchy Process (AHP), the revised AHP, the Weighted Product Model (WPM), the ELECTRE and TOPSIS. Within these, the WSM (C. Fishburn, 1967) is probably the mostly used (Triantaphyllou, 2000). It is appropriate for single dimensional cases (where all units are the same) and lies on the assumption (verified, in our case) that the total value of each alternative is equal to the sum of the products given; therefore, this was our choice to determine the relevance of the various information elements.

Table 1: Common problem properties which justify employing the Delphi method

Common problem properties

1 The problem does not lend itself to precise analytical techniques but can benefit from subjective judgements on a collective basis;

2 The individuals needed to contribute to the examination of a broad or complex problem have no history of adequate communication and may represent diverse backgrounds concerning experience or expertise;

3 More individuals are needed that can effectively interact in a face-to-face exchange; 4 Time and cost make frequent group meetings infeasible;

5 The efficiency of face-to-face meetings can be increased by a supplemental group Source: Authors based on Franceschini, 2007

Process identification Identification of representation- targets Representation- targets analysis and testing Indicators

communication process;

6 Disagreements among individuals are so severe or politically unpalatable that the communication process must be refereed and/or anonymity assured;

7 The heterogeneity of the participants must be preserved to assure the validity of the results. Regarding the remainder MCDMs, the WPM was developed to overcome some of WSM’s weaknesses by eliminating any units of measure (Triantaphyllou, 2000). However, since this is not an issue in our case, the WPM would introduce complexity unnecessarily; and so, we decided not to use it. The AHP is becoming increasingly popular and the Revised AHP is more consistent than the AHP (Triantaphyllou, 2000); however, since our criteria are not hierarchical, none of these methods is applicable; hence, we have also not decided to use them. Finally, considering the large amount of decision alternatives expected - information models of large-scale information exchange initiatives typically involve large information models - methods based on a pairwise comparison (which demand high user intervention) such as the ELECTRE and TOPSIS are not feasible because respondent fatigue limits the number of alternatives that can be ranked (Bradburn et al., 2004); hence, we have also decided not to use any of them.

The Delphi method is used for structuring a group communication process, so that it is effective in allowing a group of individuals, as a whole, to deal with a complex problem (Linstone, Harold A. Turoff, 1975). More specifically, it is appropriate for situations that, like large-scale interagency information exchange initiatives, face the problems indicated in the table above. In practice, it consists of collecting information, usually in the form of a questionnaire, from a set of experts, processing the results, iterating with experts as many times as necessary, providing the consolidated results and allowing the experts to change their evaluations (information provided on a certain topic), until consensus is reached.

The common information model used to exchange information is a key component of our artefact, and we should not confuse an information model with a data model, especially since often these concepts are used interchargeably in the literature. The main purpose of an Information Model (IM) (Pras & Schoenwaelder, 2003) is to model managed objects at a conceptual level, independent of any specific implementations. Data Models (DM) (Pras & Schoenwaelder, 2003), on the other hand, are defined at a lower level of abstraction, include many details, and are intended for implementers. Consequently, multiple DMs can be derived from a single IM.

The baseline of the questionnaire is generic and is composed by three main parts. The first is the set information elements from the common model used to exchange them. The second is the set of organizational objectives to be achieved with the exchange of information, and the third is the set of options used to classify the relevance of each information element to each objective (relevance scale). To use the questionnaire, domain experts must be involved to instantiate each of these parts with specific domain elements. Then the questionnaire is made available electronically to them, which return it dully filled when ready. The questionnaire should be designed using common best-practices for questionnaires (Bradburn et al., 2004), such as ethical principles and closed-answer formats.