In this section we discuss a general quality framework — SEQUAL [80], which is used for this evaluation work. A set of facts from the annotation approach and exemplar
1The quality categories in the quality framework correspond to the semiotic ladder [33]. 129
130 CHAPTER 8. QUALITY EVALUATION OF THE METHOD studies are identified and explained to correspond to the quality categories which are defined in the quality framework.
8.2.1 SEQUAL
SEQUAL (SEmiotic QUALity framework) is the latest version of the general quality framework of models and modeling languages which was earlier reported in [73], [76], and [78]. This quality framework was developed with the objective of providing a systematic structure for evaluating the desirable properties for conceptual models. It has been useful in wider context of understanding quality in areas such as requirements engineering [74], enterprise modeling [78] [115], interactive modeling [77], and evaluating and comparing modeling languages such as UML [75], OWL [91].
Compared with other approaches and frameworks available for evaluating modeling approaches, SEQUAL provides a systematic evaluation method and it has three unique properties:
• It distinguishes between goals and means, i.e. by separating what to achieve from how to achieve it.
• It is closely linked to linguistic and semantic concepts, which particularly includ- ing the discussion on the terms of the semiotic theory — syntax, semantics, and pragmatics.
• It is based on a constructivistic world-view, recognizing that models are usually created as part of a dialogue between the participants involved in modeling, whose knowledge of the modeling domain and potentially the domain itself changes as modeling takes place.
Early version distinguished among three quality categories for conceptual models (syntactic, semantic, and pragmatic) according to steps on the semiotic ladder [33]. The quality goals corresponding to the categories are syntactic correctness, semantic validity and completeness, and comprehension (pragmatic). The framework distin- guishes between goals and means to reach the goals. In the later extensions, more quality categories have been added so that the entire semiotic ladder is included, that is, physical, empirical, syntactic, semantic, perceived semantic, pragmatic, social and organizational quality. The main concepts of the framework and their relationships for quality of models are illustrated in Figure 8.1.
Physical quality is to check how the externalized model M could be persistent and available enabling the audience to make sense of it. Persistence is usually associated with the protection against loss or damage. It also sometime includes previous versions of the model, if these are relevant. Availability is dependent on its externalization and distributability.
Empirical quality focuses on model M itself. Empirical quality deals with the variety of elements distinguished, error frequencies, and ergonomics for CHI (Computer- Human Interaction) for documentation and modeling tools.
Syntactic quality is the correspondence between the model M and the language extension L of the modeling language. Syntactical correctness is the goal of this quality
8.2. SETTINGS FOR THE QUALITY EVALUATION 131
Figure 8.1: SEQUAL framework for discussing quality of models [80]
category, which requires that all statements in the model conform to the syntax and vocabulary of the language.
Semantic quality is the correspondence between the model M and the modeling domain D. The semantic validity and the semantic completeness are goals for this quality.
Perceived semantic quality is the correspondence between the social actors’ interpretation I of a model M and their current knowledge K of the domain D.
Pragmatic quality is the correspondence between the model M and the audiences’ interpretation of it I. Usually social pragmatic quality (to what extent people un- derstand the models) and technical pragmatic quality (to what extend tools can be made that can interpret the models) are differentiated. In addition, the pragmatic quality includes to what extent the participants are able to change the domain after interpreting the model or learning from it. The goal for pragmatic quality is compre- hension, i.e. the degree to which a model can be understood.
Social quality focuses on social actors’ interpretations I. The goal of social quality is agreement among their interpretations.
Organizational quality is the correspondence between the model M and the modeling goals G. Organizational goal validity and organizational goal completeness are related to this quality.
The quality framework of modeling languages keeps the same concepts used in the quality framework of models. However, the core concept is language extension L but not model externalization M anymore. Six quality categories of language quality are
132 CHAPTER 8. QUALITY EVALUATION OF THE METHOD identified as follows.
Figure 8.2: Language quality in the quality framework [80]
• Domain appropriateness indicates whether the modeling language addresses the problems of eliciting/representing relevant facts of the problem domain. Do- main appropriateness is primarily a means to achieve semantic quality.
• Participant appropriateness indicates whether the modeling language corre- sponds to what the participants perceive as a natural way of working, i.e. the modeling language they have known. This is primarily a means to achieve prag- matic quality.
• Modeler appropriateness indicates whether the modeling language assists the modelers in externalizing their knowledge. Modeler appropriateness is primarily a means to achieve semantic quality.
• Comprehensibility appropriateness indicates whether the social actors are able to comprehend the models made in the modeling language. Comprehensi- bility appropriateness is primarily a means to achieve empirical and pragmatic quality.
• Tool appropriateness indicates whether the modeling language lends them- selves to automated tool support or assists in support for reasoning. Tool appro- priateness could be means to achieve syntactic, semantic, and pragmatic quality through formal syntax, mathematical semantics, and operational semantics, re- spectively.
8.2. SETTINGS FOR THE QUALITY EVALUATION 133 • Organizational appropriateness indicates whether the model made in the modeling language achieves the organization’s goals. This is the means to support organizational quality.
Figure 8.2 indicates how these appropriatenesses are related to the concepts in the quality framework. Those categories are related both to the meta-model and to the notation.
8.2.2 The facts corresponding to the quality categories from the ex- emplar studies
In order to apply the general quality framework, we firstly specify the facts corre- sponding to the sets in the quality framework. The quality categories relate to the relationships between the following facts.
Annotation model
In this scenario, the annotation process and the annotation result are evaluation tar- gets. In our approach, the annotation result is an instance of the annotation model. Therefore, the PSAM model is the externalized model in the quality framework. Goals of annotation
The annotation (the PSAM modeling in our approach) is to represent the knowledge stored in the existing process models through a set of agreed semantically-defined con- cepts and formats. Therefore, the goals of annotation depend on the original modeling goals in each case and also depend on the goals of knowledge management. A number of goals are identified from the cases as follows.
• G1 - The annotation should improve the readability of the existing process models. • G2 - The annotation should help sharing process knowledge among different or-
ganizations within a domain.
• G3 - The annotation should help to analyze and validate the existing process models.
• G4 - The annotation should be helpful in the semantic reconciliation of models and to facilitate reuse and integration of models.
Modeling domain
In general, the modeling domain is about processes in the enterprise modeling domain. In this case, it is the SCO (Supply-Chain-Operation) domain.
Language extension
GPO is the meta-model of PSAM and determines the definitions of PSAM. Thus, GPO defines the syntax of the annotation model. Since GPO is created in OWL, a PSAM model is the instance of the OWL model and it has the syntactical features and constraints of OWL.
134 CHAPTER 8. QUALITY EVALUATION OF THE METHOD Modeler — model annotator
In this scenario, model annotators are modelers. They create the annotation by apply- ing their modeling and domain knowledge. In the exemplar studies, we assume that the annotators of PMA know the modeling language BPMN and their BPMN models quite well. The similar assumption for the annotators of PMB1and PMB2is the knowl- edge of the EEML modeling language and EEML models. Moreover, they understand thoroughly their business process and domain definitions. Such a role corresponds to the modeler in the quality framework.
Participant actor — annotation user
Annotation users are the consumers of the annotation results. In the cases, they make use of the annotation information in the process knowledge management activities, such as querying information, analyzing models, and eliciting/inferring interested knowledge. They correspond to the participant actors in the quality framework.
Annotation tool
The annotation tool is used to support the annotation procedure. The annotation tool — Pro-SEAT provides the functions for profile annotation, meta-model annotation, model annotation and goal annotation.