Discussion on Results of the Validation

9.4.1 Automatic vs. manual annotation

During the validation, we found that the results of the SWRL queries for RE1 (Navigation requirements) can display the most information of the models through the annotation, but still some semantics of original models are found miss-ing or incomplete in the annotation models. The results of QRule-Activity-subActivity are the same as the original models, i.e. it is semantic complete-ness of the Activity composition for the annotation. Such completecomplete-ness is guaran-teed by Pro-SEAT’s automatic transformation from Metis models to PSAM models

9.4. DISCUSSION ON RESULTS OF THE VALIDATION 159 based on the meta-model annotation. However, the results of QRule-Activity-hasPrecedingActivities, QRule-Activity-hasSucceedingActivities, QRule-Activity-hasPrecondition and QRule-Activity-hasPostcondition are not com-plete because current Pro-SEAT does not support the automatic annotation of the sequence of Activities. We have to manually annotate such information.

When checking the results of Activity-hasArtifact and QRule-Activity-hasActor, it turns out that the automatic annotation of associating Artifact or Actor-role with Activity performs better on EEML models than on BPMN models.

The reason is that EEML Resource Role (GPO:Artifact/Actor-role) is encapsulated in EEML Task (GPO:Activity) in Metis which behaves in the same way as GPO. How-ever, the BPMN Logic Process (GPO:Activity) is encapsulated in BPMN Swimlane (GPO:Actor-role) but not in the reverse way. The relations can not be automatically transformed as has_Actor-role properties in PSAM models. Based on the above anal-ysis, we can conclude that the function of automatic transformation should be improved in Pro-SEAT. RE1 (Navigation requirements) is almost fulfilled in spite of the missing annotation caused by the manual annotation.

9.4.2 Model analysis based on semantic relationships

The reference ontology is introduced in the SWRL queries for RE2 (Search require-ments). The links between the model fragments and the ontology concepts are built through the semantic annotation. When executing the SWRL queries for RE2 (Search requirements) on the three model instances respectively, we found that the synonym (same_as) (i.e. semantic equivalent) relationship is mostly used in annotating Ar-tifacts and Actor-roles, while the hypernym (kind_of) and meronym (part_of, member_of) relationships are rarely used. Nevertheless, for ontology-based annota-tions of Activities the case is just reverse: more meronym (phase_of) relaannota-tionships and hypernym than synonym are applied. Such phenomena is observed in all the three models. It shows that Artifacts and Actor-roles in the three models are not very specialized but relatively general and close to the SCOR standard. On the contrary, Activities in different models are quite various and meanwhile the modeling granularity of the Activity is smaller than the SCOR process elements.

9.4.3 Detecting missing annotation

In order to detect missing annotations, we run the query rules ² together with cor-responding inference rules ³. For instance, when only running QRule-Activity-hasArtifact on PMB1, 19 records are returned. If the query is run together with IRule-Activity-subActivity-hasArtifact, the results set consists of 30 records. By running Input-hasArtifact and IRule-Activity-Output-hasArtifact with QRule-Activity-hasArtifact the query returns 40 records. The results of the execution of those inference rules and queries on three models are listed in Table 9.4. We comparing record numbers of the query results, we can see that there are more missing annotation on Arifacts in PMB1than in PMAand in

2starting with "Q" in the formulation name (see Table 9.4)

3starting with "I" in the formulation name (see Table 9.4)

160 CHAPTER 9. VALIDATION OF APPLICABILITY

Table 9.4: Query results of the inferred annotation options

SWRL inference rules and queries PM_A PM_B1 PM_B2

QRule-Activity-hasArtifact 26 19 9

QRule-Activity-hasArtifact + IRule-Activity-subActivity-hasArtifact

27 30 12

QRule-Activity-hasArtifact + IRule-Activity-Input-hasArtifact 26 28 12 QRule-Activity-hasArtifact + IRule-Activity-Output-hasArtifact 32 33 11 QRule-Activity-hasArtifact + IRule-Activity-Input-hasArtifact +

QRule-Activity-hasActor + IRule-Activity-subActivity-hasActor 32 28 23

QRule-Activity-hasPrecedingActivities 30 20 12

PMB2. The big discrepancy is between running QRule-Activity-hasArtifact alone and running QRule-Activity-hasArtifact together with IRule-Activity-Input-hasArtifact. It indicates that many Artifacts are allocated in the sub-Activities in PMB1. Such knowledge is not explicitly represented in the original model so that it is difficult for an annotator to be aware of it when manually annotating. A simi-lar case is observed when running QRule-Activity-hasActor and IRule-Activity-subActivity-hasActor on PMB2. When comparing the three models we notice the following: most Actor-roles are modeled in the sub-Activities of PMB2; no Actor-role is attached to the sub-Activities in PMB1; since the way of modeling Actor-role in BPMN is different from EEML(see the previous paragraph), the annotation about has_Actor-role for each Activity is made manually but carefully (Only one anno-tation is missed by mistake). With respect to the annoanno-tation about the sequence of Activities, again the most missing annotations are found in PMB1. The reason is still the hierarchy of the sub-Activities. Three levels of sub-Activities hierarchy prevents an annotator from picking all the preceding and succeeding Activities inherited from the super-Activities.

Not all of the inferred results should be considered as missing annotations. They are just optional annotations to disclose more implicit knowledge carried by the models. Some results of those queries also produce noise to the evaluation be-cause they should not be the correct annotation. For example, the Activity vn

is the sub-Activity of V1 and the Activity V2 precedes V1, but vn is not the di-rect preceding Activity of V2 because vn is not the last sub-Activity of V1. The query results of Activity-subActivity-transitive-achievesHG, QRule-Activity-subActivity-transitive-positivelysatisfiesSG and

QRule-Activity-9.4. DISCUSSION ON RESULTS OF THE VALIDATION 161