Evaluation 146

The ATM case study was used to evaluate our work in order to: ƒ measure recall and precision of our tool;

ƒ identify missing elements;

ƒ show how the identified missing elements can support the development of the system by increasing its recall.

It is important highlight that this case study was used to demonstrate the performance of the tool in discovering and correcting defects in the models.

In order to demonstrate the above, we have (i) identified traceability relations between the models manually, (ii) identified traceability relations between the models using our tool, (iii) measured recall and precision by comparing the results in (i) and (ii), (iv) identified missing

elements using our tool, (v) used the results in (iv) to manually complete the models creating a new set of the models, (vi) used the tool to identify traceability relations and missing elements in the new models created in (iv), and (vii) measured the recall and precision of the results in (vi).

Our experiment has been conducted using 47 traceability rules to identify traceability relations between Prometheus elements and JACK code and 16 completeness checking rules to identify missing elements between Prometheus elements and JACK code, for the ATM case study. Table 5.3 shows the results of our experiments for (i), (ii) and (iii) activities.

Number of relations identified by the engine |T| 31 Number of relations identified by the user |M| 64 Number of missing relations 33 Number of incorrect relations 0 Number of relations that intercept T and M 31

Recall 50% Precision 100%

Table 5.3 Results of experiments for the ATM case study

As shown in Table 5.3, the experiment achieved 50% of recall and 100% of precision. The table also shows that there were 33 missing traceability relations and 0 incorrect traceability relations. The tool also returns information about missing traceability relations due to the absence of the elements related to these relations in the models. The missing relations identified by the tool are shown in Table 5.4. The number of missing relations identified by the tool can be lower (i.e. 15) than the number of missing relations identified manually (i.e. 33).The data of the traceability relations identified manually and by the tool is shown in the Appendix C.

We used the information of the missing elements identified by the tool (see Table 5.4) to fix inconsistencies and complete the models. As shown in Table 5.4, there were missing relations between (a) JACK BeliefSet and Prometheus Data, (b) JACK Plan and Prometheus Plan, (c) Prometheus Plan and JACK Plan, (d) Prometheus Goal and JACK Agent, and (e) Prometheus Message and JACK Agent.

Rule ID JACK BeliefSet Prometheus Data

RulePJ2cc1 Accounts RulePJ2cc1 Balances

Rule ID JACK Plan Prometheus Plan

RulePJ3cc1 ProcessWithdraw RulePJ3cc1 WithdrawApproved RulePJ3cc1 WithdrawCash RulePJ3cc1 WithdrawRejected

Rule ID Prometheus Plan JACK Plan

RulePJ3cc2 Withdraw Approved

RulePJ3cc2 Process Withdraw

RulePJ3cc2 Withdraw Cash

RulePJ3cc2 Withdraw Rejected

Rule ID Prometheus Goal JACK Agent

RulePJ5cc1 Request Approved

RulePJ5cc1 Request Rejected

RulePJ5cc1 Withdraw Money

RulePJ5cc1 Authorize Withdraw

Rule ID Prometheus Message JACK Agent

RulePJ12cc1 Withdraw

Table 5.4 Missing Information

More specifically, Table 5.4 shows that RulePJ2cc1 rule identifies missing traceability relations between Accounts and Balances beliefSets in JACK and some data in Prometheus. We look what data in Prometheus could be related to Accounts beliefSet and conclude that Accounts

beliefSet in JACK should be related to Accounts data in Prometheus. BeliefSets in JACK and data in Prometheus are related when the name of the beliefSet and the name of data are synonyms and included fields/aspects properties of the data is similar to the fields in the beliefSet. Figure 5.1 and 5.2 shows that Accounts beliefSet has account and pin fields and no

included fields/aspects properties has been defined to the Accounts data .

Figure 5.1 Fields of the Accounts beliefSetAccounts Descriptor <beliefSet id="b1" type="Accounts" extends="OpenWorld">

<field declarationType="key" type="int" name="account"/> <field declarationType="value" type="int" name="pin"/> ...

Similarly, we look what data in Prometheus could be related to Balances beliefSet. We concluded that Balances beliefSet in JACK should be related to Balances data in Prometheus. Figure 5.3 and 5.4 shows that Balances beliefSet has account and balance fields and no included fields/aspects properties has been defined to the Balances data.

Figure 5.3 Balances beliefSet Figure 5.2 Accounts Descriptor

<beliefSet id="b1" type="Balances" extends="OpenWorld"> <field declarationType="key" type="int" name="account"/> <field declarationType="value" type="int" name="balance"/> ...

We added accounts and balances to the included fields/aspects to the Balances data in Prometheus.

Table 5.4 also shows (i) that RulePJ3cc1 rule identifies that there are missing traceability relations between ProcessWithdraw, WithdrawApproved, WithdrawCash and

WithdrawRejected plans in JACK and plans in Prometheus; and (ii) that RulePJ3cc2 rule identifies that there are missing traceability relations between Process Withdraw,Withdraw Approved, Withdraw Cash and Withdraw Rejected plans in Prometheus and plans in JACK. Based on this information we can determine that there are missing traceability relations between ProcessWithdraw in JACK and ProcessWithdraw in Prometheus. Plans in JACK and plans in Prometheus are related when the name of the plan in JACK and the name of plan in Prometheus are synonyms, and the name of the element that triggers the plan in Prometheus and the name of the event that the plan in JACK handles are synonyms.

Figure 5.4 Balances descriptor

<plan id="p1" name="ProcessWithdraw" extends="Plan"> <handlesEvent type="WithdrawRequest" ref="event"/>

... </plan>

We observed that ProcessWithdraw plan in JACK handles WithdrawRequest event (see Figure 5.5) while no trigger properties has been defined to the Process Withdraw plan in Prometheus (see Figure 5.6).

We added Withdraw Request message to the triggers properties of the Process Withdraw

plan. Similarly, we identified that a traceability relations between ProcessWithdraw plan in JACK and ProcessWithdraw plan in Prometheus was missing. We identified that the

WithdrawRequest plan in JACK handles WithdrawResponse event (see Figure 5.7) while no trigger properties has been defined to the Withdraw Approved plan in Prometheus (see Figure 5.12).

We added Withdraw Response message to the triggers properties of the Withdraw Approved plan.

Figure 5.6 Process Withdraw descriptor

<plan id="p2" name="WithdrawApproved" extends="Plan"> <handlesEvent type="WithdrawResponse" ref="event"/> ...

</plan>

The experiment also shows that a traceability relation was also missing between Withdraw Cash plan in JACK and Withdraw Cash in Prometheus. We found that WithdrawCash plan in JACK handles Withdraw event (see Figure 5.9) while no trigger properties has been defined to the Withdraw Cash plan in Prometheus (see Figure 5.10).

Figure 5.8 Withdraw Approved descriptor

<plan id="p3" name="WithdrawCash" extends="Plan"> <import>gui.AtmClient</import>

<import>gui.AtmInterface</import>

<handlesEvent type="Withdraw" ref="event"/> ...

</plan>

Figure 5.10 Withdraw Cash descriptor

We added Withdraw message to the triggers properties of the Withdraw Cash plan. A traceability relation was also missing between Withdraw Rejected plan in JACK and

Withdraw Rejected plan in Prometheus. We found that WithdrawRejected plan in JACK handles WithdrawResponse event (see Figure 5.11) while no trigger properties has been defined to the Withdraw Rejected plan in Prometheus (see Figure 5.12).

<plan id="p4" name="WithdrawRejected" extends="Plan"> <import>gui.AtmClient</import>

<import>gui.AtmInterface</import>

<handlesEvent type="WithdrawResponse" ref="event"/> ...

</plan>

We added Withdraw Response message to the triggers properties of the Withdraw Rejected

plan.

Table 5.4 also shows that RulePJ5cc1 rule identifies missing traceability relations between

Request Approved, Request Rejected, Withdraw Money and Authorize Withdraw goals in Prometheus and some agents in JACK. We examined Prometheus models and found that we have not defined that Prometheus agents are supposed to achieve Request Approved,

Request Rejected, Withdraw Money and Authorize Withdraw goals. We updated the model and defined that Request Approved, Request Rejected, and Withdraw Money goals in Prometheus are achieved by Atm agent and Authorize Withdraw is achieved by Bank

agent.

Table 5.5 shows the results of our experiment after rectifying the documents and running the tool for the new versions of the models.

As shown in Table 5.5, the experiment achieved 100% of recall and 100% of precision, demonstrating an increase in the recall with respect to the initial results in the Table 5.3.

Number of relations identified by the engine |T| 64 Number of relations identified by the user |M| 64 Number of missing relations 0 Number of incorrect relations 0 Number of relations that intercept T and M 64

Recall 100% Precision 100%