Not-so-faulty Train-Gate-Controller

We begin by revisiting the train-gate-controller example from Section 6.2.2.2. As previously, the controller acts as an arbiter, attempting to ensure that there is a mutual exclusion over the tunnel (i.e., that only one train is in the tunnel at any one time).

We illustrate the interactions between a model with two trains and one controller in Figure 6.10. This figure illustrates that Train1communicates only with the Controller,

and similarly that Train2communicates only with the Controller. As such, we consider

the modular verification of a train against its controller “environment” (i.e., the agents inside the shaded yellow rectangle in Figure 6.10).

Controller

Train1 Train2

Fig. 6.10. Interactions in the Train-Gate-Controller Model

As with the evaluation for etav in Section 6.2.2.2, we consider trains that are enriched with a counter. In the faulty model, this counter is used to store an “error threshold” value that, if exceeded, allows the trains to exhibit a non-deterministic fault. However, as the technique of Chapter 5 is incomplete (i.e., if the guarantee check returns a false, we cannot derive a definitive answer about the monolithic check), using broken trains does not permit a useful evaluation. Consequently, we still consider trains with counters, but with the exception that the faults can never occur.

In Figure 6.11, we provide a sample ISPL agent representing a non-faulty train with a maximum counter of 10 (i.e., the variable counter defined in Train1’s local state ranges from 0 to 10). It can be observed from this figure that, once Train1is in the

tunnel, it stays there unless the tunnel performs the action leave1. That is, not only

does the controller (in Figure 6.11 this agent is the Environment) decide if the train should enter the tunnel, but also when it should leave the tunnel. Nonetheless, it could be possible to create a train that could stay in the tunnel indefinitely, should its designer choose.

6.5.1.1 Propositions and Specifications

In the evaluation that follows, we select a train-local temporal-epistemic formula to verify. As the train-gate-controller model is effectively a mutual exclusion problem, it is sensible to consider only the following proposition:

train1 in tunnel

This proposition holds only when the local state of the train is assigned to be the value corresponding to “in the tunnel”.

Based on the above proposition, we consider the following formula:

φ = AG (train1 in tunnel → AX (KTrain1(AX¬train1 in tunnel)))

which has the reading: “if the train is in the tunnel, at the next state it knows at the following state it is out of the tunnel”.

Being able to verify if each train satisfies φ in isolation is beneficial as it allows us to give guarantees that each train has been designed correctly. It is argued that the faulty

6.5 Evaluating ModularMCMAS 165 1 A g e n t T r a i n 1 2 V a r s: 3 s t a t e : { wait , in , a w a y }; 4 c o u n t e r : 0 . . 1 0 ; 5 end V a r s 6

7 A c t i o n s = { enter , leave , return , s i g n a l };

8 9 P r o t o c o l: 10 s t a t e = w a i t : { signal , e n t e r }; 11 s t a t e = in : { l e a v e }; 12 s t a t e = a w a y : { r e t u r n }; 13 end P r o t o c o l 14 15 E v o l u t i o n: 16 17 s t a t e = in and c o u n t e r = c o u n t e r + 1 if 18 c o u n t e r < 10 and s t a t e = w a i t and 19 A c t i o n = e n t e r and 20 E n v i r o n m e n t .A c t i o n = e n t e r _ 1 ; 21 22 s t a t e = in and c o u n t e r = 0 if c o u n t e r = 10 and 23 s t a t e = w a i t and A c t i o n = e n t e r and 24 E n v i r o n m e n t .A c t i o n = e n t e r _ 1 ; 25 26 s t a t e = w a i t if s t a t e = w a i t and 27 (!( E n v i r o n m e n t .A c t i o n = e n t e r _ 1 ) 28 or A c t i o n = s i g n a l ) ; 29 30 s t a t e = a w a y if s t a t e = in and A c t i o n = l e a v e and 31 E n v i r o n m e n t .A c t i o n = l e a v e _ 1 ; 32 33 s t a t e = in if s t a t e = in and 34 !( E n v i r o n m e n t .A c t i o n = l e a v e _ 1 ) ; 35 36 s t a t e = w a i t if s t a t e = a w a y and A c t i o n = r e t u r n ; 37 end E v o l u t i o n 38 end A g e n t 39 40 - - E v a l : 41 - - t r a i n 1 _ i n _ t u n n e l if T r a i n 1 . s t a t e = in ; 42 - - 43 - - I n i t : 44 - - T r a i n 1 . s t a t e = a w a y and T r a i n 1 . c o u n t e r = 0;

Fig. 6.11. Example ISPL for a Single Train

trains of Section 6.2.2.2 are indeed faulty; their presence can lead to the possibility of the mutual exclusion on the tunnel being invalidated.

6.5.1.2 Modular Verification of Trains

We now move on to performing modular verification of a selected train.

To perform modular verification, we first identify the assumption that we will show the environment satisfies (i.e., where Ae= Controller). The assumption selected is the

A-LTL formula:

ψ = G (train i enter → X train i leave)

This formula specifies that, if the environment signals to the train to enter the tunnel, at the next state it signals to the train to leave the tunnel.

We note that, unlike approaches discussed in Section 2.1.2, generating such an assumption is still a manual process.

Figure 6.12 shows the dra2ispl-generated property environment for Train1for the

assumption ψ. Following the presentation in Section 6.3.2, the single action condition is enforced via the translation of the deterministic Rabin automaton for the assumption into an agent.

In the context of the assumption ψ, and for Ai= Train1, we look to verify that:

Ai|=ψCTLKAG(train1 in tunnel → AX (KTrain1(AX¬train1 in tunnel)))

using agr-mcmas. The A-LTL assumption ψ is used to constrain the behaviour of Aito

disallow the infinite path under which Aiidles permanently in the tunnel.

As such, in our evaluation, we need to perform two checks using the modular approach:

1. Ae|=A-LTLψ

2. Ai|=CTLKψ AG(train1 in tunnel→ AX (KTrain1(AX¬train1 in tunnel)))

In what follows, we refer to the first check as the assumption check and the second as the guarantee check.

1 A g e n t E n v i r o n m e n t 2 V a r s: 3 s t a t e : { s_0 , s_1 , s_2 , s_3 }; 4 end V a r s 5 A c t i o n s = { o t h e r _ a c t i o n , enter_1 , l e a v e _ 1 }; 6 P r o t o c o l: 7 O t h e r : { o t h e r _ a c t i o n , enter_1 , l e a v e _ 1 }; 8 end P r o t o c o l 9 E v o l u t i o n: 10 s t a t e = s_0 if s t a t e = s_0 and A c t i o n = o t h e r _ a c t i o n ; 11 s t a t e = s_1 if s t a t e = s_0 and A c t i o n = e n t e r _ 1 ; 12 s t a t e = s_0 if s t a t e = s_0 and A c t i o n = l e a v e _ 1 ; 13 s t a t e = s_2 if s t a t e = s_1 and A c t i o n = o t h e r _ a c t i o n ; 14 s t a t e = s_2 if s t a t e = s_1 and A c t i o n = e n t e r _ 1 ; 15 s t a t e = s_0 if s t a t e = s_1 and A c t i o n = l e a v e _ 1 ; 16 s t a t e = s_2 if s t a t e = s_2 ; - - s e l f l o o p 17 s t a t e = s_0 if s t a t e = s_3 and A c t i o n = o t h e r _ a c t i o n ; 18 s t a t e = s_1 if s t a t e = s_3 and A c t i o n = e n t e r _ 1 ; 19 s t a t e = s_0 if s t a t e = s_3 and A c t i o n = l e a v e _ 1 ; 20 end E v o l u t i o n 21 end A g e n t 22 23 - - E v a l : 24 - - L_0 if E n v i r o n m e n t . s t a t e = s_0 or E n v i r o n m e n t . s t a t e = s_1 ; 25 - - U_0 if E n v i r o n m e n t . s t a t e = s_2 ; 26 - - 27 - - I n i t : 28 - - E n v i r o n m e n t . s t a t e = s_3 ;

6.5 Evaluating ModularMCMAS 167

Comparison 1.

We begin with an exaggerated example: we assume that each train has a maximum counter of 10,000. Furthermore, we consider an environment suitably extended with its own “timeout counter” of 4, and that it can arbitrate between a total of 6 trains. This timeout counter is used to ensure that, if the environment selects a train to enter the tunnel, it will try up to n times to allow the train to enter. If after the n-th time the train does not enter, the controller will assume that the train has “timed out” and selects another train to enter the tunnel.

We begin with a straight comparison between mcmas-1.0, at-mcmas and agr-mcmas.

We refer to the checks by mcmas-1.0and at-mcmas as “monolithic verification”, as

they consider the system as a whole. The memory usage and time taken for verifying the above model using the three tools is shown in Table 6.8.

Table 6.8. Train-Gate-Controller: Comparison 1 – Memory and Time

Mode Time (s) Memory (KiB)

Monolithic Verification DNF

Guarantee Check 99.987 489,612

Assumption Check 0.289 27,688

For mcmas-1.0or at-mcmas, neither tool completed monolithic verification within

two hours. Furthermore, even by reducing each train’s counter from 10,000 to 200, this did not assist either tool in completing the verification task within two hours.

In light of the failures of the two monolithic tools, we can see the clear benefit of agr-mcmas when decomposing the verification task. As is to be expected, the guarantee check—with a counter of 10,000—generates a much larger model than the assumption check (where the controller only has a counter of four), and therefore it is the guarantee check that has the greater resource requirement (i.e., in memory and time).

As both monolithic approaches did not complete within our designated time limit, we attempt to evaluate the effectiveness of agr-mcmas in reducing the state-space by considering the number of BDD variables required to encode the model. This comparison is shown in Table 6.9.

From Table 6.9, we can also observe that, interestingly, checking the assumption requires one extra action variable. When performing the verification of the environment against the assumption, we are required to compose the Environment-under-test with the observer automaton for the property (i.e., the agent who can only perform the null-action nop, which is unobservable by all other agents in the system), as well as the minimum “skeleton” for each additional agent that the environment communicates with. As such,

Table 6.9. Train-Gate-Controller: Comparison 1 – BDD Variables and Reachable States Mode State Variables Action Variables Total Variables Max. # States Reachable # States

Monolithic Verification 100 17 217 ≈ 1.268×1030 _unknown

Guarantee Check 18 4 40 262,144 65,538

Assumption Check 22 18 62 4,194,304 1,665

it is therefore necessary to create the composition of universal agents for each other component in the model, plus the property observer with a single action. For each other component, the number of BDD variables required to encode the actions for each concrete agent vs. each skeleton is the same. However, it is necessary to allocate the null action in the observer its own BDD variable. This results in the assumption checking requiring one additional BDD action variable in comparison to the monolithic check.

The column “Max. # States” is equal to the total number of states representable for a given number of BDD variables; for example, with 10 variables, there are 210_{= 1024 =}

1.024 × 103representable states.

As we can see, comparing the representable state-spaces alone shows the potential reductions that assume-guarantee can bring. The maximum number of instantaneous states that need to be considered in either parts of the assume-guarantee check is 4,194,304 states; by comparison, for the monolithic approach, these techniques have to cater for the potential of analysing a state-space of that is twenty four orders of magnitude larger.

Comparison 2.

Given the inability of either monolithic tool to complete verification within the allocated time, we now consider a smaller example, which both tools can complete within our two hour time limit. We evaluate the tools on a model with three trains, each with a maximum counter of 5, an environment with a timeout of 1, and consider the same formula as previously.

Table 6.10 shows the number of BDD variables and number of reachable states considered in this second model. As both at-mcmas and mcmas-1.0construct the same

set of reachable states—it is just their verification approach that differs—it is of no surprise that both tools generate the same number of states.

However, it can be immediately seen that agr-mcmas has to explore significantly fewer states: we see a reduction of three orders of magnitude between the modular and monolithic approaches.

6.5 Evaluating ModularMCMAS 169 Table 6.10. Train-Gate-Controller: Comparison 2 – BDD Variables and Reachable States

Mode State Variables Action Variables Total States

Monolithic Verification 23 10 40,961

Assumption Check 13 11 65

Guarantee Check 7 4 57

Table 6.11. Train-Gate-Controller: Comparison 2 – Memory and Time

Mode Nodes in product graph Memory (KiB) Time (s)

WEAA Monolithic (at-mcmas) 825,563 217,504 344.364

BDD Monolithic (mcmas-1.0) N/A 28,100 0.147

Assumption Check 396 25,292 0.101

Guarantee Check 57 24,112 0.023

We now consider the memory and time used across the three tools (Table 6.11). Un- like Table 6.10, the memory and time is different between mcmas-1.0and agr-mcmas.

As is immediately obvious, at-mcmas’s “hybrid state” approach for verifying formulae is clearly not effective. It requires an order of magnitude more memory, as well as taking three orders of magnitude more time. It is also not surprising that, as agr-mcmas has to consider fewer states, the number of nodes in the product graph is also smaller compared to that of the monolithic check with at-mcmas.

However, although the automata-theoretic approach does not seem favourable in the monolithic sense, its extension to support modular verification is clearly favourable – taking both less memory and less cumulative time. While these results are promising, they are not quite as dramatic as we would have hoped. As we will show in the next section, the model selected for allowing a direct comparison between the three tools does not permit us to demonstrate the true potential of agr-mcmas. By focusing on larger models that are outside the abilities of at-mcmas, and therefore excluding it from our comparison, we can demonstrate the expected benefits of the modular technique.

Comments

While the results shown in Section 6.5.1 highlight the potential benefits of compositional verification, we note that this might be seen as a slightly synthetic benchmark as, apart from the environment, the model is comprised entirely of homogeneous agents. We argue that, while techniques such as parametric verification [Kouvaros and Lomuscio,

2013a; Kouvaros and Lomuscio, 2013b] or symmetry reduction [Cohen et al., 2009a; Cohen et al., 2009b] would be applicable in the case of models containing many homogeneous agents, these would still require construction of the composition of at least the environment and one agent. By comparison, in the approach presented here, we only check the composition between the assumption and the component, and the environment and the assumption.