The Solution Process

Secondly, and more importantly, we consider the performance of the numerical solution phase. Although the exact computation carried out varies depending on the instance of probabilistic model checking being performed, this phase always comprises a number of iterations, the key element of each being a single traversal of an offset-labelled MTBDD using the TraverseOffsets algorithm. Table 6.2 shows timing statistics for the four model checking case studies described in the previous section. For each example, it lists the size of the model (number of states), the number of iterations performed and the average time per iteration for each of our three implementations. Here, as always, we give actual (wall) times measured on a 440 MHz 512 MB Sun Ultra10. A ‘-’ symbol indicates that an example could not be model checked due to memory limitations. For the Kanban system, polling system and FMS case studies, we increased the parameter N as far as possible for each implementation. For the BRP examples, this was not the case since the computation for some of the larger models we built suffered from round-off errors.

The results can be summarised as follows. The new hybrid approach represents a marked improvement over the original MTBDD implementation. There is an impressive reduction in average iteration time for all examples. In the best cases (Kanban system for N = 3 and polling system for N = 10), it is improved by a factor of more than 100. Unfortunately, the times remain much slower than for sparse matrices.

The most noteworthy result, though, is that our hybrid approach can handle larger models than both of the other implementations. This is a result of their relative memory requirements. As in the previous chapter, we do not attempt a detailed comparison with the memory requirements for MTBDDs since accurate information is very difficult to obtain. We can see from Table 6.2, however, that many of the examples cannot be performed using the MTBDD implementation due to insufficient memory. A comparison of the memory required for hybrid and sparse matrix approaches is given in Table 6.3.

Model N States Iterations Time per iteration (sec.) MTBDD Sparse Hybrid 1 160 101 0.03 0.001 0.001 2 4,600 166 2.22 0.002 0.03 Kanban 3 58,400 300 45.5 0.05 0.35 system 4 454,475 466 - 0.41 4.79 5 2,546,432 663 - 2.70 26.1 6 11,261,376 891 - - 123.8 8 3,072 310 0.31 0.001 0.004 10 15,360 406 13.1 0.01 0.03 Polling 12 73,728 505 - 0.05 0.16 system 14 344,064 606 - 0.30 0.89 16 1,572,864 709 - 1.56 4.46 18 7,077,888 814 - - 23.1 500 27,505 3,086 0.38 0.01 0.12 1,000 55,005 6,136 0.67 0.01 0.24 BRP 1,500 82,505 9,184 0.99 0.02 0.36 2,000 110,005 12,230 1.26 0.03 0.47 2,500 137,505 15,274 1.53 0.04 0.59 3 6,520 569 6.13 0.002 0.11 4 35,910 733 48.3 0.02 0.84 FMS 5 152,712 825 - 0.10 3.35 6 537,768 968 - 0.37 13.3 7 1,639,440 1,040 - 1.14 33.6 8 4,459,455 1,182 - - 102.0

Model N Vector Matrix storage (KB) Total storage (KB) storage (KB) Sparse Hybrid Sparse Hybrid

1 3 × 1 8 10 11 13 2 3 × 36 348 33 455 141 Kanban 3 3 × 456 5,455 48 6,823 1,417 system 4 3 × 3,551 48,414 96 59,066 10,748 5 3 × 19,894 296,588 123 356,270 59,805 6 3 × 87,979 - 154 - 264,092 8 3 × 24 186 12 258 84 10 3 × 120 1,110 19 1,470 379 Polling 12 3 × 576 6,192 26 7,920 1,754 system 14 3 × 2,688 32,928 34 40,992 8,098 16 3 × 12,288 168,960 44 205,824 36,908 18 3 × 55,296 - 55 - 165,943 500 4 × 215 441 59 1,301 919 1,000 4 × 430 883 61 2,602 1,781 BRP 1,500 4 × 645 1,324 63 3,902 2,643 2,000 4 × 859 1,766 63 5,203 3,499 2,500 4 × 1,074 2,207 65 6,504 4,361 3 4 × 51 451 213 655 416 4 4 × 281 2,711 546 3,835 1,668 FMS 5 4 × 1,193 11,990 974 16,762 5,746 6 4 × 4,201 42,744 1,497 59,548 18,302 7 4 × 12,808 129,853 2,324 181,085 53,556 8 4 × 34,839 - 4,132 - 143,490

Table 6.3: Memory requirements for numerical solution using the hybrid approach

We present statistics for the same four case studies: the amount of memory to store the matrix; the amount to store the iteration vectors; and the total memory, which is the sum of these. All values are rounded to the nearest kilobyte.

For the hybrid implementation, the memory required to store the matrix is determined by the number of nodes in the offset-labelled MTBDD. Each node requires 20 bytes, as was the case for a standard MTBDD. Although we must store an additional integer (the offset) on each node, we can reuse the space previously occupied by the node’s reference count (see Section 3.7.3). This is not required because we never need to manipulate this new data structure: we simply create it once, traverse it repeatedly and then discard it. The memory required to store the matrix in sparse format is computed as described in Section 3.6.

In both cases, we assume that all vectors are stored as arrays of doubles (8 bytes per entry). The number of arrays required varies depending on the model checking problem.

All the iterative methods require at least two, to store the previous and current approxi- mations. We also store the diagonal entries of the matrix in an array because this gives a significant improvement in speed. Additionally, for CSL time-bounded until we require one to keep track of the weighted sum of vectors, and for PCTL until over DTMCs we need one to store the vector b of the linear equation system A · x = b. For CSL steady- state, this is not necessary since b is the zero vector. The actual number of arrays needed for each example is given in the table.

From Table 6.3, the main conclusion we draw is that, despite the potential increase in MTBDD size and storage of additional information, the amount of memory required to store the matrix using the hybrid approach is still far less than with sparse matrices. At best, the ratio is four orders of magnitude. In fact, the limiting factor, in terms of memory, now becomes the storage of the iteration vectors.

On the 512 MB Sun Ultra10 used for these experiments, the largest probabilistic model successfully analysed with the hybrid technique had over 11 million states; with sparse matrices, the largest had about 2.5 million states. The choice of machine for our experiments was dictated by the fact that we needed a single-user workstation in order to generate accurate timings. To test the scalability of our approach, we also ran experiments from the same case studies on a shared machine with 1 GB of RAM. On this, the largest model handled by the hybrid approach had 41 million states, compared to 7 million for the sparse matrix implementation. In general, we find that we can increase the size of solvable model by approximately a single order of magnitude.