An Extension for MDPs

In this thesis, we will also be developing symbolic techniques for the analysis of MDPs. We will again wish to compare the performance of these methods with explicit alternatives. A comparison of this type is slightly more problematic than with DTMCs or CTMCs since there is no standard storage scheme for MDPs.

In fact, our requirements for MDP storage are slightly unusual. As described in

Section 3.1.2, we consider the choice between different possible behaviours in each state of an MDP as being nondeterministic. We do not attempt to distinguish between these choices in any way. By contrast, in most other applications they are labelled with actions. An analysis of the MDP in this situation might entail, for example, determining the sequence of actions which maximises the probability of some event occurring. In our case, checking a typical PCTL formula would involve simply computing the maximum probability that the event occurs and verifying that it does not exceed some specified bound, not determining the actions.

Because of this, we can opt for a fairly simple data structure to store MDPs. We will use a trivial extension of sparse matrices. As discussed previously, we can think of an MDP as being represented by a matrix, but with each state described by several rows, rather than a single row as for DTMCs and CTMCs. Again, we store the value and column index of every matrix entry in two arrays val and col . This time, however, we require two levels of indexing to associate each entry with both a state of the MDP and a nondeterministic choice in that state. We name these arrays row and nc, respectively.

The idea is illustrated most clearly through a simple example. Figure 3.14 gives the matrix representation for a 4 state MDP and the sparse matrix based data structure which stores it. Again, we only show non-zero entries in the matrix. To determine the entries corresponding to state 0, we first read the values of row [0] and row [1]. This reveals that state 0 contains two nondeterministic choices and that these have index 0 and 1 in the array nc. From nc[0] and nc[1], we see that the first nondeterministic choice comprises a single entry (0, 1) = 1. Likewise, from nc[1] and nc[2], we can deduce that the second choice comprises two entries: (0, 1) = 0.5 and (0, 2) = 0.5.

From the algorithms for PCTL model checking of MDPs in Section 3.3.2, we see that the key operation required is a combined matrix-vector multiplication and maximum or minimum operation. This can be implemented on the sparse data structure in a very similar way to ordinary matrix-vector multiplication. The algorithm for the ‘maximum’ case is given in Figure 3.15. It takes an MDP, stored in arrays row , nc, col and val , a vector stored in an array b and returns the result in an array res. The corresponding algorithm to determine the minimum is almost identical.

          1 · · · · 0.5 0.5 · · · 1 · · 1 · · · · · 1 0.8 · · 0.2           1 0.5 0.5 1 1 1 0.8 0.2 0 1 2 2 1 3 0 3 0 1 3 4 5 6 8 0 2 3 5 6 val col nc row

Figure 3.14: A 4 state MDP and its sparse storage

SparseMVMultMax(row , nc, col , val , b) 1. for (i := 0 . . . |S|−1) 2. li:= row [i] 3. hi := row [i + 1]−1 4. res[i] := −1 5. for (j := li. . . hi) 6. lj := nc[j] 7. hj := nc[j + 1]−1 8. sum := 0 9. for (k := lj. . . hj)

10. sum := sum + val [k] × b[col [k]] 11. endfor

12. if (sum > res[i]) then res[i] := sum 13. endfor

14. endfor 15. return res

Again storing matrix entries as 8 byte doubles and indices as 4 byte integers, the amount of memory required to represent an MDP with this scheme can be computed as 4(|S| + 1) + 4(nnc + 1) + 12nnz bytes, where |S| is the number of states, nnz is the number of non-zero entries and nnc is the total number of nondeterministic choices over all states,

i.e. nnc =P

s∈S|Steps(s)|.

3.7

Multi-Terminal Binary Decision Diagrams

We now introduce multi-terminal binary decision diagrams (MTBDDs), the data structure upon which much of the work in this thesis is based. MTBDDs are actually an extension of binary decision diagrams (BDDs), originally created by Lee [Lee59] and Akers [Ake78], popularised by Bryant [Bry86], and applied to model checking by McMillan, Clarke and

others [BCM+_{90, McM93]. A BDD is a rooted, directed acyclic graph which represents a}

Boolean function of the form f : IBn→ IB.

MTBDDs were first proposed in [CMZ+_{93] and then developed independently in}

[CFM+_{93] and [BFG}+_{93]. In [BFG}+_{93], they were christened algebraic decision diagrams}

(ADDs) but the two data structures are identical. MTBDDs extend BDDs by represent- ing functions which can take values from an arbitrary set D, not just IB, i.e. functions of

the form f : IBn→ D. In the majority of cases, D is taken to be IR and this is the policy

we adopt here. Note that BDDs are in fact a special case of MTBDDs.

Let {x1, . . . , xn} be a set of distinct, Boolean variables which are totally ordered as

follows: x1 < · · · < xn. An MTBDD M over x = (x1, . . . , xn) is a rooted, directed acyclic

graph. The vertices of the graph are known as nodes. Each node of the MTBDD is classed as either non-terminal or terminal. A non-terminal node m is labelled with a variable var (m) ∈ x and has exactly two children, denoted then(m) and else(m). A terminal node m is labelled by a real number val (m) and has no children. We will often refer to terminal and non-terminal nodes simply as terminals and non-terminals, respectively.

The ordering < over the Boolean variables is imposed upon the nodes of the MTBDD. For two non-terminals, m1 and m2, if var (m1) < var (m2), then m1 < m2. If m1 is a non-

terminal and m2 is a terminal, then m1 < m2. We require that, for every non-terminal m

in an MTBDD, m < else(m) and m < then(m).

Figure 3.16(a) shows an example of an MTBDD. The nodes are arranged in horizontal levels, one per Boolean variable. The variable var (m) for a node m can be found at the left end of the level which contains it. The two children of a node m are connected to it by edges, a solid line for then(m) and a dashed line for else(m). Terminals are drawn as squares, instead of circles, and are labelled with their value val (m). For clarity, we omit the terminal with value 0 and any edges which lead directly to it.

(a) 3 9 4 x1 x2 x3 M (b) x1 x2 x3 fM 0 0 1 3 0 1 0 9 1 1 0 9 1 0 0 4 1 0 1 4 otherwise 0

Figure 3.16: An MTBDD M and the function it represents

An MTBDD M over variables x = (x1, . . . , xn) represents a function fM(x1, . . . , xn) :

IBn→ IR. The value of fM(x1, . . . , xn) is determined by tracing a path in M from the root

node to a terminal, for each non-terminal m, taking the edge to then(m) if var (m) is 1 or else(m) if var (m) is 0. The function represented by the MTBDD in Figure 3.16(a) is

shown in Figure 3.16(b). For example, the value of fM(0, 1, 0) can be read as 9. We also

use the notation fM[x1 = 0, x2 = 1, x3 = 0] or fM[x = (0, 1, 0)]. This is often convenient

when we wish to express the value of an MTBDD without explicit reference to the ordering of its Boolean variables.

We define a minterm of an MTBDD as a valuation of its Boolean variables which results in a non-zero value. The MTBDD in Figure 3.16, for example, has 5 minterms, each corresponding to a row in the table. Note that there is not necessarily a one-to-one mapping between paths in the MTBDD and minterms. For example, the path to the 4 terminal in Figure 3.16(a) corresponds to two different minterms.

For an MTBDD M, over variables (x1, . . . , xn), we define its cofactor M|xi=b (where

b = 0 or 1) as the MTBDD over variables (x1, . . . , xi−1, xi+1, . . . , xn) representing the

function fM(x1, . . . , xi−1, b, xi+1, . . . , xn). Conversely, the function fM for an MTBDD M,

with top variable x1, can be evaluated recursively using its cofactors:

fM = x1· fM|_x1=1 + (1 − x1) · fM|_x1=0

Note that, because of the recursive nature of the data structure, each node of an MTBDD is itself an MTBDD. Hence, we will often consider an MTBDD and the node which represents it (i.e. its root node) to be interchangeable.

The reason that MTBDDs can often provide compact storage is because they are stored in a reduced form. Firstly, if nodes m and m0 are identical (i.e. var (m) = var (m0),

then(m) = then(m0) and else(m) = else(m0)), only one copy of the node is stored. We

refer to this as sharing of nodes. Secondly, if a node m satisfies then(m) = else(m), it is removed and any incoming edges from levels above are redirected to its unique child.

(a) 0 3 9 0 9 0 4 4 x2 x3 x1 M (b) 3 9 4 x3 x1 x2 M (c) 3 9 4 x1 x2 x3 M Figure 3.17: Reducing an MTBDD

We will refer to this as a skipped level. Figure 3.17 illustrates the way that the MTBDD in Figure 3.16 has been reduced: (a) shows the full binary tree representing the same function; in (b), shared nodes have been merged; and in (c), nodes with identical children have also been removed, introducing skipped levels.

In this thesis, unless stated otherwise, we assume that all MTBDDs are fully reduced in this way. Under this assumption, and for a fixed ordering of Boolean variables, the data structure can be shown to be canonical, meaning that there is a one-to-one correspondence between MTBDDs and the functions they represent. As we will see later, this canonicity property has crucial implications for performance.

Another important consideration from a practical point of view is variable ordering. The size of an MTBDD (i.e. number of nodes) representing a given function is extremely sensitive to the ordering of its Boolean variables. This has a direct effect on both the storage requirements for the data structure and the time needed to perform operations on it. We will consider this topic in more depth in Chapter 4.

The size of an MTBDD is also affected by the number of terminals it contains, or equivalently, the number of distinct values taken by the function which it represents. Compact MTBDDs can only be obtained by maximising sharing within the nodes of the graph. A high number of terminals reduces the capacity for sharing and so increases the number of nodes in the MTBDD.