Controlling GSMDPs

As in the MDP case, searching for control strategies on GSMDP implies defining rewards r(s, e) or r(s, e, s0_{) associated to transitions and introducing policies and criteria.}

The same characteristics arise with GSMDP than in the GMSP case: the transition function for the global semi-Markov process does not retain Markov’s property without aug- menting the state space. In other words, only the augmented process is Markovian, the natural process is not. In the classical MDP framework, one can make use of the transition function’s Markov’s property to prove that there exists a Markovian policy (depending only on the current state) which is at least as good as any history-dependent policy (Cf. [Put- erman, 1994]). In the GSMDP case however, this is no longer possible because the natural process is not Markovian.

In order to define criteria and to find optimal policies, we need — in the general case — to allow the policy to depend on the whole execution path of the process. [Younes and Simmons, 2004] define execution paths for a GSMDPs. An execution path of length n from natural state s0 to state sn is a sequence ρ = (s0, t0, e0, s1, . . . , sn−1, tn−1, en−1, sn) where ti

is the sojourn time in state si before event ei triggers. As in [Younes and Simmons, 2004],

we define the discounted value of ρ by: V_γπ(ρ) =n−1X i=0 γTi  γti_k(s i, ei, si+1) + Z _ti 0 γ t_c(s i, ei)dt  (11.1) where k and c are traditional SMDP lump sum reward and reward rate functions4_{, and}

Ti =Pi−1j=0tj.

One can then define the expected value of policy π in state s as the expectation over all execution paths starting in s:

Vπ γ(s) = Esπ  Vπ γ(ρ)  (11.2) This provides a criterion for evaluating policies. The goal is now to find policies that maximize this criterion. The main problem here is that it is hard to search the space of history-dependent policies. So the simplest solution would be to make our process Marko- vian again by using the supplementary variable technique and then to search for an optimal policy in the space of Markovian policies defined over the augmented state space.

Using the supplementary variable technique consists in augmenting the natural state space with just enough variables so that the distribution over future augmented states only depends on the current values of these variables. As in [Nielsen, 1998], we can augment the natural state s of the process with all the clock readings and show that this operation brings Markov behavior back to the GSMDP process. We will note this augmented state space (s, c) for convenience.

Unfortunately, as foreseen at the end of the previous section, it is unrealistic to define policies over this augmented state space since clock readings contain information about the future of the system. From here, several options are possible:

4_{see equation 2.26 for details on k and c.}

11.5. MDPs, continuous time and concurrency • One could decide to sacrifice optimality and to search for “good” policies among a restricted set of policies, for example the policies defined on the current natural state only.

• One could also search for representation hypothesis that simplify the GSMDP model and that make natural state Markovian again.

• One could compute optimal policies on the augmented state space (s, c) and then derive a policy on observable variables only.

• Finally, one could look for a set of observable variables which retain Markov’s property for the process, for example the set composed of the natural state of the process s, the duration for which each active event ei has been active τi and its activation state si.

We will note this augmented state (s, τ, sa).

This series of options leads to consider the question of observability in the stochastic process. Namely, it is important to know which variables are observable and what prior knowledge we have concerning the relationship between observations and the real state of the process. The field of Partially Observable MDPs (POMDPs, [Kaelbling et al., 1998]) studies this question in detail. We won’t enter the question of partial observability in MDPs here and will consider the natural state to be fully observable.

The optimization approach taken by [Younes and Simmons, 2004] is based on the second option listed above. Namely, the authors approximate any distribution on time by a chain of exponential distributions called a phase-type distribution. These distributions, as presented in [Neuts, 1981], allow to fit any number of known moments of a prior distribution using only chains of exponential distributions. Once this approximation is made, they introduce extra states for the intermediate steps (the phases) corresponding to the inner states of the chains. Using the memoryless property of exponential distributions, this brings the GSMDP back to a time-homogeneous continuous time MDP (CTMDP). Then, they perform uniformization, as in [Puterman, 1994], in order to transform the CTMDP into a standard MDP which can be solved using standard discrete MDP methods.

This approach necessitates to be able to approximate the duration functions with phase- type distributions. On top of that, it is limited to discrete natural states. We wish to avoid — as much as possible — making hypothesis on the model itself and on the underlying distributions. Moreover, many of the variables we will consider are continuous variables or a mix of continuous and discrete variables. We also need, for the problems at hand, to consider methods for policy search which can handle state spaces with large dimensions. Con- sequently, in the list above, we will look for a solution corresponding either to option 1, 3 or 4.

Chapter 11. Concurrency: an origin for complexity

Because of the non-Markov behaviour of GSMDPs’ natural process, there is no guarantee that there exists an optimal policy verifying Markov’s property. On the other hand, searching for a policy in the space of history-dependent policies is not an acceptable solution. Hence, one needs to make a choice between:

• Sacrificing optimality by restricting the policy search space.

• Finding the correct minimal number of observable variables to add to the natural process in order to regain Markov behaviour.

• Constructing policies on non-observable variables and then use a priori knowl- edge to derive policies on observable variables.