In this section, we gave a survey of the measure theoretic foundations that are needed throughout this thesis. As will turn out in the next section, we need the cylinder set construction and infinite product σ-fields to define infinite trajectories of continuous-time Markov decision processes and interactive Markov chains. Moreover, the Lebesgue integral and Fubini’s theorem allow to define their semantics precisely.
Most of the material presented here is based on chapters one, two and four of the excellent book “Probability & measure theory” by Robert Ash and Catherine Dol´eans-Dade [ADD00]. Other references include the book “Real Analysis and Probability” by Richard Dudley [Dud02] and “Probability and Measure” by Patrick Billingsley [Bil95].
Some remarks about the difficulties in extending the notion of length to a class of subsets
of the reals that is larger than the Lebesgue measurable sets can be found in the German book “Stochastik f¨ur Informatiker” by Rudolf Mathar [MP90] and in [Ben76].
The proofs of Lemma 2.6 and Lemma 2.10, as well as the proof of Lemma 2.11 are mostly omitted in the literature. Hence, they have been proved anew and adapted to Ash’s notation, which is used throughout this thesis.
Finally, some enlightening details about the Vitali set construction, especially about the cardinality of the sets involved, can be found in the English translation of Kanovei’s paper [Kan91]. The remaining material presented in Sec. 2.3 is mostly based on a lecture note [vRS92] from Radboud University, Nijmegen.
Proof is the idol before whom the pure mathematician tortures himself.
(Sir Arthur Eddington)
In this thesis, we discuss a variety of probabilistic and stochastic models that describe the system behavior either in discrete or in continuous time. Therefore, this chapter in-troduces the basic models that we will use throughout the thesis. For each model, we try to convey its informal behavior before we formally define its semantics.
As our models evolve in time, their behaviors are described as the outcomes of com-pound random experiments which can be formalized in an infinite-dimensional product space, where each dimension corresponds to a fixed time-point. We refer to Sec. 2.5.4 for the probability theoretic construction of such spaces. The underlying mathematical tool that allows us to reason about these models formally, is called astochastic process.
Accordingly, this overview chapter starts by shortly introducing the concepts of dis-crete and continuous stochastic processes. Then we discuss the special cases of disdis-crete- discrete-and continuous-time Markov chains in more detail, as their properties are essential for the class of models that we are confronted with. Most of the material that we provide here is based on the standard textbook [Kul95], which provides an excellent introduction to Markov processes.
In the second part of this chapter, we introduce nondeterminism in Markov chains and thereby obtain discrete- and continuous-time Markovdecision processes, where the latter are at the core of our studies in the forthcoming chapters. Discrete-time Markov de-cision processes are discussed in the textbook [Put94] in great detail. Moreover, [Put94]
contains an introduction to continuous-time Markov decision processes in Chapter 11.
3.1 Stochastic processes
As we aim at an algorithmic verification mechanism for nondeterministic and stochastic systems, we are mostly interested in the subclass of stochastic processes that have a finite state space, as they can be stored in finite memory. Within the scope of this thesis, we therefore restrict to systems that have a finite state space. In this setting, a stochastic process is defined as follows:
Definition 3.1 (Stochastic process). A stochastic process on a finite state spaceS is a collection{Xt}t∈Tof random variables Xt, where the parameter t ranges over a parameter setT. Each Xttakes on values that are in the finite state spaceS.
Usually, the parametert is interpreted as time; accordingly, for t ∈ T, the value of Xtis the state that is occupied by the stochastic process at timet. In case of a discrete stochastic process, the parameter set T is a subset of N (finite or countably infinite), whereas for continuous stochastic processes, the set T is a connected subset of R≥0. To ease notation, we use the natural numbers to refer to the discrete time parameters and the nonnegative reals for the continuous time domain.
To describe one possible evolution of a stochastic process, letπ ∶ T → S be a function such thatπ(t) ∈ S describes the state that the stochastic process occupies at time t. Each such function describes a trajectory of the underlying stochastic process; in mathematics, eachπ ∶ T → S is called a sample path of the stochastic process.
Now, a stochastic process {Xt}t∈T evolves randomly along one of its sample paths.
Therefore, a sample path can be seen as one possible outcome of the compound random experiment that is associated with the entire stochastic process. To link this view of a stochastic processes to the measure theoretic results of the previous chapter, we define thesample space of a stochastic process as the collection of all its sample paths, i.e. we set Ω ={π ∶ T → S} = ST. Accordingly, each random variableXtis a measurable function Xt ∶ (Ω, F) → (S, 2S), where F denotes the σ-field generated by the measurable cylin-ders1. Hence, given a sample pathπ ∶ T → S, the random variable Xtmapsπ to the state that is occupied onπ at time t, i.e. Xt(π) = π(t).
Now, let P be a probability measure on the measurable space(Ω, F). If we are inter-ested in the probability that at timet ∈ T, the stochastic process is in state s ∈S, we have to compute the probability measure of the set of all sample paths that are in states at timet. Formally, this probability can be denoted as follows:
P({Xt=s}) = P (Xt−1(s))
=P({π ∶ T → S ∣ Xt(π) = s})
=P({π ∶ T → S ∣ π(t) = s}) .