The reciprocal class of a Markov chain, a basic example

Then hP is a processes in R(P) that has endpoint distribution µ1.

• Let h be defined as above, then the initial distribution of the h-transform changes as follows.

Since hP01(dxdy) = h(y)P01(dxdy) we can compute the marginal of the initial state. For any B ∈ B(R^d) we have

hP0(B) = Z

B×R^dh(y)P01(dxdy)

= Z

B×R^d

ρ1(y) Z

R^d

φ(z, y)µ0(dz)

!−1

φ(x, y)µ0(dx)dy

= Z

B







 Z

R^d

ρ1(y)φ(x, y) R

R^dφ(z, y)µ0(dz)dy







 µ0(dx).

Note that for a pinned reference process the initial law does not change, we have P^x₀ = δ^{x}

and hP^x₀= δ^{x}.

4.4. The reciprocal class of a Markov chain, a basic example.

In this paragraph we transfer the concepts of reciprocal processes and reciprocal classes to the discrete time setting for processes with finite state space. To justify this transfer we briefly explain why the dynamical properties of a Markov chain are similar to the properties of a Markov process in D(I, R^d) with deterministic jump-times that are constant between the jumps. The concept of discrete time reciprocal processes was first introduced by Bernstein [Ber32]. More recently authors have been interested in the classification of such “reciprocal chains” having Gaussian marginals, see e.g. Levy, Frezza, Krener [LFK90], Levy, Ferrante [LF02] and Carravetta [Car08].

The toy examples at the end of this paragraph then give a first impression of problems that arise, when comparing the bridges of pure jump Markov processes in continuous time.

4.4.1. Embedding of Markov chains using deterministic jump-times.

Let Y= (Yi)_0≤i≤mbe an arbitrary discrete time process with finite state space {y₁, . . . , yn} ⊂ R^ddefined on an arbitrary probability space (Ω, A, P).

Definition 4.20. The process Y is Markov if for any 0 ≤ i ≤ m theσ-fields σ(Y0, . . . , Yi) and σ(Yi, . . . , Ym) are independent given Y_i. The process Y is reciprocal if for any 0 ≤ i ≤ j ≤ m the σ-fields σ(Y0, . . . , Yi, Yj, . . . Ym) andσ(Yi, . . . , Yj) are independent given Y_i, Yj.

Let us justify the above definitions by an embedding: We fix any 0 < t1 < · · · < tm <

1 throughout this paragraph. Using these as deterministic jump-times, we define the continuous time c`adl`ag process Y^(m)as follows:

(4.21) Y^(m)_t :=



















Y₀, for t ∈ [0, t1), Y1, for t ∈ [t1, t2), ...

Y_m, for t ∈ [tm, 1].

The process Y^(m) is a c`adl`ag process that jumps m-times at the deterministic time points t₁, . . . , tmand is constant in between the jumps. Moreover Y^(m) :Ω → D(I, R^d) is measur-able. Define PY := P ◦ (Y^(m))⁻¹.

Lemma 4.22. The process Y = (Yi)0≤i≤m is Markov resp. reciprocal if and only if PY has the Markov property resp. the reciprocal property on D(I, R^d).

Proof. Let F(X)= f (Xs1, . . . , Xs_k) ∈ S_d. By definition there exist {i1, . . . , ik} ⊂ {1, . . . , m} such that F(Y^(m)) = f (Yi1, . . . , Yik) and for arbitrary t ∈ I there exists i ∈ {1, . . . , m} such that Y^(m)_t = Yi. Then

EY(F(X) | F_[0,t])= E(F(Y^(m))|σ(Y^(m)s , s ≤ t)) = E( f (Yi1, . . . , Yi_k)|σ(Y1, . . . , Yi)) and by a similar computation

EY(F(X)|X_t)= E( f (Yi1, . . . , Yik)|Y_i).

The claimed equivalence concerning the Markov property is now immediate. To proof the equivalence of the reciprocal property for the discrete and continuous time models one

advances in the same fashion. 

All results from § 4.1, § 4.2 and § 4.3 now apply to the setting of a discrete time process Y. In particular the definition of the reciprocal class of a Markov chain is again that of a mixture over the bridges.

Corollary 4.23. Let Y be a Markov chain and ˜Y = ( ˜Yi)_0≤i≤manother {y₁, . . . , yn}-valued random process. Then PY˜ ∈ R(PY) if and only if the following decomposition holds:

(4.24) P( ˜Y ∈ . ) = Xn i,j=1

P(Y ∈ . |Y0= yi, Ym = yj)P( ˜Y0 = yi, ˜Ym= yj).

Proof. Just use the same identification of random variables as in the proof of Lemma

4.22. 

We define the reciprocal class of a given Markov chain as follows.

Definition 4.25. Let Y be a Markov chain. Then we define the reciprocal class R(Y) as the collection of all {y₁, . . . , yn}-valued processes ( ˜Y_i)_1≤i≤mthat have the same bridges

(4.26) P( ˜Y ∈ . | ˜Y0 = yi, ˜Ym = yj)= P(Y ∈ . |Y0= yi, Ym = yj) holds P( ˜Y₀∈ . , ˜Ym ∈ . )-a.s.

4.4.2. The reciprocal class of a time-homogeneous Markov chain.

Let (Y_i)_0≤i≤mbe a time-homogeneous Markov chain on the finite state space {y₁, . . . , yn} ⊂ R^d. Denote the initial law and transition matrix by

µ0({y_i}) := P(Y0= yi) for i ∈ {1, . . . , n}, and (pi j)_1≤i,j≤n:=

P(Y₁= yj|Y₀ = yi)

1≤i,j≤n. The reciprocal class of Y is defined by the identity of bridges in (4.26). Let us now study the bridges of Y for various m, n ∈ N (the number of “jumps” and the number of different states).

Example 4.27. Let the number of states n ∈ N be arbitrary, m = 2 and Y, ˜Y be two time-homogeneous Markov chains. We compute

P(Y₁= yk|Y₀= yi, Y2= yj)= p_ikp_{k j} Pn

l=1p_ilp_{l j}, and P( ˜Y1 = yk|Y˜₀= yi, ˜Y2= yj)= ˜p_ik˜p_{k j} Pn

l=1˜p_il˜p_{l j}, where the denominator is just the two-step probability of going from i to j. Define pi j(2) := P(Y2= y_j|Y₀= yi), then the equality of the bridge between y_iand y_jis given if and only if

p_ikp_{k j}

p_{i j}(2) = ˜p_ik˜p_{k j}

˜p_{i j}(2), ∀k ∈ {1, . . . , n}.

The condition found in the above example is easily extended to arbitrary m ∈ N.

Corollary 4.28. Let n, m ∈ N and Y, ˜Y be two time-homogeneous Markov chains. Then ˜Y ∈ R(Y) if and only if for all 1 ≤ i, j ≤ n we have

(4.29) p_ik1p_k1k2· · ·p_km−1j

p_{i j}(m) = ˜p_ik1˜p_k1k2· · ·˜p_km−1j

˜p_{i j}(m) , ∀1 ≤ k1, . . . , km−1 ≤n, where p_{i j}(m) := P(Ym= yj|Y₀ = yi).

Proof. This follows directly from the definition of the reciprocal class through equality of

the bridges. 

In the rest of this paragraph we compare the bridges of different time-homogeneous Markov chains Y and ˜Y in examples, using the comparison result of Corollary 4.28.

Example 4.30. We study a simple message transmission model, a Markov chain that switches between two states {y₁, y2}, where the transition probabilities are given by

P(Y₁ = yj|Y₀ = yi)

1≤i,j≤2 = α 1 −α

1 −β β

! , 

P( ˜Y₁= yj|Y˜₀ = yi)

1≤i,j≤2 = α˜ 1 − ˜α 1 − ˜β β˜

! , whereα, β, ˜α, ˜β ∈ (0, 1).

y1 y2

1 −β

1 −α

α β

We use the localization result of Corollary 4.15 and transfer it into the discrete time setting by Lemma 4.22 to get a necessary condition for the equality of the bridges: For any lengths m ∈ N of the Markov chain, it is necessary that already the bridges for m= 2 of Y and ˜Y are identical. Thus we have to find conditions onα, β, ˜α, ˜β such that

P(Y₁= y1|Y₀= y1, Y2= y2)= p₁₁p₁₂

p₁₁p₁₂+ p12p₂₂ = α(1 − α) α(1 − α) + (1 − α)β

=! α(1 − ˜α)˜ α(1 − ˜α) + (1 − ˜α) ˜β˜ .

This can only be the case if there existsδ > 0 with α˜ α = β˜

β = δ.

Another necessary condition for the equality of all bridges is P(Y₁ = y1|Y₀ = y1, Y2= y1)= p₁₁p₁₁

p11p11+ p12p21 = α²

α²+ (1 − α)(1 − β)

=! (δα)²

(δα)²+ (1 − δα)(1 − δβ), which is equivalent to the quadratic equation

δ²(1 −α)(1 − β) = (1 − δα)(1 − δβ) ⇔ δ²(1 −α − β) + δ(α + β) − 1 = 0.

If we assume thatα, β ≈ 1, then the positive solutions to this quadratic equation are δ_1,2= α + β ± p(α + β)²− 4(α + β − 1)

2(α + β − 1) = (α + β) ± (α + β − 2) 2(α + β − 1) , such that

δ1 = 1, or δ2= 1 α + β − 1 are solutions. But it is easy to see, thatδ2is too large, since in this case

α =˜ α

α + β − 1 > 1 ⇔ 1 > β,

which was an assumption. Thus the reciprocal classes of Y and ˜Y only coincide ifα = ˜α and β = ˜β.

Example 4.31. Let m, n ∈ N be arbitrary and let the time-homogeneous Markov chains Y and ˜Y have the transition probabilities p_ii= α, p_i,i+1 = 1 − α and ˜pii= ˜α, ˜p_i,i+1= 1 − ˜α for 1 ≤ i ≤ n − 1 and pnn = ˜pnn = 1.

y₁ 1 −α y₂ α

y₃ 1 −α

α α

· · ·

1 −α 1 −α y_n−1 α

y_n 1 −α

1

We may interpret this as a comparison of birth processes with absorption in n. Let us first compare the bridges from y_ito y_jfor i ≤ j< n and k := j − i ≤ m. Condition (4.29) is

(4.32) α^m−k(1 −α)^k

m k

α^m−k(1 −α)^k

=! α˜^m−k(1 − ˜α)^k

m

k
˜α^m−k(1 − ˜α)^k, these bridges coincide. But note that for the bridge from yn−1to ynwe have

P(Y1= yn|Y0= yn−1, Ym = yn)= 1 −α

P(Ym = yn|Y0= yn−1)

=! 1 − ˜α

P( ˜Ym= yn|Y˜₀ = yn−1). The denominator is easily computed as P(Y_m = yn|Y₀= yn−1)= P^m−1_k=0 α^k(1−α). Thus R(Y) = R( ˜Y) if and only ifα = ˜α, even if most bridges coincide in this example: R(Y)∩R( ˜Y) contains all elements Y of either reciprocal class with P( ¯¯ Ym = yn)= 0.

Example 4.33. Next we consider a random walk on a circle moving in one direction: Let p_ii= α, p_i,i+1 = 1 − α and ˜pii = ˜α, ˜pi,i+1 = 1 − ˜α for 1 ≤ i ≤ n − 1 and pnn = α, pn1 = 1 − α and ˜pnn = ˜α,

˜p_n1= 1 − ˜α.

y₁ 1 −α y₂ α

y₃ 1 −α

α α

· · ·

1 −α 1 −α y_n−1 α

yn

1 −α α

1 −α

Both chains are following the circle y₁ →y2→ · · · → yn→y₁ →. . . . To compare the bridges we have to take into account the difference between the number of states n and the number of jumps m.

If m < n we can argue as in the preceding Example 4.31 that ˜Y ∈ R(Y) for all ˜α ∈ (0, 1). But if n ≤ m< 2n, then condition (4.29) for the probability of going from yk exactly one time around the

“circle” by taking n jumps first and then staying in y_kis

P(Y₁= y_k+1, . . . , Yn−1= yk−1, Yn= yk, Y_n+1= yk, . . . , Ym = yk|Y₀= yk, Ym = yk)

= αⁿ(1 −α)^m−n

m!

(n−1)!αⁿ(1 −α)^m−n+ (1 − α)^m

=! α˜ⁿ(1 − ˜α)^m−n

m!

(n−1)!α˜ⁿ(1 − ˜α)^m−n+ (1 − ˜α)^m. Put c := _(m−n)!^n! , then the above condition is equivalent to

(_1−α^α )ⁿ

c(_1−α^α )ⁿ+ 1 = (_{1− ˜}^α˜_α)ⁿ

c(_{1− ˜}^α˜_α)ⁿ+ 1 ⇔ 1 −α

α = 1 − ˜α α˜ , which can only be the case ifα = ˜α.

Example 4.34. Let p_i,i−1 = α, p_i,i+1 = 1 − α and ˜p_i,i−1 = ˜α, ˜p_i,i+1 = 1 − ˜α for some α, ˜α ∈ (0, 1) with boundary reflection p₁₂= ˜p12 = 1 and p_n,n−1 = ˜p_n,n−1= 1. This is a birth and death process with re-insertion of an individual if the population died out and certain death of one individual if the population has reached the number of n individuals.

y₁ y₂

1

α

y₃ 1 −α

α

· · · α 1 −α

y_n−1 1 −α

α

y_n 1 −α

1

Both chains are (asymmetric) finite random walks on a domain with reflecting boundary. We can easily check that the reciprocal classes of Y and ˜Y do not coincide if α , ˜α using the localization result of Corollary 4.15: Assume that n ≥ 4 and m ≥ 3, then

P(Y₁ = y1, Y2= y2|Y₀ = y2, Y3= y3)= α(1 − α)

α(1 − α) + 2α(1 − α)² = 1 3 − 2α

=! 1 3 − 2 ˜α, where the denominator is computed by

P(Y0 = y2, Y3= y3) = P(Y0= y2, Y1= y1, Y2= y2, Y3 = y3) +P(Y0= y2, Y1= y3, Y2= y4, Y3 = y3)

+P(Y0= y2, Y1 = y3, Y2= y2, Y3= y3).

But this necessary condition is only satisfied ifα = ˜α.

5. The reciprocal classes of Brownian diffusions

In this section we present a theory of reciprocal classes of Brownian diffusions, which are defined in Definition 5.6. The objective is to give a synopsis of results by different authors in a uniform setting in preparation of the treatment of reciprocal classes of jump processes in Sections 6 and 7. We are able to improve some of these results, and present a new approach to euclidean quantum mechanics that is based on analogies to classical mechanics.

Since all processes treated in this section have continuous sample paths, we present our results in the frame of the canonical setup on C(I, R^d), the space of R^d-valued continuous functions on I.

In the presentation of § 5.1, § 5.2 and § 5.4 we closely follow the articles by Rœlly and Thieullen [RT02], [RT05]. They characterized the reciprocal class of a Brownian motion with drift by a duality formula. Their characterization is based on reciprocal invariants in the sense of Clark [Cla90], we present his result in § 5.3.

We present two applications. In § 5.5 we introduce an optimal control approach to determine the motion of a charged particle in an electromagnetic field. We define the dynamics of the motion of a charged particle that is immersed in a thermal reservoir and under the influence of an external electromagnetic field. The effective motion of the particle is then described by the solution of a stochastic optimal control problem under boundary constraints in the sense of Wakolbinger [Wak89] and Dai Pra [DP91]. Following their result we provide a new interpretation of the duality formula as a stochastic Newton equation. Our approach is closely related to Zambrini’s euclidean quantum mechanics, see e.g. Zambrini [Zam85], but also L´evy and Krener [LK93] and Krener [Kre97].

A second application is the identification of the behavior of the reciprocal class of a Brownian diffusion under time reversal in § 5.6. This approach by duality formula is an alternative to the computations of Thieullen, who presented a similar result in [Thi93]. In combination with the results from § 5.5 we see, that the dynamics of a charged particle in a thermal reservoir are time reversible in the same sense as the deterministic dynamics of a charged particle in an electromagnetic field.

In document Reciprocal classes of Markov processes (an approach with duality formulae) (Page 58-63)