Entropy Rate Estimation for Markov Chains with Continuous Distributions
Puning Zhao and Lifeng Lai
Abstract—Entropy rate estimation has a broad range of applications such as bioinformatics, feature clustering etc.
Although there are many existing work on the estimation of entropy rate for Markov chains with discrete distributions, the understanding for the entropy rate estimation of Markov chains with continuous distributions is limited. In this paper, efficient methods for estimating the entropy rate for Markov chains with continuous distributions are proposed. Moreover, we derive bounds on the convergence rate of the proposed entropy rate estimators.
Index Terms—Entropy rate, estimation, kNN, Markov chains
I. INTRODUCTION
Entropy rate is an important quantity in information theory and statistics. It can be understood as the fundamental limit of predicting the next step in a stochastic process. Corre- spondingly, the estimation of entropy rate for a stochastic process, especially a stationary Markov chain, has been used in a wide variety applications. For example, it can be used in bioinformatics [1–5] for analyzing DNA sequences, feature clustering and image registration [6], blind source separation [7, 8], economics [9], and many other signal processing related fields [10, 11].
The estimation of the entropy rate for Markov chains with discrete distributions has been discussed in some recent interesting works [12–15]. A simple and intuitive method is plug-in method, in which one estimates the stationary dis- tribution and the transition probability matrix first and then calculates the entropy rate. [13] proved that this estimator converges almost surely and is asymptoticly normal. [14]
provided a finite sample bound on the estimation error. These results show that the simple plug-in method is efficient if the alphabet size is finite and fixed. If the alphabet size grows with the sequence length, then the simple plug-in method is no longer optimal. In [15], a new method was proposed, which can handle the case with large state space.
This method estimates the conditional entropy given each
Puning Zhao and Lifeng Lai are with Department of Electrical and Computer Engineering, University of California, Davis, CA, 95616. Email:
{pnzhao,lflai}@ucdavis.edu. This work was supported by the National Science Foundation under grants CCF-17-17943, ECCS-17-11468, CNS- 18-24553 and CCF-19-08258.
previous state based on some efficient entropy estimators for identical and independently distributed (i.i.d) samples, such as those proposed in [16, 17]. It exhibits clear advantage over the simple plug-in estimator, since it estimates the conditional entropy directly, instead of estimating the full transition matrix. Such an improvement is significant if the alphabet size is large. Moreover, it is shown in [15]
that this new method achieves the minimax optimal sample complexity.
Despite that the entropy rate estimation has been widely discussed for discrete distributions, the previous methods and the theoretical analysis can not be straightforwardly gener- alized to continuous distributions. For discrete distributions, for all samples at a fixed state s, their next states are i.i.d conditional on s. However, for continuous distributions, we can not use this property, because we can not expect that there are a large number of samples located at the same state. As a result, it is impossible to find some states that are conditionally i.i.d, which makes the analysis much harder.
In this paper, our goal is to estimate the entropy rate for continuous distributions. In particular, we propose two competitive methods to estimate the entropy rate of Markov Chains with continuous distributions.
For the first method, our design is based on the fact that for stationary and homogeneous Markov chain, the entropy rate equals to the conditional entropy of a state given its previous state. Therefore, it is natural to design a method based on the combination of two Kozachenko-Leonenko (KL) entropy estimators [18], in which one of them estimates the joint entropy, while the other estimates the marginal entropy. The final estimate of the entropy rate can then be calculated by subtracting the marginal entropy estimate from the joint entropy estimate. We name this method as 2KL entropy rate estimator. The 2KL method is simple to use with little parameter tuning, and has good empirical perfor- mance. However, it is very difficult to rigorously characterize the convergence rate of this 2KL method. The analysis techniques for the KL entropy estimator [19–24] cannot be applied for the analysis of the proposed 2KL entropy rate estimator. The main reason is that the existing techniques for analyzing the KL entropy estimator relies heavily on the fact that the available samples are independent, while
the samples obtained from the Markov chain case are not independent anymore.
To overcome the lack of rigorous convergence rate char- acterization of the 2KL entropy rate estimator, we propose another estimator that is amenable to analysis and has similar or better performance than the 2KL estimator. The main idea of the new estimator is to divide the support set into bins.
For each bin, we find all samples falling in this bin and then find their next states. After that, we apply the KL entropy estimator, on these states whose previous steps are all in the same bin. We call this as Bin-KL entropy rate estimator.
The Bin-KL entropy rate estimator shares some similarity with that in [15], since both [15] and our method estimates the conditional entropy directly instead of estimating the full probability mass function (pmf) or probability density function (pdf). However, unlike discrete distributions, for continuous distributions, the states whose previous steps are within the same bin are not i.i.d or conditional i.i.d given the previous state, since their distributions are still slightly different due to different locations of the previous states, even if those previous states are in the same bin. As a result, the analysis of the new proposed method becomes harder. For this estimator, we are able to provide a new analysis to show that this method is consistent, and derive its convergence rate. Our analysis uses some techniques from previous works on the KL entropy estimator [21, 23, 24].
Consider that the samples are no longer i.i.d, we modify the previous analysis and carefully bounded the effect of the mutual dependence between each steps. To the best of our knowledge, our work is the first attempt to propose a method to estimate the entropy rate of Markov chains with continuous distributions, and bound its convergence rate.
The remainder of this paper is organized as follows. In Section 2, we introduce the basic background as well as the notation used in this paper. In Section 3, we introduce the proposed entropy rate estimation methods. We then show the theoretical analysis in Section 4. Finally, we provide numerical examples and concluding remarks in Sections 5 and 6, respectively.
II. PRELIMINARIES
Consider a first order ergodic Markov chain X1, X2, . . ., in which Xi ∈ S, S ⊂ Rd is a compact set. Each Xi is a continuous random variable, is conditionally independent with X1, . . . , Xi−2given Xi−1. The entropy rate is defined as
¯h = lim
n→∞
1
nh(X1, . . . , Xn), (1) in which h(X1, . . . , Xn) is the joint differential entropy of X1, . . . , Xn:
h(X1, . . . , Xn)
= −
Z
f (x1, . . . , xn) ln f (x1, . . . , xn)dx1. . . dxn. The joint pdf f (x1, . . . , xn) is usually unknown in prac- tice. We need to estimate the entropy rate from samples.
Suppose we are given a realization of the Markov chain, which is a sequence with length N , i.e. X1, . . . , XN, in which X1is an arbitrary initial state. Our goal is to estimate the entropy rate ¯h based on these N samples.
In this paper, we assume that the Markov chain is time homogeneous, which means that there exists a transition kernel p, such that p(x, ·) = fi+1(·|Xi = x) for all i = 1, 2, . . . and x ∈ S, in which fi+1(·|Xi = x) is the conditional pdf of Xi+1 given Xi = x. Denote π as the pdf of the stationary distribution, which satisfies R π(x)p(x, y)dx = π(y).
Then we have
¯h = lim
n→∞
1
nh(X1, . . . , Xn)
= lim
n→∞h(Xn|Xn−1) = Z
π(x)h(p(x, ·))dx, (2) in which h(p(x, ·)) = −R p(x, y) ln p(x, y)dy. is the con- ditional entropy of a state given the previous state.
III. PROPOSEDMETHODS
In this section, we propose two methods, called 2KL method and Bin-KL method respectively, to estimate the entropy rate based on the expression (2).
A. 2KL Method
For a time homogeneous and uniformly ergodic Markov chain, we have
¯h = lim
n→∞
1
nh(X1, . . . , Xn) = lim
n→∞h(Xn|Xn−1). (3) Therefore, we can estimate the entropy rate by estimating the conditional entropy of next state given the previous state.
Define Zi = (Xi, Xi+1) for i = 1, . . . , N − 1, then ¯h =
n→∞lim[h(Zn) − h(Xn)]. h(Zn) and h(Xn) slightly change over n. However, they will converge as n increases. Hence, a possible idea is to use two entropy estimators to estimate h(Z) using Zi, i = 1, 2, . . . , N − 1 and to estimate h(X) using Xi, i = 1, 2, . . . , N separately, and then calculate the estimated conditional entropy.
The most common method for estimating the entropy for continuous random variable is the KL estimator, which cal- culates the differential entropy based on k nearest neighbor distances. If the nearest neighbor distances are large, then the random variable has a high differential entropy, and vice versa. It was first proposed in [18], and was then analyzed
in [20, 21, 23–26]. In our case, the expressions of the KL estimates for h(X) and h(Z) are
ˆh(X) = −ψ(k) + ψ(N ) + ln cd+ d N
N
X
i=1
ln Xi, (4)
ˆh(Z) = −ψ(k) + ψ(N − 1) + ln c2d+ d N
N
X
i=1
ln Zi, (5)
in which ψ is the digamma function, ψ(t) = Γ0(t)/Γ(t), Γ is the Gamma function, Γ(t) =R ut−1e−udu. cdand c2dare the volumes of the d-dimension and 2d-dimension unit balls respectively. If we use `2metric, then cd= πd/2/Γ(d/2+1).
c2d can be defined similarly. Xi is the distance of Xi
to its k nearest neighbors among {X1, . . . , XN}, and Zi is the distance of Zi to its k nearest neighbors among {Z1, . . . , ZN −1}. We call this method as 2KL method, as it is a combination of two KL estimators.
This method is simple to use since the only parameter we need to tune is k. According to the analysis of KL estimators in [23, 24, 27], the optimal k is fixed with respect to sample size N . Furthermore, we will show in Section V that the 2KL method has good empirical performance. However, it is challenging to rigorously characterize the convergence property of this method. In particular, although there are many existing works that analyze the performance of KL entropy estimators for the case with i.i.d data under vari- ous assumptions [19–24], these analysis can not be easily generalized to analyze the 2KL entropy rate estimator for the Markov chains because samples are no longer i.i.d. The challenge with the rigorous performance analysis of the 2KL method motivates us to design an alternative entropy rate estimator that is amenable to analysis and has similar or better empirical performance as the 2KL method in the next subsection.
B. Bin-KL Method
To address the lack of rigorous performance analysis of the 2KL method issue, we propose an alternative entropy rate estimator that has rigorous convergence characterization and has similar or better empirical performance. We name this new method as Bin-KL method. The main idea of the Bin-KL method is to estimate π(x) and h(p(x, ·)). However, since the distribution of each Xi is continuous, we can not estimate them at every x. Therefore, we divide the support S into m bins, denoted as B1, . . . , Bm, such that each bin is sufficiently small. Assume that the transition kernel p and the corresponding pdf of the stationary distribution π is continuous, then they will be sufficiently close within each bin. Hence, we can let the estimation of π and ˆh(p(x, ·)) to be the same in each bin.
π(x) can then be estimated by ˆπ(x) = n(Bj)/N V (Bj), in which V (Bj) is the volume of Bj, and
n(Bj) =
N −1
X
i=1
1(Xi∈ Bj) (6)
is the number of samples falling in Bjamong the first N −1 samples. This means that π(x) can be simply estimated by the fraction of all samples falling in Bj.
For the estimation of h(p(x, ·)) for x ∈ Bj, we denote Ij= {i|Xi−1∈ Bj}, (7) which is a set of indices of samples whose previous step belong to Bj. Obviously, the cardinality of Ij is |Ij| = n(Bj). Denote (Xj1, Yj1), . . . , (Xj,n(Bj), Yj,n(Bj)) as a random permutation of (Xi−1, Xi) for i ∈ Ij. Note that the distributions of Yj1, . . . , Yj,n(Bj) are close to each other, since their previous steps are all in Bj, and the previous steps are close to each other. With this observation, we can use the KL entropy estimator to estimate h(p(x, ·)) for x ∈ Bj. The estimated value of h(p(x, ·)) is the same for all x ∈ Bj. Therefore, we use ˆhj to denote such estimated result. If n(Bj) ≥ k, then
ˆhj = −ψ(k) + ψ(n(Bj)) + ln cd
+ d
n(Bj)
n(Bj)
X
l=1
ln jl, (8)
in which jl is the distance from Yjl to its k-th near- est neighbor among Yj1,. . .,Yj,l−1, Yj,l+1,. . .,Yj,n(Bj). If n(Bj) < k, then the KL entropy estimator can not be used.
In this case, we just set ˆhj = 0.
Setting ˆhj = 0 when the number of samples within Bj
is less than k will inevitably cause some estimation bias.
However, we can ensure that the number of bins grows slower than the total sample size N , then the expected number of samples within each bin will also grows with N . As a result, the probability that n(Bj) < k becomes smaller with the increase of N . This ensures that the additional bias caused by setting ˆhj= 0 converges to zero.
Combining these two steps, the proposed Bin-KL entropy rate estimator is written as
ˆ¯
h =
Z ˆ
π(x)ˆh(p(x, ·))dx
=
m
X
j=1
Z
Bj
ˆ
π(x)ˆh(p(x, ·))dx
=
m
X
j=1
n(Bj) N
hˆj. (9)
The Bin-KL method has two design parameters, m and k. The optimal choice of m grows with sample size N . We
will characterize the optimal growth rate of m with sample size N in the convergence analysis section. On the contrary, the optimal k does not grow with N . Therefore, k can be selected as a fixed value. Although the selection of k can partially affect the bias and variance of this estimator, it does not impact their convergence rates.
We now compare the Bin-KL method and the 2KL method. As will be shown in the numerical simulation, both methods have good empirical performance. The mean square errors of both methods converge to zero. The performances of these two methods depend on the distributions, but they generally have comparable empirical performances. A major difference is that we have rigorous convergence analysis for the Bin-KL method (shown in Section IV), while for the 2KL method, we are not able to provide such analysis due to difficulties discussed in Section III-A.
Another aspect to compare is the time complexity. We discover that the Bin-KL method usually requires less time than the 2KL method. This is because the k nearest neighbor search has a higher time complexity than assigning bins.
For the Bin-KL method, the KL estimator is used for every bin, in which the number of samples is much less than the total sample size N . However, for the 2KL method, the KL estimator is used on the whole dataset. As a result, the 2KL method is slower than the Bin-KL method, especially when the sequence length is large.
Finally, we would like to remark that the 2KL method has a broader range of applications. It can be used for both distributions with bounded and unbounded support, while the Bin-KL method can only be used on distributions with bounded support. If the support is unbounded, the number of bins will be infinite. To solve this problem, it is possible to improve the bin method so that it can adaptively divide the support into bins with different sizes, such that the bin size is larger where the pdf of the stationary distribution is low.
IV. CONVERGENCEANALYSIS
In this section, we provide a theoretical analysis of the convergence rate of the Bin-KL entropy rate estimator (9).
Our analysis is based on the following assumptions.
Assumption 1. We make the following assumptions:
(a) The support setS has finite volume VS, finite surface areaAS and finite diameterD;
(b) The conditional pdf is both upper and lower bounded, i.e.fL≤ p(x, y) ≤ fU for two constantsfL andfU.
(c)p(x, y) is L-Lipschitz;
(d) There exist two constant R and α, such that for all x ∈ S and r ≤ R, we have
V (B(x, r) ∩ S) ≥ αV (B(x, r)), (10)
in whichVS,AS,D, Cb,fL,fU,L, R are all finite positive constants, andα ∈ (0, 1).
We now comment on these assumptions. Assumption (a) is an assumption that is necessary for our bin splitting method. It is possible to design an adaptive bin splitting strategy to the estimate the entropy rate if the distribution of Xi has an unbounded support. However, in this paper, we focus on the case that S is bounded for simplicity.
Assumption (b) restricts the lower and upper bound of the transition probability. These bounds are important to calculate the convergence rate. Such assumption has already been made in similar works on the estimation of information theoretic functionals [23, 24, 27]. In Assumption (c), we assume that p(x, y) is Lipschitz in both x and y. The overall convergence of the mean square error may be faster if we assumes smoothness of p(x, y) with a higher order.
Assumption (d) is a regularity assumption on the shape of the support, which is satisfied by almost all common support sets. For example, Assumption (d) holds if the support set is convex or is the union of a finite number of convex sets.
We would like to remark that in previous works on the estimation of entropy rate for discrete distributions [14, 15], there are some assumptions about the uniform ergodicity of Markov chain as well as the corresponding mixing time, which indicates how fast the distribution converges to the stationary distribution. This assumption is necessary to get the convergence rate of the entropy rate estimator in [14, 15].
In our Assumptions (a)-(d), we do not state such assumption explicitly, because the uniform ergodicity can actually be derived from Assumption (b), which restricts the lower and upper bound of the transition kernel.
Based on these assumptions, we have the following the- orem regarding the convergence rate of the Bin-KL entropy rate estimator defined in (9).
Theorem 1. The mean square error of the Bin-KL entropy rate estimator can be bounded by
E[(ˆ¯h − ¯h)2] . m
N ln2N + m−d2 +m N
2d
. (11)
To optimize the convergence rate, we letm grow with N as m ∼
(
Nd+2d if d ≤ 2
N12 if d > 2. (12) With this choice, the convergence rate of the mean square error of the Bin-KL estimator becomes
E[(ˆ¯h − ¯h)2] . (
N−d+22 ln2N if d ≤ 2
N−1d if d > 2. (13) The main idea of the proof of Theorem 1 is to bound the estimation error of ˆπ and ˆhj separately. The main difficulty is that ˆhj are not i.i.d for different j, thus the overall bound
of the estimator can not be obtained by simply bounding the bias and variance of each ˆhj. To cope with this problem, we designed a new approach that is different from the traditional analysis on KL estimator [23, 24]. The detailed proof can be found in Appendix A.
Theorem 1 shows the convergence rate of the Bin-KL entropy rate estimator. An intuitive understanding of (11) is that the first term comes from the variance of ˆhj defined in (8), the second term comes from the bias of ˆhj, while the third term comes from the bias and variance of ˆπj.
V. NUMERICALEXAMPLES
In this section, we provide numerical simulations to validate our theoretical analysis. We use the following dis- tributions as the ground truth. For all of the distributions used in this section, X1 ∼ U([0, 1]d), in which U denotes uniform distribution, d is the dimensionality. In this section, we use d = 1, 2, 3. For each i = 2, 3, . . .,
X0i−1 = Xi−1+ Zi−1, (14) Xij = X0i−1,j− bX0i−1,jc, (15) in which b·c is the floor function. By operation (15), it is ensured that all Xi’s are within [0, 1]d. The distribution of Z is different for different cases. In the first case, Z ∼ U([−0.1, 0.1]d). In the second case, Z ∼ U([−0.3, 0.3]d).
In the third case, each component Zj follows a distribution with pdf fZj(zj) = 1.5−zj, for zj ∈ [0, 1] and j = 1, . . . , d.
In the fourth case, each component Zj follows a Gaussian distribution with µ = 0, σ = 0.2.
From the above construction, X1, . . . , XN is a Markov chain, in which each random variable is supported on [−1, 1]d. The entropy rates of these four distribu- tions are ¯h1 = ln(0.2)d, ¯h2 = ln(0.6)d, ¯h3 =
1
2−18ln 2 −98ln32 d, ¯h4 = −0.2249d, in which ¯h4 is calculated numerically.
In all of the trials, we fix k = 3 for both Bin-KL method and 2KL method. Moreover, for the Bin-KL method, we use m =
bN13c if d = 1
max{m0|(m0)1d ∈ N, m0≤√
N } if d ≥ 2. (16) Such setting is based on (12), which requires m ∼ N1/3 for d = 1 and otherwise m ∼√
N . The reason that we require m1/d to be an integer is that we would like the number of bins is the same in each dimension, so that each bin is a regular hexahedron.
Now we show the plots of the mean square error vs the sample size. Both two coordinates are set to be log scale, so that we can have a clear view of the convergence rates. In each plot, we compare the results of the Bin-KL method and the 2KL method. Each point on the curves in the figures are averaged from T = 500 trials. The sample sizes range from
102 103 104 105
N 104 103 102 101
Mean square error
Bin-KL 2KL
(a) d = 1
102 103 104 105
N 101 100
Mean square error
Bin-KL 2KL
(b) d = 2
102 103 104 105
N 100 101
Mean square error
Bin-KL 2KL
(c) d = 3
Fig. 1. Plots of the mean square error of the Bin-KL method and the 2KL method for the first distribution.
102 103 104 105
N 104 103 102
Mean square error
Bin-KL 2KL
(a) d = 1
102 103 104 105
N 101 100
Mean square error
Bin-KL 2KL
(b) d = 2
102 103 104 105
N 100
Mean square error
Bin-KL 2KL
(c) d = 3
Fig. 2. Plots of the mean square error of the Bin-KL method and the 2KL method for the second distribution.
100 to 100, 000. The results are shown in Figures 1, 2, 3, 4 respectively for the four different cases discussed above.
From Figures 1, 2, 3 and 4, it can be observed that the mean square errors of both Bin-KL method and 2KL method converge to zero as the sequence length N increases. Both methods have comparable performance, with each one being slightly better than the other one depending on the underly distribution. For the first and the second distribution, the 2KL method performs better, while for the third and fourth distribution, the Bin-KL method performs better.
Moreover, it can be observed that the curves of 2KL method are smooth, while for the Bin-KL method, the curves are less smooth, especially when the dimension is
102 103 104 105
N 105 104 103 102
Mean square error
Bin-KL 2KL
(a) d = 1
102 103 104 105
N 102 101
Mean square error
Bin-KL 2KL
(b) d = 2
102 103 104 105
N 101
Mean square error
Bin-KL 2KL
(c) d = 3
Fig. 3. Plots of the mean square error of the Bin-KL method and the 2KL method for the third distribution.
102 103 104 105
N 105 104 103 102
Mean square error
Bin-KL 2KL
(a) d = 1
102 103 104 105
N 102 101 100
Mean square error
Bin-KL 2KL
(b) d = 2
102 103 104 105
N 101 100 101
Mean square error
Bin-KL 2KL
(c) d = 3 Fig. 4. Plots of the mean square error of the Bin-KL method and the 2KL method for the fourth distribution.
Distribution d Bin 2KL Theoretical rate 1
1 0.71 1.05 0.67
2 0.51 0.58 0.50
3 0.33 0.40 0.33
2
1 0.77 1.02 0.67
2 0.56 0.57 0.50
3 0.35 0.39 0.33
3
1 0.98 1.01 0.67
2 0.52 0.50 0.50
3 0.34 0.33 0.33
4
1 1.04 0.97 0.67
2 0.67 0.58 0.50
3 0.59 0.56 0.33
TABLE I
THE EMPIRICAL AND THEORETICAL CONVERGENCE RATES OFBIN-KL AND2KLMETHOD FOR THE ENTROPY RATE ESTIMATION.
high. This is because according to (16), m does not change continuously with N . When the sequence length N reaches some threshold, m suddenly changes, and thus the resulting mean square error changes abruptly. As a result, there are usually several turning points in the curve. This effect is especially obvious if the dimensionality is high.
Finally, we list the empirical convergence rates of the Bin- KL method and the 2KL method for different cases, and compare them with the theoretical rates. The empirical rates are calculated by finding the negative slope of the curves by linear regression, while the theoretical rates come from (13).
The results are shown in Table I, in which the theoretical rate is denoted as β if the mean square error converges with O(N−β) or O(N−βpoly(ln N )), in which poly denotes any polynomial.
From Table I, it can be observed that some empirical convergence rates agree with the theoretical rates, and other empirical results are actually faster than the theoretical one.
We explain such difference as following. The assumption we make is a relatively weak condition, and practical distri- butions may satisfy some stronger conditions. For example, we assume that the transition kernel p is Lipschitz, while actually p may be second order smooth, i.e. p may have bounded Hessian in the support. As a result, the real con- vergence rate of the mean square error can actually be faster than our theoretical prediction.
VI. CONCLUSION
In this paper, we have proposed two methods to estimate the entropy rate of Markov chains, in which each step follows a continuous distribution. The first method, the 2KL method, combines two KL entropy estimators directly.
This method is intuitively correct but hard to analyze. The second method, the Bin-KL method, is based on a hybrid of bin splitting and the KL entropy estimator. For this method, we have conducted a theoretical analysis to show the consistency of this method, and have derived its convergence rate. Numerical simulations have shown that both methods perform well for the distributions satisfying our assumptions,
and the convergence rate of the Bin-KL method agrees with our theoretical prediction.
APPENDIXA PROOF OFTHEOREM1
Recall that the entropy rate can be expressed as (2). For each bin Bj, pick a point cj∈ Bj, such that
Z
Bj
π(x)(h(p(cj, ·)) − h(p(x, ·)))dx = 0. (17) Since Bj is a connected set for all j, h(p(x, ·)) is continuous with respect to x, it is guaranteed that such cjexists. Define Pπ(Bj) =R
Bjπ(x)dx as the probability mass of Bj under the stationary distribution π. From (2) and (17), we have
¯h =
m
X
j=1
Z
Bj
π(x)h(p(cj, ·))dx
=
m
X
j=1
Pπ(Bj)h(p(cj, ·)). (18) Recall that (Xj1, Yj1), . . . , (Xj,n(Bj), Yj,n(Bj)) are defined as a random permutation of (Xi−1, Xi) for i ∈ Ij, in which Ij= {i|Xi−1∈ Bj}, we have
ˆ¯
h − ¯h =
m
X
j=1
n(Bj) N
ˆhj− Z
π(x)h(p(x, ·))dx
:= I1+ I2+ I3, (19)
in which I1 =
m
X
j=1
n(Bj)
N (ˆhj− E[ˆhj|Xj1, . . . , Xj,n(Bj)]), (20) I2 =
m
X
j=1
n(Bj)
N (E[ˆhj|Xj1, . . . , Xj,n(Bj)] − h(p(cj, ·))), (21) I3 =
m
X
j=1
n(Bj)
N h(p(cj, ·)) −
m
X
j=1
Pπ(Bj)h(p(cj, ·)). (22) Hence
E[(ˆ¯h − ¯h)2] = E[(I1+ I2+ I3)2]
≤ 3E[I12] + 3E[I22] + 3E[I32]. (23) Bound of E[I12]. Note that ˆhj are not mutually independent for different j. Therefore, we can not bound E[I12] by simply bounding the variance for each ˆhj. Instead, we obtain an upper bound of pn(Bj)(ˆhj − E[ˆhj|Xj1, . . . , Xj,n(Bj)]), which holds for all j with high probability. Recall that ˆhj is defined as
ˆhj = −ψ(k) + ψ(n(Bj)) + ln cd+ d n(Bj)
n(Bj)
X
l=1
ln jl. (24)
Now we define a truncation of jl:
ρjl= max{jl, rT j}, (25) in which rT j will be determined later, and
ˆhT j= −ψ(k) + ψ(n(Bj)) + ln cd+ d
n(Bj)ln ρjl. (26) From Lemma 20.6 in [20] and Lemma 11 in [23], each point Yjl is among the k nearest neighbors of at most kγdpoints in {Yj1, . . . , Yj,l−1, Yj,l+1, . . . , Yj,n(Bj)}, in which γd is the minimum number of cones with angle π/6 that can cover the space Rd. Moreover, from (25), ρjl ≥ rT j. And from Assumption (a), ρjl ≤ D. As a result, when ρjl for some l changes, ln ρjl will change no more than ln(D/rT j). If we replace Yl with Yl0, then the corresponding change of ˆhT j
is bounded by
ˆhT j(Yj1, Yj,n(Bj)) − ˆhT j(Yj1, . . . , Yjl0 , . . . , Yj,n(Bj))
≤ 2kγdd n(Bj)ln D
rT j
. (27)
Note that Yj1, . . . , Yj,n(Bj) are conditionally independent given Xj1, . . . , Xj,n(Bj). Hence, from McDiarmid’s in- equality, we have
P
ˆhT j− E[ˆhT j|Xj1, . . . , Xj,n(Bj)] ≥ t
≤ 2 exp
− 2t2
n(Bj)2kγ
dd n(Bj)lnrD
T j
2
= 2 exp
"
− t2n(Bj) 2k2d2γd2ln2lnrD
T j
#
. (28)
(28) bounds the deviation of the truncated estimator from its expectation with respect to its expectation. In order to bound I1, which is the deviation of the full estimator ˆhjwith respect to its expectation, we need to bound P(ˆhT j 6= ˆhj).
Define
Pj+(B(y, )) = Z
B(y,)
sup
x∈Bj
p(x, u)du, (29)
and n−lj (B(y, jl)) = Pn(Bj)
l0=1 1(Yj,l0 ∈ B(y, jl), l0 6=
l) as the number of samples within B(y, jl) among {Yj1, . . . , Yj,n(Bj)} except Yjl. Then
P(jl6= ρjl|Yjl= y)
(a)= P(jl< rT j|Yjl= y)
(b)
≤ P(n−lj (B(y, rT j)) ≥ k|Yjl= y)
(c)
≤ e−(n(Bj)−1)Pj+(B(y,rT j)) e(n(Bj) − 1)Pj+(B(y, rT j)) k
!k
≤ en(Bj)Pj+(B(y, rT j)) k
!k
(d)
≤ en(Bj)fUcdrdT j k
!k (30). Here (a) comes from the definition of ρjl in (25). (b) holds because jl is the k nearest neighbor distance of Yjl, which is smaller than rT j if there are at least k points in B(y, rT j) among {Yj1, . . . , Yj,n(Bj)} except Yjl. (c) uses Chernoff inequality. (d) comes from Assumption (b), which requires that the conditional pdf is no more than fU, hence Pj+(B(y, )) ≤ fUV (B(y, )) ≤ fUcdd, in which cdis the volume of a unit ball.
Hence,
P(ˆhT j6= ˆhj) = P
n(Bj)
X
l=1
ln ρjl 6=
n(Bj)
X
l=1
ln jl
= P
∪n(Bl=1j){ρjl6= jl}
≤ n(Bj) en(Bj)fUcdrdT j k
!k . (31) Moreover,
E[ˆhT j|X1, . . . , Xj,n(Bj)] − E[ˆhj|Xj1, . . . , Xj,n(Bj)]
= d
n(Bj)E
n(Bj)
X
l=1
lnρjl
jl
|Xj1, . . . , Xj,n(Bj)
. (32) For each l,
E
lnρjl
jl
= Z ∞
0
P
lnρjl
jl > u
du
(a)= Z ∞
0
P(jl< rT je−u)du
(b)
≤ Z ∞
0
en(Bj)fUcdrdT j k
!k
e−kdudu
= 1
kd
en(Bj)fUcdrdT j k
!k
, (33)
in which (a) holds because according to (25), if ρjl > jl, then ρjl = rT j, and (b) can be shown in the same way as (30). Therefore,
E[ˆhT j|Xj1, . . . , Xj,n(Bj)] − E[ˆhj|Xj1, . . . , Xj,n(Bj)]
≤ 1 k
en(Bj)fUcdrT jd k
!k
. (34)
From (28), (31) and (34), P
ˆhj− E[ˆhj|Xj1, . . . , Xj,n(Bj)]
≥ t +1 k
en(Bj)fUcdrdT j k
!k
≤ 2 exp
"
− t2n(Bj) 2k2d2γd2lnrD
T j
#
+n(Bj) en(Bj)fUcdrdT j k
!k
. (35)
Let
rT j= k
en(Bj)fUt4kN1k
!d1
, (36)
then P
ˆhj− E[ˆhj|Xj1, . . . , Xj,n(Bj)]
≥ t + 1 kN t4
≤ 2 exp
"
− t2n(Bj) 2k2d2γd2lnrD
T j
#
+n(Bj)
N t4 . (37)
Define Z = max
j n12(Bj)
ˆhj− E[ˆhj|Xj1, . . . , Xj,n(Bj)] . (38) Then
E[I12] ≤ E
m
X
j=1
n12(Bj)
N Z
2
≤ E[Z2]m
N, (39) in which the last step holds because
m
X
j=1
n12(Bj) N ≤ m12
N
m
X
j=1
n(Bj)
1 2
=r m
N. (40) Now it remains to bound E[Z2]. We show the following lemma.
Lemma 1. There exists a constant C1, such that E[Z2] ≤ C1ln2N .
Proof. Please see Appendix B-A for the proof.
Hence
E[I12] . m
Nln2N. (41)
Bound of E[I22]. To bound E[I22], according to the defi- nition of I2 in (21), it is important to bound I2j, which is defined as
I2j:= |E[ˆhj|Xj1, . . . , Xj,n(Bj)] − h(p(cj, ·))|. (42) Then
|I2| ≤
m
X
j=1
n(Bj)
N I2j. (43)
Define P (x, A) =R
Ap(x, u)du for any x ∈ S and A ⊂ S.
Then we decompose I2j as following:
I2j = |−ψ(k) + ψ(n(Bj)) + ln cd
+ d
n(Bj)E
n(Bj)
X
l=1
ln jl|Xj1, . . . , Xj,n(Bj)
−h(p(cj, ·))|
≤
1 n(Bj)
n(Bj)
X
l=1
E[ln P (cj, B(Yjl, jl))|Xn(Bj j)]
−ψ(k) + ψ(n(Bj))|
+
1 n(Bj)
n(Bj)
X
l=1
E[ln P (cj, B(Yjl, jl))|Xn(Bj j)]
−E[cddjl|Xn(Bj j)]
+ h(p(cj, ·))
, (44)
in which Xn(Bj j) represents (Xj1, . . . , Xj,n(Bj)). The last step holds because E[ln p(cj, Yjl)] = h(p(cj, ·)).
To bound the first term in (44), we define Pj−(B(y, )) =
Z
B(y,)
inf
x∈Bjp(x, u)du. (45) Then we show the following lemma.
Lemma 2. For all l ∈ {1, . . . , n(Bj)},
E[ln Pj+(B(Yjl, jl))|Yjl, Xn(Bj j)] ≥ ψ(k) − ψ(n(Bj)), (46) E[ln Pj−(B(Yjl, jl))|Yjl, Xn(Bj j)] ≤ ψ(k) − ψ(n(Bj)). (47) Proof. Please see Appendix B-B for detailed proof.
Moreover, for any > 0,
ln Pj+(B(y, )) − ln Pj−(B(y, ))
≤ Pj+(B(y, ) − Pj−(B(y, )) Pj−(B(y, ))
(a)
≤ 1
Pj−(B(y, )) Z
B(y,)
sup
x∈Bj
p(x, u) − inf
x∈Bjp(x, u)
! du
(b)
≤
√
dLacdd P (cj, B(y, )) −√
dLacdd
(c)
≤
√ dLa fL−√
dLa ≤ 2√ dLa
fL , (48)
for sufficiently small a. (a) comes from the definition of Pj+ and Pj− in (29) and (45), respectively. (b) uses Assumption (c), which requires that p(x, y) is L-Lipschitz. (c) uses Assumption (b), which requires p(x, y) ≥ fL.
Also consider that
ln Pj−(B(y, )) ≤ ln P (cj, B(y, )) ≤ ln Pj+(B(y, )). (49) Using (48), (49) and Lemma 2, we have
E[ln P (cj, B(Yjl, jl))|Yjl, Xn(Bj j)] − ψ(k) + ψ(n(Bj))
≤ 2La√ d fL
. (50)
Now we bound the second term in (44). Note that
1 n(Bj)
n(Bj)
X
l=1
E[ln P (cj, B(Yjl, jl))|Xn(Bj j)]
−E[cddjl|Xn(Bj j)]
+ h(p(cj, ·))
=
1 n(Bj)
n(Bj)
X
l=1
E
"
lnP (cj, B(Yjl, jl)) p(cj, Yjl)cddjl
#
, (51)
since (Xj1, Yj1), . . . , (Xj,n(Bj), Yj,n(Bj)) are randomly permuted from all points (Xi−1, Xi) for i ∈ Ij, and hence the expectations are the same for all l. Now we bound the right hand side of (51) with the following lemma.
Lemma 3. For all l,
E[ln P (cj, B(Yjl, jl))/(p(cj, Yjl)cddjl)]
≤ C2n−1d(Bj) for some constant C2.
Proof. Please see Appendix B-C for detailed proof.
From (44), (50), (51) and Lemma 3, we have I2j ≤ 2La√
d/fL+ C2n−d1(Bj). From (43), we have
|I2| ≤
m
X
j=1
n(Bj) N I2j
≤
m
X
j=1
n(Bj) N
2La√ d fL
+ C2n−1d(Bj)
!
= 2La√ d fL
+C2
N
m
X
j=1
n1−1d(Bj)
≤ 2La√ d fL
+ C2m1dN−d1. (52) The bin size a is proportional to m−1d. Therefore, we have
E[I22] . m−2d+m N
d2
. (53)
Bound of E[I32]. We begin with the following two lemmas whose proofs can be found in Appendix B.
Lemma 4. There exists a constant Cb, such that the n- step transition kernel satisfiessup
x∈S
kpn(x, ·) − πk∞≤ Cbθn, in which π is the pdf of stationary distribution, and θ = 1 − fLVS/2.
Lemma 5. a) For any set A ⊂ S, |E[n(A)] − N Pπ(A)| ≤ CbV (A)/(1 − θ), in which n(A) = PN
i=11(Xi ∈ A) is the number of samples inA, and Cb,θ are the constants in Lemma 4;
b) For any two disjoint setsA, B ⊂ S, A ∩ B = ∅,
|E[n(A)n(B)] − N2Pπ(A)Pπ(B)|
≤ 2CbV (A)Pπ(B) + 2CbV (B)Pπ(A) 1 − θ
+ Cb2
(1 − θ)2V (A)V (B) + Cb2
1 − θN V (A)V (B) + N Pπ(A)Pπ(B). (54) Define Ch = max{| ln fL|, | ln fU|}, such that
|h(p(cj, ·))| ≤ Ch always holds. Then based on Lemma 5, we have
E[I32]
= E
m
X
j=1
h(p(cj, ·)) n(Bj)
N − Pπ(Bj)
2
=
m
X
j=1 m
X
l=1
h(p(cj, ·))h(p(cl, ·))
E
n(Bj)n(Bl)
N2 − Pπ(Bj)n(Bl) N
−Pπ(Bl)n(Bj)
N + Pπ(Bj)Pπ(Bl)
≤ Ch2
m
X
j=1 m
X
l=1
E
n(Bj)n(Bl)
N2 − Pπ(Bj)Pπ(Bl)
−Pπ(Bj)E n(Bl)
N − Pπ(Bl)
−Pπ(Bl)E n(Bj)
N − Pπ(Bj)
= O 1 N
. (55) From (23), (41), (53) and (55), we have
E[(ˆ¯h − ¯h)2] . m
N ln2N + m−2d+m N
2d + 1
N. (56) We select the optimal m as following.
Case 1: d ≤ 2. Let m ∼ Nd+2d , and the convergence rate of the mean square error will be E[(ˆ¯h − ¯h)2] . N−d+22 ln2N .
Case 2: d > 2. Let m ∼ N12, and the convergence rate of the mean square error will be E[(ˆ¯h − ¯h)2] . N−1d.
APPENDIXB PROOF OFLEMMAS
A. Proof of Lemma 1
From (37) and the definition of Z in (38), P
Z > t + 1 kN12t4
≤
m
X
j=1
"
2 exp
"
− t2
2k2d2γd2lnrD
T j
#
+n(Bj) N t4
# .(57)
From (36), we have rT j ≥
k/(efUt4kN1+1k)1d . Then from (57), we know that there exist two constants c1 and c2such that
P
Z > t + 1 kN12t4
≤ 2m exp
− t2
c1ln N + c2ln t
+ 1
t4. For t > 1, t > 1/(kN12t4), thus
P(Z > 2t) ≤ 2m exp
− t2
c1ln N + c2ln t
+ 1
t4. (58) Hence for t > 2, we have
P(Z > t) ≤ 2m exp
− t2
4c1ln N + 4c2ln t
+16
t4. (59) Then
E[Z2]
= Z ∞
0
P(Z >√ u)du
≤ u0+ Z ∞
u0
2m exp
− u
4c1ln N + 2c2ln u
+16
u2
du
≤ u0+ Z ∞
u0
2me−8c1 ln Nu + 2me−4c2 ln uu +16 u2
du.
The above steps hold for arbitrary u0 > 0. We let u0 = max{16c1, 4} ln2N , then it is straightforward to show that the integration in the above equation converges to zero as N → ∞. Therefore E[Z2] = u0+ o(1) ≤ C1ln2N for some constant C1.
B. Proof of Lemma 2
Throughout the proof of this lemma, all the probabili- ties and expectations are conditional to Xn(Bj j). Therefore, Yj1, . . . , Yj,n(Bj)are conditionally independent. In the fol- lowing proof, we omit the notation Xn(Bj j) for simplicity.
Define a random variable U with cdf FU(u) = 1 −
k−1
X
s=0
n(Bj) − 1 s
us(1 − u)n(Bj)−s−1. (60)
Define r+(y, t) implicitly as Pj+(B(y, r+(y, t))) = t. Then P Pj+(B(Yjl, jl)) > t|Yjl= y
= P(l> r+(Yl, t)|Yl= y)
(a)= P n−lj (y, r+(t)) < k|Yjl= y
(b)
≥ P(RB< k). (61)
In (a),
n−lj (y, r+(y, t)) =
n(Bj)
X
l0=1
1(Yjl0 ∈ B(y, r+(y, t)), l0 6= l).
In (b), RB is a binomial random variable with parameter (n(Bj) − 1, Pj+(B(y, r+(y, t))), i.e. (n(Bj) − 1, t). Then
P(RB< k) =
k−1
X
s=0
n(Bj) − 1 s
ts(1 − t)n(Bj)−s−1
= P(U > t). (62)
Hence
P(Pj+(B(Yjl, jl)) > t|Yjl= y) ≥ P(U > t), (63) and
E[ln Pj+(B(Yjl, jl))|Yl= y] ≥ E[ln U ]. (64) Similarly, we have
E[ln Pj−(B(Yjl, jl))|Yl= y] ≤ E[ln U ]. (65) Now it remains to calculate E[ln U ]. From the cdf of U , we can derive the pdf as following:
fU(u) = 1
B(k, n(Bj) − k)uk−1(1 − u)n(Bj)−k−1, (66) in which B denotes the Beta function. Therefore, E[ln U ] = ψ(k) − ψ(n(Bj)). The proof is complete.
C. Proof of Lemma 3
The following steps hold for all l. For simplicity, we omit the index j and l. Moreover, all the probabilities and expec- tations are conditional on Xn(Bj j). We also omit the notations of conditional probability or conditional expectation.
Then we discuss three cases: (1)B(Y, ) ⊂ S, ≤ min {fL/(2L), R}; (2)B(Y, ) 6⊂ S, ≤ min {fL/(2L), R}; (3) > min {fL/(2L), R}. We give bounds for these cases separately.
Case 1. Since B(Y, ) ⊂ S and ≤ min{fL/(2L), R}, we have
lnP (cj, B(y, )) p(cj, y)cdd
≤
lnp(cj, y) − L
p(cj, y)
≤
lnfL− L
fL
≤ 2L
fL . (67) Hence,
lnP (cj, B(Y, )) p(cj, Y)cdd 1
B(Y, ) ⊂ S, ≤ min fL 2L, R
≤ 2L
fLE
1(B(Y, ) ⊂ S, ≤ min fL
2L, R
)
