Here, a quantitative measure to evaluate the accuracy of the PF in repre-senting the posterior is introduced. The numerical test presented here can
be used in a generic PF framework.
It is known that for linear Gaussian models, the Kalman filter [20], de-scribed in section 2.4 of chapter 2, provides a optimal tracking estimate [32, 81, 90]. This thesis proposes to test the faithfulness of the PF in accor-dance to its KS statistic agreement [62] with the theoretically optimal Kalman filter. The KS test2 has the advantage of providing a reliable measure of the accuracy [279] of the estimate of the posterior. However, the test has been under-utilised in the PF literature and has never been used to measure re-sampler performance. The classical one-sample KS test evaluates the misfit between the cdfs of two uni-dimensional ordered datasets [273] by comput-ing the largest absolute difference. If the empirical distribution Xn for a set of I independent and identical distributed (i.i.d) uni-dimensional particles {xi}Ii=1 is defined as
Xn(y) = number of elements in the sample ≤ y
I = |{xi : xi ≤ y}|
I (5.16)
Then the KS statistic for a given cdf X is
κKS = supy |Xn(y) − X(y)| (5.17) This procedure however, cannot be applied to multi-dimensional distributions because there is no unique way to order the data. KS testing for multi-dimensional Gaussian models [274] can be conducted using the two-sample KS statistic [275, 276] by comparing the cdfs from two multi-dimensional datasets which are ordered in some fashion. One method is to compare the cdfs of the two samples with all possible combinations of sorting, and take the largest of the set of resulting KS statistics. Multi-dimensional KS testing has been used previously for other PF problems [277, 280]. A few other performance evaluation schemes that may also be adaptable to PF performance metrics can be found in [281].
In this thesis, for KS testing in linear Gaussian and multi-dimensional
2The KS test was first used in this dissertation in chapter 4.
models, a modified KS testing approach is proposed. The rationale for this proposal is to exploit the availability of the mean and covariance estimate provided by the theoretically optimal Kalman filter. The procedure will now be presented.
5.5.1 The modified KS testing approach
Mathematically, the proposed KS statistic testing is conducted as follows:
At time k, let the Kalman mean estimate be µk and the covariance estimate be Pk. Let Gk be the Cholesky decomposition of Pk. Consider that the D-dimensional resampled particle set {xik, wki}Ii=1 is available and that the particle states along the dth dimension are denoted by {xid,k}Ii=1. Firstly, each dimension of the particle state as represented by the particles are de-meaned and de-correlated using the theoretically optimal Kalman mean and covariance. This is done according to
yid,k= G−1k (xid,k− µk) (5.18) for i = 1, ..., I and d = 1, ..., D. The new particles {yd,ki }i=1,...,I, d=1,...,D are then sorted as
{Ψid,k, θd,ki }Ii=1= sort({yid,k}Ii=1) ; d = 1, ..., D (5.19) where {Ψid,k}Ii=1 contains the sorted particle states along the dth dimension and {θd,ki }Ii=1contains the indices of the particle states (in the dth dimension) in the sorted order. Then for i = 1, ..., I and d = 1, ..., D, the cdf of the particle states is evaluated according to
ˆ
The emperical cdf is the error function related to the integral of the standard normal distribution and is measured according to
˘
ck(d, i) = Φ(Ψid,k) = 1 + erf(Ψid,k/√ 2)
2 (5.21)
where the error function is described as erf(s) = 2
√π Z s
0
exp(−t2) dt (5.22)
Finally, the KS statistic is measured by conducting the one-sample KS test along each dimension separately and then taking the maximum KS deviation value amongst all the dimensions. That is,
κKS= supd supi |ˆck(d, i) − ˘ck(d, i)| (5.23) In words, the procedure is as follows: After resampling, the particle states are de-meaned and de-correlated using the theoretically optimal Kalman mean and covariance. Then the particle states along each dimension are sorted and the cdf (along that dimension) is constructed by evaluating the cumulative sum of the weights taken in the sorted order. Finally, the one-sample KS test is conducted along each dimension separately by comparing the cdf of the particle states and the error function, and maximum KS deviation value amongst all the dimensions is taken.
If the PF representation is identical to that of the Kalman filter distri-bution, then (5.18) will result in a set of uncorrelated particles which are normally distributed with zero mean and unit variance. The cdf of the states of these particles (after sorting) will be close to that of the error function, i.e., the KS deviation will be small. Conversely, if the PF representation is not identical to that of the Kalman filter distribution, then the cdf of the particle states will differ from that of the error function, i.e., the KS deviation will be large. Overall, the proposed method tests the representation misfit of the PF against a reference distribution — the theoretically optimal Kalman filter.
The procedure is now illustrated using a simple example.
Illustration of the proposed KS test:
Consider a two dimensional state space. Let the Kalman filter mean be µ = [6, 8]T and its covariance be
P =
The first scenario is shown in Fig. 5.5. Here, 1000 particles are drawn from a 2D normal distribution with mean µPF = µ and covariance PPF = P . These particles are plotted (as red dots) over the contour describing the Kalman filter distribution. The de-mean and de-correlation of these particles will result in a new set of particles (plotted as black dots). The contour of a standard normal distribution is also shown for reference. Since the PF representation is identical to that of the Kalman filter distribution, the deviation of the cdf of particles from the error function of N (0, 1) is small.
−5 0 5 10 Error function − Dim 1 PF CDF − Dim 2 Error function − Dim 2
(b) cdf
Figure 5.5: KS test illustration when the PF representation is close to the optimal Kalman filter distribution.
The second scenario is shown in Fig. 5.6. Here, 100 particles are drawn from a 2D normal distribution with mean µPF = µ and covariance PPF = P . Since the PF representation is identical to that of the Kalman filter distribution, the deviation of the cdf of particles from the error function of N (0, 1) is small.
However, the figure shows that the accuracy of approximation depends on the number of particles I, and insufficient I leads to poor representation of the distribution N (0, 1). Hence the deviation of the cdf of particle states
from the error function of N (0, 1) is larger when compared to the case of Error function − Dim 1 PF CDF − Dim 2 Error function − Dim 2
(b) cdf
Figure 5.6: KS test illustration when the PF representation is close to the optimal Kalman filter distribution, but the number of particles I is low.
The third scenario is shown in Fig. 5.7. Here, 1000 particles are drawn from a 2D normal distribution with mean and covariance different from µ and P respectively. Since the PF representation is not identical to that of the Kalman filter distribution, the deviation of the cdf of particles from the error function of N (0, 1) is large.