Since the criteria developed in this chapter are based on the output of an MCMC sampling, they will be subject to simulation variability. Therefore, we measure the variability of these criteria by following the idea proposed byZhu and Carlin(2000). This is a "brute force" approach to check the stability of the DIC via computing the variance of DIC, var(DIC), which it is estimated by its sample variance
s2(DIC) = 1 L− 1 L
∑
l=1 (DICl− DIC)2,where L denotes the number of independent MCMC runs. Nevertheless, this approach is computationally expensive as it requires several runs. We also apply this approach for all criteria considered in this chapter.
5.8 Summary
In this chapter we have introduced well-known likelihood-based criteria, namely the AIC, BIC and DIC, for model selection in a HMM context, assuming an application requiring a fixed but unknown number of the states. We have used the data augmentation approach, where the parameter space is extended by adding hidden data for the unknown states. This provides several closed forms of the likelihood for the HMM, namely, the recursive (observed), complete data and conditional likelihood. Hence, it gave rise to define several versions of those criteria. More specifically, we have extended the original definitions of AIC and BIC, which use traditionally the classical approach, using the Bayesian principle, given two types of likelihood functions, namely the conditional and recursive likelihood. We introduced three cases for each criterion. In the first case, we introduced two versions that are called the AICrec1 and BICrec1, where the
term of model fit for both criteria is the expected recursive deviance evaluated at the posterior samples of the model parameters. These versions are inspired byBrooks(2002), who applied such versions to autoregressive models. The second and third cases of each criterion are new applications in the HMMs context. In the second case, they are called as the AICrec2 and BICrec2,
we proposed that the term of model fit of these two criteria is the recursive deviance evaluated at the posterior means of the model parameters. In the last case, they are called as the AICrec3 and
BICrec3, we proposed that such versions are based on a minimum recursive deviance observed
through an MCMC run. Given the conditional likelihood in the closed form, we also introduced three versions of the AIC and BIC. In the first case, we denoted these criteria as the AICcon1and
BICcon1, the term of model fit is the expected conditional deviance evaluated at the posterior
draws of the model parameters. These two versions are based on the same as the idea proposed
by Brooks (2002). The second and third cases of each criterion are new applications in the
HMMs context. In the second case, the criteria are referred to as the AICcon2 and BICcon2, we
proposed that the term of model fit of these two criteria is the expected conditional deviance, given a joint Maximum a posteriori (MAP) estimator of the state-dependent parameters θ and hidden states z. In the third case, they are denoted as the AICcon3 and BICcon3, we proposed that
such versions are based on a minimum conditional deviance observed through an MCMC run. In addition, we have introduced several versions of the original DIC. We have constructed these versions based on the type of likelihood and the concept of focus as proposed byCeleux et al.
(2006). Firstly, we introduced two versions of the DIC based on the recursive deviance, namely, the DICrec1 and DICrec2. The first version, DICrec1 based on the posterior recursive deviance
DICrec2, is a new modified version of the observed DIC3 introduced byCeleux et al.(2006).
In this latter version, we proposed that the focus is based on a minimum recursive deviance observed through an MCMC run, as a function estimator that differs from what is introduced
by Celeux et al. (2006) which was the expected density function. On the other hand, given
a conditional deviance obtained through an MCMC run, we also introduced two versions of the DIC based on the conditional deviance, namely, the DICcon1 and DICcon2. The first version,
DICcon1, is the same as the conditional version DIC7proposed byCeleux et al.(2006), where the
focus is the joint MAP estimator approximated using the best vector of state-specific parameters and hidden states of the model. The second version, DICcon2, is a new modified version of the
DIC7, where the focus is based on a functional estimator that is a minimum conditional deviance observed through an MCMC run.
Finally, we have considered the model selection issue from a predictive perspective. In this aspect, we contributed in applying a new criterion in the HMMs context, the so-called widely applicable information (WAIC) (Watanabe,2009) which considers, for our knowledge, a new application for the HMMs till writing this thesis.
Appendix