1.3.1 Theoretical Work
In Chapter II, we present the analysis of leave-one-out KDE plug-in estimators of general divergence functionals. We derive expressions for the bias and the variance of these plug-in estimators without boundary correction when the support set of the densities is bounded. We generalize the theory of optimally weighted ensemble estimation derived in [187] to obtain two KDE divergence functional estimators that achieve the parametric MSE convergence rate when the densities have smoothness parameter s ≥ d and s > d/2 under different conditions on the functional. The estimators are computationally tractable as the weights are calculated via an offline convex optimization problem. We then derive the asymptotic distribution of the weighted ensemble estimators which enables us to construct confidence intervals and perform hypothesis testing.
A similar analysis of leave-one-out k-nn plug-in estimators of general divergence functionals is given in Chapter III. Expressions for the bias and variance of these k-nn plug-in estimators without boundary correction are derived. The generalized theory of optimally weighted ensemble estimation presented in Chapter II is applied to obtain two k-nn divergence functional estimators that achieve the parametric MSE convergence rate when the densities have smoothness parameter s ≥ d and s > d/2 under different conditions on the functional. The asymptotic distribution of these weighted ensemble estimators are also derived. These k-nn estimators are typically more computationally tractable than the KDE estimators in Chapter II as there exist
many methods for computing the k-nearest neighbors that are computationally easier than calculating the KDE.
The analysis techniques used in Chapters II and III extend easily to the problem of estimating functionals of one (i.e. entropy functionals) or more distributions. Thus ensemble estimators for both KDE and k-nn plug-in estimators of entropy functionals (and functionals of 3 or more distributions) can be derived. However, extending these techniques to mutual information functionals requires a bit more care due to the possible dependencies between different samples. Under a similar setting, we extend the theory derived in Chapters II and III to provide nonparametric estimators general mutual information functionals under two cases: 1) the data have purely continuous components; 2) the data have a mixture of continuous and discrete components. To the best of our knowledge, our work is the first to derive MSE convergence rates for the latter case. The theory of optimally weighted ensemble estimation is applied to obtain estimators that achieve the parametric rate and the asymptotic distribution of these estimators is derived. This work is contained in Chapter IV.
1.3.2 Applications of Theory
The remaining chapters of this thesis are devoted to applications of our theoretical work described in Chapters II through IV. In Chapter V, we estimate the intrinsic dimension of sunspot image data using entropy-based estimators. We use the intrinsic dimension estimates to determine the size of a reduced dimension representation and to determine whether linear methods of dimensionality reduction are appropriate. The results of Chapter V are used in Chapter VI to reduce the dimension of the data and then cluster the sunspot images using divergence estimates as input to the clustering algorithm. Bounds on the Bayes error of a sunspot image classification problem are also estimated using our divergence functional estimators derived in the previous chapters.
We apply similar methods to high frequency oscillations (HFOs) measured from the brain in epilepsy patients. Typical analyses of HFOs have assumed that the data lie on a linear manifold that is global across time, channels, and patients. We estimated the intrinsic dimension of the data using entropy-based estimators to ex- amine these assumptions and to aid in dimensionality reduction. We further estimate bounds on the Bayes error to quantify the distinction between two classes of HFOs (those occurring during seizures and those occurring due to other processes). This analysis provides the foundation for future clinical use of HFO features and guides the analysis for other discrete events such as individual action potentials or multi-unit activity.
1.3.3 Publications Thereotical Work
1. K. Moon and A. Hero, “Ensemble estimation of multivariate f -divergence,” in IEEE Internatoinal Symposium on Information Theory (ISIT), 2014 [140]. 2. K. Moon and A. Hero, “Multivariate f -divergence estimation witn confidence,”
in Advances in Neural Information Processing Systems (NIPS), 2014 [141]. 3. K. Moon, K. Sricharan, K. Greenewald, and A. Hero, “Improving convergence of
divergence functional ensemble estimators,” in IEEE International Symposium on Information Theory (ISIT), 2016 [146].
4. K. Moon, K. Sricharan, K. Greenewald, and A. Hero, “Nonparametric ensemble estimation of distributional functionals,” submitted to IEEE Transactions on Information Theory, March 2016 [145].
5. K. Moon, K. Sricharan, and A. Hero, “Ensemble Estimation of Mutual In- formation,” submitted to Advances in Neural Information Processing Systems
(NIPS), 2016.
6. K. Moon, K. Sricharan, and A. Hero, “Nearest neighbor ensemble estimation of distributional functionals,” in preparation for submission to IEEE Transactions on Information Theory.
7. K. Moon, M. Noushad, S. Sekeh, and A. Hero, “Nonparametric mutual infor- mation measures,” in preparation for submission to ICASSP.
Application to Sunspot Data
1. K. Moon, J. Li, V. Delouille, F. Watson, and A. Hero, “Image patch analysis and clustering of sunspots: A dimensionality reduction approach,” in IEEE International Conference on Image Processing (ICIP), 2014 [142].
2. K. Moon, V. Delouille, A. Hero, “Meta learning of bounds on the Bayes classifier error,” in IEEE Signal Processing and SP Education Workshop, 2015 [139]. 3. K. Moon, J. Li, V. Delouille, R. De Visscher, F. Watson, and A. Hero, “Im-
age patch analysis of sunspots and active regions. I. Intrinsic dimension and correlation analysis,” Journal of Space Weather and Space Climate, 2016 [144]. 4. K. Moon, V. Delouille, J. Li, R. De Visscher, F. Watson, and A. Hero, “Im- age patch analysis of sunspots and active regions. II. Clustering via matrix factorization,” Journal of Space Weather and Space Climate, 2016 [143].
Application to HFO Data
1. S. Gliske, K. Moon, W. Stacey, and A. Hero, “The intrinsic value of HFO features as a biomarker of epileptic activity,” in IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2016 [75].