In this chapter two yet partly unsolved problems concerning NMF decompositions X ≈ WH were addressed employing Bayesian methods.
A Bayesian criterion of a best solution, given the correct number of components K, was obtained by integrating over all possible weight matrices W to estimate the posterior distribution of the basis matrix H. In order to render the related high dimensional integral tractable, a Laplace approximation was utilized. In case of a Gaussian likelihood this method yields a simple Bayesian optimality criterion for NMF named BOC. It was shown that in the Gaussian likelihood case the BOC describes the same optimal solution as described by the determinant criterion in the noise-free case which was presented in chapter 3.2.4.
Further, a variational Bayes NMF algorithm for a Gaussian likelihood and rectified Gaussian priors named VBNMF was introduced which can be seen as a Bayesian generalization of the multiplicative Lee-Seung algorithm for Gaussian NMF. Both algorithms (the maximum likelihood procedure and its Bayesian extension) are based on Jensen’s inequality to decouple parameters or expectations thereof by introducing a bound containing auxiliary variables.
Although variational Bayes procedures have already been applied previously to the NMF problem, a complete derivation for the Gaussian likelihood case with simulations has not been published so far. While [Cem09] concentrates on the KL-divergence cost function for NMF, [FC09] mentions the feasibility of the method in the Gaussian likelihood case only. Furthermore they mention that their maximum likelihood version for Gaussian NMF does not coincide with the original multiplicative NMF algorithm, while the method presented here does so.
The non-negative factor analysis based on Bayesian learning presented in [HK07] has also some par- allels with the VBNMF method. The paper mentions that the computations with rectified Gaussian priors are only possible if their location parameter is set to zero and present some special constuction of a Gaussian prior and a rectification. Maybe the Jensen trick and the additional parameters qikj
circumvent this problem in case of the VBNMF algorithm, since the location parameters µHk0 and
µW0k are free and the model is still tractable.
In this respect, the VBNMF algorithm as a direct generalization of the Lee-Seung multiplicative NMF algorithm for Euclidean NMF seems to be a new contribution to the best of my knowledge.
In simulation studies, the ability of the VBNMF algorithm to automatically detect the optimal number of components in constructed toydata was demonstrated. If all prior assumptions are valid, the correct number K is perfectly determined by the algorithm. Moreover, if the prior assumptions are slightly violated or even if the likelihood function differs from the model assumption, the correct factorization rank can be determined by either evaluating the lower bound to the log evidence Bq or by automatic
relevance determination.
Decompositions of binary real world data show that the VBNMF algorithm in principle does dis- tinguish between important and irrelevant components. Potential improvement of the method could be achieved by allowing more flexible shapes for the prior distributions which better suit the data. However, this improvement requires more sophisticated optimization strategies for the determination of optimal hyperparameter estimates.
Chapter 9
Summary and outlook
This thesis investigated extensions to the Non-negative matrix factorization (NMF) machine learning technique and their potential usefulness for failure analysis in semiconductor microchip manufacturing. As reviewed in the introduction (chapter 1), there is a huge arsenal of data analysis techniques used to gather information from the immense amount of data arising during and after wafer fabrications. The main idea followed here is to apply Blind Source Separation techniques, which interpret the wafer fabrication as a black box in which a set of underlying root causes somehow generates malfunctions on the chips.
NMF is suitable to discover latent structures in high dimensional data which are generated as strictly additive linear combinations of non-negative variables.
Chapter 2, gave an introduction to the NMF technique and motivated NMF problems by an air pol- lution example. After an overview of application fields of NMF so far, the basic principles concerning non-negatively constrained optimization are discussed. Further, two main problems are highlighted: The uniqueness problem of NMF and the lack of adequate methods to determine an optimal number of components in previously unknown data.
Before NMF was tested in a practical environment, the uniqueness issue was discussed in detail in chapter 3. By geometrical considerations the ambiguity of NMF to produce unique solutions was explained and a determinant criterion was introduced to fix this uniqueness problem. An algorithm named detNMF was designed which incorporates the determinant criterion directly and simulations demonstrate its ability to recover the correct solutions in constructed toydata examples. A further illustrative example was provided which distinguishes the new method from an established approach based on sparsity assumptions. Further, an alternative explanation of the apparent superiority of so called multilayer approaches for NMF could be derived by means of the determinant criterion. In chapter 4, NMF was directly applied to a real world data set which was generated from measurement data containing different test categories called BINs. It was demonstrated that NMF clearly recovers the structure of the data. However, the determination of an optimal number of underlying components turned out to be tricky. While in this chapter, the existing NMF methodology was applied to suitable aggregated data which approximately satisfies the linear NMF model, chapter 5 presents a completely new developed extension of NMF for binary data.
The new method called binNMF (binary Non-negative matrix factorization) is based on a superposi- tion approximation for fail probabilities of single chips. It turned out that these pass/fail probabilities can be modeled by means of a nonlinear function of two non-negative matrices. These parallels to usual NMF were exploited to develop an effective two stage optimization procedure which was demon- strated to discover localized features in binary test data sets.
The rest of the thesis is dedicated to the use of Bayesian techniques in combination with NMF and their ability to answer the two questions concerning the uniqueness and the optimal number of sources.
After two literature chapters on Bayesian methods in general (6), and their combination with NMF (7) further Bayesian extensions to NMF were presented in chapter 8. The idea behind these extensions was to exploit the statistical well-founded Bayesian methodology in order to find answers to the two main problems of usual NMF: uniqueness and the optimal number of components.
In the first part, a Bayesian optimality criterion was derived which describes the optimal solution of a NMF problem in absence of prior knowledge, formulated as MAP (maximum a posteriori) estimation. Unlike usual Bayesian approaches which try to explicitly model certain characteristics such as sparse- ness or smoothness of the components directly, here we used the Bayesian formalism to explicitly model the absence of prior information. An optimal set of basis vectors can be determined then by integrating over all possible weight matrices, given the correct factorization rank.
In the special case of a Gaussian likelihood, the Bayesian criterion can be shown to describe the same optimal solution which was proposed in chapter 3 by the determinant criterion. Unlike the usual direct implementation of certain characteristics in usual Bayes NMF approaches, here and in the determinant criterion an indirect relation between the basis components follows automatically from geometrical considerations and the Bayesian formalism.
In the second part of chapter 8, a variational Bayes algorithm named VBNMF was developed. It is based on the assumptions of a Gaussian likelihood and independent rectified Gaussian prior distri- butions on the parameter matrices. It was shown that the VBNMF algorithm is a direct Bayesian extension to a popular multiplicative NMF algorithm.
Simulations demonstrate that the VBNMF is able to discover the actual number of components in toydata sets. Remarkably, the correct number of components could also predicted well in cases where not all a priori assumptions hold true.
Application on real binary data demonstrates the general power of the method to allow conclusions on the actual number of components under certain assumptions. At the same time, the real data application shows the direction of further improvements by modeling more flexible shapes for the prior distributions.
9.1
Main contributions
To the best of my knowledge, NMF techniques have not been applied to the kind of wafer test data investigated here so far.
Separate from the exploration of this new application field for NMF and extensions, a variety of new aspects concerning the general methodology were developed here for the first time.
These findings can best be summarized by the expression extensions to NMF. First, the charac- terization of optimal solutions by a determinant criterion (chapter 3), its distinction from sparsity approaches and the simple explanation of the superiority of multilayer NMF algorithms over usual ones by means of the determinant criterion are original contributions which have been presented at the ICA conference 2009 [SPTL09].
The complete binNMF methodology including its optimization strategy as discussed in chapter 5 is a new extension of NMF to binary data sets. Although there are some approaches to the analysis of binary datasets in the literature, none of the existing techniques covers all aspects required for the special kind of binary pass/fail data considered here (see the discussion in paragraph 5.6). Parts of this chapter have been published GfKl conference 2008 [SPL10], and the at the ICA conference 2009 [SPL09].
Finally, the Bayesian extensions discussed in chapter 8 contain partly novel contributions.
As the discussion in paragraph 8.3 shows, the VBNMF algorithm as discussed here seems to be new in this form. However, there are several possibilities to develop a variational Bayes extension for NMF as has been mentioned elsewhere.
9.2. OUTLOOK 121