Probabilistic Modelling with Deep Learning

Deep neural networks literally refer to the multi-layer neural network models connecting multiple processing layers sequentially, to learn complex and abstract features from given datasets. Traditionally, classical deep neural networks are a deterministic model rather than probabilistic models, since it is normally designed to predict the best estimation of uncertain outputs, which generates a deterministic number/vector as the output. However, this thesis seeks for a probabilistic model of deep neural networks, in other words, the proposed deep neural networks should be able to learn and generate probability distribution. This section discusses the probabilistic deep learning model and its theoretical basis: Bayesian Modelling and Variational Inference (VI). Bayesian Modelling is highly-rated since it can cope with most complex problems in the real world due to its model capacity. However, the implementation of Bayesian Modelling heavily suffers from its expensive computational costs caused by extensive sigma functions and cross-corpora calculations. In addition, they perform poorly with very small data sets. Therefore, in this section, Deep Learning models are employed as the model basis for Bayesian Modelling. Dropout Regularization units are specifically applied to introduce random features to the enable the deterministic deep learning model for probabilistic Bayesian Modelling. The efficacy and effectiveness are discussed and demonstrated in following sections.

4.4.1.1 Probabilistic Modelling based on Bayesian Modelling

Classical deep learning models map model inputs to the corresponding outputs in the form of deterministic numerical vectors. Given model input variables and output variables as variables 𝑥 and 𝑦, classical deep neural networks can be simply regarded as a synthesized function 𝑓(∗). Specifically, this function 𝑓 is the combination of matrices multiplication and activation functions of each processing layer. The model parameters are denoted as 𝜃 = {𝑊, 𝑏, … }, which consists of weight matrices, bias matrices and other parameters of certain network architectures. Therefore, the model outputs can be obtained given input variable 𝑥 on condition of parameter vectors 𝜃:

𝑦 = 𝑓(𝑥; 𝜃), where both 𝑥, 𝑦 are deterministic variables, which indicates classical deep learning models are deterministic models.

In terms of the probabilistic model, both model inputs and outputs are further considered as random variables. To express notations in consistency, a tilde accent is added to notations to replace the corresponding deterministic variables with random variables, i.e., 𝑦̃ refers to the uncertainty of model output. In this case, the deterministic model is predicting an estimate on the expectation of probability distribution of uncertain output 𝑦̃, i.e.:

𝔼_𝜃[𝑦̃] = 𝔼_𝜃[𝑓(𝑥; 𝜃)] = 𝔼_𝜃[𝑝(𝑦̃|𝑥, 𝜃)] ( 4-1 )

Assuming the probabilistic model of the corresponding deep neural network is 𝑓^′, the probabilistic outputs of the deep neural network can be then formulated as:

𝑦̃ = 𝑓^′(𝑥; 𝜃̃)~𝑝(𝑦̃|𝑥; 𝜃̃) ( 4-2 )

where the conditional distribution 𝑝(𝑦̃|𝑥; 𝜃̃) is the probabilistic model of the deep neural network. The comparison between the classical deterministic model and probabilistic model of deep neural networks are shown in Figure 4-1 below:

Figure 4-1 Illustration of two types of deep neural networks: a) deterministic model; b) probabilistic model.

Given historical datasets 𝑋, 𝑌 as training data, we can obtain model parameters that are most likely to generate label of dataset 𝑌 from probabilistic deep neural network given inputs 𝑋. Mathematically, we look for the posterior distribution of model parameter 𝑝(𝜃̃|𝑋, 𝑌) on given dataset 𝑋, 𝑌. By invoking Bayes’ theorem, the posterior distribution of interest can be formulated by:

𝑦̃ = 𝑓^′(𝑥; 𝜃̃)~𝑝(𝑦̃|𝑥; 𝜃̃) ( 4-3 )

𝑝(𝜃̃|𝑋, 𝑌) =𝑝(𝑌|𝑋; 𝜃̃)𝑝(𝜃̃)

𝑝(𝑌|𝑋) ( 4-4 )

With appropriate assumption on the prior distribution of parameters 𝑝(𝜃̃), the posterior distribution of model parameters can be more easily estimated with integrals over training datasets. This process is widely known as Bayesian Modelling [89, 90, 91].

For the testing realizations of inputs 𝑥^∗, the prediction on the targeted distribution of 𝑦^∗ can be accordingly inferred as:

𝑝(𝑦^∗|𝑥^∗; 𝑋, 𝑌) = ∫ 𝑝(𝑦^∗|𝑥^∗; 𝜃̃)𝑝(𝜃̃|𝑋, 𝑌)𝑑𝜃̃ ( 4-5 )

Notably, 𝑝(𝑦^∗|𝑥^∗; 𝜃̃) is the output distribution of the probabilistic model of the deep neural network, which can be simply sampled and simulated by feed in realizations of 𝑥^∗ to the well-tuned deep neural network. Therefore, the targeted posterior distribution 𝑝(𝑦^∗|𝑥^∗; 𝑋, 𝑌) can be easily implemented by sampling approaches such as Gibbs Sampling [92].

4.4.1.1 Variational Inference to Train Probabilistic Models

As discussed in the previous section, to invoke feed-forward prediction of the probabilistic model, the key component is to obtain the posterior distribution 𝑝(𝜃̃|𝑋, 𝑌) with given data samples. Normally, this distribution cannot be modelled directly by deep networks due to the fixed structure of deep networks. To approximate the targeted distribution with the probability model of deep network 𝑞(𝜃̃), variational inference is introduced in the training process to minimize the difference between 𝑝(𝜃̃|𝑋, 𝑌) and 𝑞(𝜃̃).

A measure that indicates the similarity between the targeting distribution 𝑝(𝜃̃|𝑋, 𝑌) and the distribution of the deep neural network 𝑞(𝜃̃) can be evaluated by the KL divergence:

𝐾𝐿(𝑞(𝜃̃)||𝑝(𝜃̃|𝑋, 𝑌)) = ∫ 𝑞(𝜃̃) 𝑞(𝜃̃)

𝑝(𝜃̃|𝑋, 𝑌)𝑑𝜃̃ ( 4-6 )

The KL divergence approaches its minimum when the distribution of deep learning model 𝑞(𝜃̃) is close to the targeting posterior distribution 𝑝(𝜃̃|𝑋, 𝑌), which can be denoted as 𝑞^∗(𝜃̃). Therefore, we can replace the posterior distribution with neural network model in equation (4-13) to (4-19):

𝑝(𝑦^∗|𝑥^∗, 𝑋, 𝑌) = ∫ 𝑝(𝑦^∗|𝑥^∗, 𝜃̃)𝑞^∗(𝜃̃) 𝑑𝜃̃ ( 4-7 )

This process means the probability model of output 𝑦^∗ with given inputs 𝑥^∗ can be quantified by sampling through the deep learning model with variational parameter 𝜃̃

that follows the distribution 𝑞^∗(𝜃̃).

Notably, the KL divergence is intractable in many cases due to the posterior distribution in the integral form. Hence, minimizing the KL divergence is replaced by an equivalent formulation, i.e., evidence lower bound (ELBO) [93].

ℒ(𝜃̃) ∶= ∫ 𝑞(𝜃̃)𝑙𝑜𝑔 𝑝(𝑌|𝑋, 𝜃̃)𝑑𝜃̃ − 𝐾𝐿(𝑞(𝜃̃)||𝑝(𝜃̃)) ( 4-8 )

Through optimization that maximizes the ELBO term, the KL divergence between 𝑞^∗(𝜃̃) and 𝑝(𝜃̃) can be accordingly minimized. Hence, the feed-forward of proposed model can be further simplified into compact form as:

𝑝(𝑦^∗|𝑥^∗, 𝑋, 𝑌) = ∫ 𝑝(𝑦^∗|𝑥^∗, 𝜃̃)𝑞^∗(𝜃̃) 𝑑𝜃̃ ≈ 𝑞^∗(𝑦^∗|𝑥^∗, 𝜃̃) ( 4-9 )