Dirichlet Priors for Bayesian Estimation in Simplex Spaces

Sometimes, as it is the case in this thesis (see chapter 6), it will be useful to estimate uncertain quantities in structured spaces other then Rn. For instance, the following expression describes a vector w belonging to a simplex space:

w = (w1 w2 · · · wd)| ∈ Sd−1= ( w ∈ Rd: d X k=1 wk= 1, ∀ j wj ≥ 0 ) . (2.34)

Note that we are interested on representing w as a random vector of proportions lying in the (d−1)-simplex space, Sd−1. It is worth noting that simplex spaces possess elementary properties that may simplify the resulting analytical expressions when performing Bayesian inference over proportions. The first property regards a basic notion of complementarity. It describes the fact that each individual variable (proportion) wi can be completely described as the complement of

the sum over the remaining variables to the unity, i.e., wi = 1 −

 P

k6=iwk



. This is the reason why simplex spaces are denoted as Sd−1 and why we only need to account for d − 1 variables when depicting Sd−1 (see Fig. 2.4). In other words, each and every variable in a simplex space complements one another; i.e., any subset of d − 1 variables from the simplex space yields complete information about the value of the remaining variable.

The second intuitive property is given as a geometrical interpretation of the maximum entropy discrete distribution represented as a point in Sd−1_{, when viewing the simplex as a}

natural space for representing all possible discrete probability distributions over d orthogonal states. Take d = 3 as an example, for which S2 _{can be depicted in an extended form as an}

(a) D(1, 1, 1) (b) D(2, 2, 2) (c) D(3, 3, 3)

Figure 2.4: Contour plots for the Dirichlet distribution in the 2-simplex.

of S2_{all perfectly coincide at the point w}

C = (1/3 1/3 1/3)|, which results from the intersection

of the three lines of symmetry of S2 _[81]8_{. It turns out that w}

C thus also represents the discrete

distribution of maximal Shannon’s entropy (see section 2.2.1).

The Dirichlet Distribution

One of the most important questions when designing Bayesian inference methods for simplex spaces is: what distribution should we use to represent our uncertainty regarding w ∈ Sd−1_?

We could in principle try fitting an unbounded distribution such as the multivariate Gaussian to represent the data collected in Sd−1_{. But such parametric model would lead to large estimation}

errors nearby the boundaries of Sd−1, since the Gaussian would always attribute positive density to unfeasible samples, i.e., points that lie outside Sd−1.

A more plausible choice could be to fit distributions in the surface of a unit hypersphere, such as the von Mises-Fisher distribution [18]. But while such distribution conveys some structure, i.e., the elements in Sd−1 are also unit vectors and can be thus represented in the unit d − 1 hypersphere surface, there are still elements outside of Sd−1 _{that could be attributed positive}

probability density.

One sound choice that is just right for modeling uncertainty in the Sd−1 _{simplex (i.e., that}

indeed takes Sd−1 _{support) is the Dirichlet Distribution (DD) [160], which is a multivariate}

version of the Beta distribution. Because any multinomial probability distribution can be rep- resented as a point in Sd−1_{, the DD is also known as a distribution over distributions. Thus, the}

DD can also be used to represent the underlying uncertainty over true probability distributions, what conveys a notion of second-order uncertainty.

This is one of the main reasons why the DD is becoming popular in several Bayesian machine learning applications, e.g. text mining and topic models [19]. Another reason is that the DD is part of the exponential family [160], which means that the conjugate prior used for estimating the posterior distribution using Bayes Theorem is also from the exponential family. This very

8_{This property only holds for equilateral triangles. For other triangles, Euler [81] proved that those notable}

2.3. Learning and Prediction Under Uncertainty 51

much simplifies the calculations, otherwise, one would need to rely on numerical methods to estimate the posterior.

In order to motivate its usage, consider a simple coin flipping experiment. The goal is to represent our uncertainty about the probability that a coin flip will yield heads, based on previously observed flippings. We may be very well dealing with a fair coin and therefore we may want to assign P (heads) = P (tails) = 1₂. On the other hand, we have no idea from where this coin came from or how it was fabricated. So we do not want to rule out the possibility that the coin is biased. Here is when the DD usage becomes very handy: we can assign positive probability density to all the possible heads odds. Not only that, but also, by using the DD9_,

we can model the decrement of our uncertainty about the heads odds as we observe more and more flippings, making the probability distribution P to progressively peak at the true odds value.

Let w denote a vector of proportions so that wk represents the proportion of the k-th item.

For example, in financial portfolio applications, wk can be the proportion of wealth invested in

the k-th asset. The DD can be written as: w ∼ D(α) = Γ ( P kαk) Q kΓ(αk) Y k wαk_k −1, (2.35) where Γ(x) = R∞ 0 t

x−1_e−t_{dt is the Gamma function}10 _{and the parameter α = (α}

1 · · · αN)|

determines the shape of the distribution (with αk > 0 for all k). Note that if the value of

all but one of the N components of w are given, then the value of the remaining one is also known because P

kwk = 1. Thus, the Dirichlet density will lie in a d − 1 hyperplane (the

(d − 1)-simplex) in d-dimensional space.

The DD can be re-parametrized by the mean vector m =_αCα1 · · ·αN αC

|

, where αC =

P

kαkis

known as the concentration parameter, which controls the dispersion of the distribution around m: the higher αC is, the more concentrated the density will be around the mean vector in the

(d − 1)-simplex (see Fig. 2.4).

The concentration parameter has an interesting interpretation in terms of learning (and therefore uncertainty reduction): if each αk is interpreted as pseudo-counts11, then the more we

accumulate evidence, the more certain we will be about the location of w in the (d − 1)-simplex. Remark: When α = 1, the DD describes a uniform distribution in the (d−1)-simplex, which is the maximum entropy distribution for random vectors whose moments are not constrained [163], i.e., whose only constraint is that it is contained in the distribution support.

The moments of the DD are:

E[wk] =

αk

αC

, (2.36)

9_{In the case of a simple coin experiment, the DD actually reduces to a Beta distribution.}

10_{Γ(x) is an extension of the factorial function in the real line, defined as Γ(x) = (x − 1)Γ(x − 1) and Γ(1) = 1.}

For a positive integer n, Γ(n) = (n − 1)!.

11_{Pseudo-counts are used to adjust models based on observed frequencies of occurrences. Rare (but not}

impossible) events that were not yet observed are usually inaccurately attributed zero probability, but this can be artificially corrected by adding small quantities to such frequencies so the model can account for such events.

Var(wk) =

αk(αC− αk)

α2

C(αC + 1)

. (2.37)

The covariances are always non-positive for any two proportions in the DD, since whenever wk

is incremented, some or all other proportions must be necessarily decremented so as to satisfy P kwk = 1. Thus, Cov(wi, wj) = −αiαj α2 C(αC + 1) . (2.38)

It also is worth noting that the marginal distribution of wk is a Beta distribution, i.e.,

wk ∼ Beta(αk, αC − αk). (2.39)

Remark: James and Moisimann [124] showed that the Dirichlet distribution is characterized by neutrality, which is a strong notion of statistical independence on spaces whose coordinates must add up to a certain sum. That is to say, if w ∼ D(α), then the vector w is completely neutral because wk ∼ Beta(αk, αC − αk) is provenly independent of

w(−k) = ( w1 1 − wk · · · wk−1 1 − wk wk+1 1 − wk · · · wd 1 − wk )|∼ D(α(−k)_{), (2.40)}

for all 1 ≤ k ≤ d, what implies that any permutation of w is neutral [124]. We can thus see that the neutrality property of the DD is thus intuitively linked with the complementarity property of Sd−1_{(see section 2.3.5): although every element in w perfectly complement the sum}

of the others, removing an element wk from w while equally redistributing its mass across the

remaining d − 1 elements yields a new Dirichlet distribution which is completely independent of wk.

The Dirichlet Distribution as the Conjugate Prior of a Multinomial

It is also well-known that the DD and the multinomial distribution are the conjugate priors of each other [160]. Let x be a vector of N observed frequencies of occurrence (counts) following a multinomial distribution with parameter p (here, p is a proportion vector in the (d − 1)-simplex specifying the probabilities of observing an item k). Its probability mass function is expressed as: x ∼ M ult(x|p) = ( P kxk)! Q kxk! Y k θ_kxk. (2.41)

If p follows a DD (i.e., p ∼ D(p|α)), then the posterior distribution of p given that we observed the counts x also follows a Dirichlet:

p(p|x) = p(x|p)p(p) = M ult(x|p)D(p|α) ∝Y k θxk_k Y k θαk_k −1 ∝Y k θαk−1+xk_k = D(α + x).