with MapReduce (Dean and Ghemawat,2008), a well-known disk-based distributed framework, Spark provides an in-memory processing solution which bypasses the heavy disk I/O bottleneck of reloading the data when performing iterative machine learning methods. In addition, the platform also supports high-level APIs which are more friendly to developers and especially supplies REPL (read-eval-print-loop) environment for data scientists.
Our second goal in this work is to investigate a truly distributed collapsed variational Bayes for HDP model called SkHDP-CVB. We will use the distributed graph struc- ture called GraphX – an embedded graph processing framework built on top of Apache Spark to represent the sparse relationship between documents and words. This data structure is naturally fitted with the bag-word representation in topic modelling and built with many efficient operations on distributed systems. As a consequence, it usually takes days (to weeks) to learn the large corpora with a single machine. We can learn the same data with the distributed learning algorithms for hours.
The rest of the chapter is organised as follows: In Section 5.2, we review the HDP and present a truncated view of the model. Section5.3 describes our truly collapsed variational Bayes algorithm for HDP. Distributed implementation of the proposed algorithm with GraphX on Apache Spark is provided in Section 5.4. A compre- hensive evaluation of the capability and scalability of the proposed algorithm on large-scale datasets with approximately billions of data points is provided in Section 5.5. Finally, Section 5.6 concludes the chapter.
5.2
Recent Advances in HDP Inference Algorithms
In this section we briefly review the truncation representation of Hierarchical Dirich- let processes, the stick-breaking based generative process for HDP, which is conve- nient to develop the collapsed variational Bayes inference algorithms, and variational inference for HDP. This section then lays the groundwork for our model developed in the sequel.
5.2.1
Truncation representation of Dirichlet process
Stick-breaking representation for Dirichlet process G0 ∼ DP (γ, H) which is de-
5.2. Recent Advances in HDP Inference Algorithms 99 follows ¯ G0 = K X k=1 βkδφk, β 0 k ∼Beta (1, γ), βk =β 0 k K−1 Y t=1 1−βt0, k < K, βK = 1. (5.1)
We call that Dirichlet process is approximated at truncated level K. The truncation error of truncated Dirichlet process priors is bounded according to the following theorem.
Theorem 5.1. (Ishwaran and James, 2002) The truncated stick-breaking represen- tation of Dirichlet process has the error bound as
ˆ Ω |DPK(X)−DP∞(X)|dX ≤4 1−E "K−1 X k=1 βk #n ≈4nexp − K −1 γ ! , where βkare the stick random lengths defined in Equation (5.1), andnis the number of observations from the process.
The theorem guarantees that the truncated Dirichlet process is exponentially accu- rate with respect to the truncation levelK. The bound for the prior also implies the bound for the truncated Dirichlet process posterior. Let consider the case when we have observednsamples from Dirichlet process, the indicator vectorZ ,(z1, . . . , zn) denoted the sticks that data come from. When using the truncated Dirichlet process, the support for each zi is {1,2, . . . , K} while it is {1,2, . . .} for Dirichlet process. We denote Zn∈ {1,2, . . . , K}n as the indicator for truncated Dirichlet process and
Z∞ ∈ {1,2, . . .}nas the indicator for Dirichlet process. It is obvious thatZn ∈Z∞.
The bound for truncated Dirichlet process posterior DPK(Zn|X) is stated as fol- lows
Lemma 5.2. (Ishwaran and James,2002) The error of stick-breaking representation of Dirichlet process posterior is bounded as
ˆ Ω X Z∈Z∞ |DPK(Z |X)−DP∞(Z |X)|DP∞(X)dX =O nexp − K −1 γ !! , where DPK(Z |X) and DP∞(Z |X) are the posteriors for Z under truncated
and original Dirichlet process respectively; DP∞(X) is marginal density under the
Dirichlet process.
As the result of the above theorem and lemma, we can truncate the Dirichlet process at a certain level with some desired error bound. In the following sections, we recall the HDP model without truncations and some variational inference methods that use truncated version of the Dirichlet process for variational distribution. However, we use the truncated version for the model and obtain analytically closed forms for the variational updates in Section 5.3.
5.2. Recent Advances in HDP Inference Algorithms 100
(a) Generative view (b) Stick breaking view (c) Stick breaking combining Chinese restaurant franchise view
Figure 5.1: Graphical representation for the HDP model.
5.2.2
Variational Inference for HDP
Let us consider the case when we have a corpus with J documents. With the assumption that each document is related to several topics, we can model each document as a mixture of latent topics using Dirichlet process mixture. Though different documents may be generated from different topics, they usually share some of the topics together. This is also known sharing statistical strength across groups which is a key property of the Hierarchical Dirichlet Process (HDP) (Teh et al.,
2006). In HDP, the topics among documents are coupled using another Dirichlet process mixtureG0. For each document, a Dirichlet processGj, j = 1, . . . , J, is used to model its topic distribution. Gj (s) are ’linked’ together by G0; and due to the
discreteness of G0, topics are naturally shared among Gj (s). Formally, generative representation with the graphical model shown in Figure 5.1a is as below:
G0 |γ, H ∼DP(γ, H) Gj |α, G0 ∼DP(α, G0) (5.2)
θji |Gj ∼Gj xji |θji ∼F(θji). The stick breaking representation of HDP is described as follows
β =β1:∞∼GEM(γ), G0 = ∞ X k=1 βkδφk, (5.3) πj =πj1:j∞∼DP(α, β), Gj = ∞ X k=1 πjkδφk, zij ∼πj, φk ∼H(λ), xji ∼F(φzji), (5.4)
where the GEM(γ) distribution is constructed as follows
β= (βk)∞k=1 s.t βk =vkY s<k
5.2. Recent Advances in HDP Inference Algorithms 101
However, in Equation (5.4), stick lengthπj for each documentGj can be construct using stick breaking process with the value of β as (Teh et al., 2006)
ujk ∼Beta αβk, α 1− k X l=1 βl !! , πjk =ujk k−1 Y l=1 (1−ujl), k= 1,2, . . . ,∞.
Wang et al. (Wang et al., 2011) used stick-breaking construction combining with Chinese restaurant franchise to develop variational inference method for HDP. The developed representation using two stick-breaking constructions for corpus and doc- ument level is defined as follows
ψjt ∼G0, π 0 jt ∼Beta (1, α), πjt =π 0 jt t Y l=1 1−πjl0 , Gj = ∞ X t=1 πjtδψjt,
where the corpus construction remains the same in Equation (5.3). To map the atoms in the document level ψjt to the atoms in the corpus level φk, a series of indicator variablescj ={cjt}
∞
t=1 is introduced and drawn from Mult(β) distribution,
cjt ∼ Mult(β). The mapping is ψjt = φcjt. The document-level indicator variable
zji will map customeri in restaurantj to table t and be used as the index ofcjt for generating the word xji,
zji ∼Mult (πj) xji∼F
φcjzji
.
The graphical model for stick-breaking and Chinese restaurant franchise is de- picted in Figure 5.1c. Bayesian learning means to learn the posterior distribution p(β,φ1:K,Z,C,π1:J) which is intractable. Variational inference scheme uses the tractable proxy distribution q(β,φ1:K,π1:J) as follows
q(β,φ1:K,Z,C,π1:J) = q(β) K Y k=1 q(φk) J Y j=1 " q(πj) nj Y i=1 q(zji) T Y t=1 q(cjt) # (5.5) where qβ|γβ1,γβ2and qπj |γπj1,γπj2
are stick-breaking distributions truncated at level K and T respectively; qzji |µzji
and qcjt |µcjt
are Multinomial distri- butions of T and K dimensions; qφk |λφk is Dirichlet distribution. Note that we removed the hyperparameters of variational distributions to reduce the clutter of Equation (5.5).
Using the standard derivation of variational approximation (Bishop, 2006; Wang
et al., 2011), we can obtain the variational updates for stick-breaking distributions
as follows γπjt1 = 1 +njt, γjtπ2 =α+ T X s=t+1 njs, γkβ1 = 1 +nk, γkβ2 =γ+ K X l=k+1 nl,