Estimating the Test Documents’ Topic Distributions

The topic distribution θ0 on the test documents are required in performing various evaluations on topic models. These topic distributions are unknown and hence need to be estimated. Standard practice uses the first half of the text in each test document to estimateθ0, and uses the other half for evaluations. However, since abstracts are relatively shorter compared to articles, adopting such practice would mean there are too little text to be used for evaluations. Instead, we used only the words from the publication title to estimate θ0, allowing more words for evaluation. Moreover, title is also a good indicator of topic so it is well suited to be used in estimating θ0. The estimated θ0 will be used in perplexity and clustering evaluations below. We note that for the clustering task, both title and abstract text are used in estimating θ0 as there is no need to use the text for clustering evaluation.

We briefly describe how we estimate the topic distributions θ0 of the test docu- ments. Denoting wdn to represent the word at position n in a test document d, we estimate the topic assignment zdn of wordwdn independentlyby sampling from their predictive posterior distribution given the learned author–topic distributions ν and topic–word distributionsφ:

p(zdn|wdn,ν,φ)∝νadkφkwdn , (7.30)

noting that the intermediate distributionsφ0 are integrated out (see Appendix A.6). We then build the customer countscθd _{for these test documents from the sampled}

z (for simplicity, we set the corresponding table counts as half the customer counts). With these, we then estimate the document–topic distributionθ0from Equation (7.20). If citation network information is present, we refine the document–topic distribution θ0_d using the linking topic ydj for train document jwhere xdj = 1. The linking topic ydj is sampled from the estimatedθ0_d and is added to the customer countscθ0d, which

§7.7 Experiments and Results 97

Doing the above gives a sample of the document–topic distribution θ_d0(r). We adopt a Monte Carlo approach by generatingR= 500 samples ofθ0_d(r), and calculate the Monte Carlo estimate ofθ0_d, as described in Section 5.5.1.

7.7.3 Goodness-of-fit Test

Perplexity, as detailed in Section 5.5.2, is a popular metric used to evaluate the goodness-of-fit of a topic model. Since perplexity is negatively related to the like- lihood of the observed words W given the model, the lower the better. Here, the perplexity can be computed as:

perplexity(W) =exp −∑ D d=1∑ Nd n=1logp wdn|θd0,φ ∑D d=1Nd ! , (7.31)

where p w_dn|θ0_d,φis obtained by summing over all possible topics:

p wdn|θ0d,φ = K

∑

∑

Here we note that the distributionsφ0andθare integrated out (following the method shown in Appendix A.6).

We calculate the perplexity corresponds to both the training data and test data. Note that the perplexity estimate is unbiased since the words used in estimatingθ0 are not used for evaluation. We present the perplexity result in Table 7.4, showing the significantly (at 5 % significance level) better performance of CNTM against the baselines on the ML, M10 and AvS datasets. For these datasets, inclusion of citation information also provides additional improvement for model fitting, as shown in the comparison of the CNTM with and without network component. Note that for the CS, Cora and PubMed datasets, the nonparametric ATM was not performed due to the lack of authorship information.

7.7.4 Document Clustering

Next, we evaluate the clustering ability of the topic models. As mentioned in Sec- tion 7.6, for M10 and AvS datasets, we assume their ground truth classes correspond to the query categories used in creating the datasets. The ground truth classes for CS, Cora and PubMed datasets were provided. Note we do not use the ML dataset since it has only one category.

We evaluate the clustering performance with purity and normalised mutual infor- mation (NMI), as discussed in Section 5.5.4. Purity is a simple clustering measure which can be interpreted as the proportion of documents correctly clustered, while NMI is an information theoretic measures used for clustering comparison.

98 Bibliographic Analysis on Research Publications

Table 7.4: Perplexity for the training and test documents for all datasets, a lower perplexity means better model fitting. We can see that the CNTM with the citation network generally outperforms the other topic models in model fitting. Note that the nonparametric ATM is not performed for the last three datasets due to the lack of authorship information in these datasets.

Model Perplexity

Training Test Training Test

ML M10 Bursty HDP-LDA 4904.2±71.3 4992.9±65.6 2467.9±34.8 2825.6±61.4 Nonparametric ATM 2238.2±12.2 2460.3±11.3 1822.4±15.0 2056.4±18.3 CNTM w/o network 2036.3±4.6 2118.1±3.7 922.6±11.0 1263.9±8.8 CNTM w network 1919.5±8.8 2039.5±11.7 910.2±13.3 1261.0±25.7 AvS CS Bursty HDP-LDA 2460.4±66.4 2612.8±91.7 1498.4±4.1 1616.8±38.8

Nonparametric ATM 2225.9±45.5 2511.9±52.4 N/A N/A

CNTM w/o network 1540.2±18.5 1959.2±2.4 1506.8±4.4 1609.5±39.2 CNTM w network 1515.9±2.1 1938.9±10.4 1168.6±27.3 1588.2±93.9 Cora PubMed Bursty HDP-LDA 678.3±1.7 706.3±16.8 300.0±0.3 300.2±1.2 CNTM w/o network 554.8±14.1 881.1±110.9 299.9±0.2 300.1±1.3 CNTM w network 527.0±8.7 719.0±111.4 350.5±20.1 297.3±3.2

The clustering results are presented in Table 7.5. We can see that the CNTM greatly outperforms the PMTLM in NMI evaluation. Note that for a fair comparison against PMTLM, the experiments on the CS, Cora and PubMed datasets are evaluated with a 10-fold cross validation. Additionally, we would like to point out that since no author information is provided on these 3 datasets, the CNTM becomes a variant of HDP-LDA, but with PYP instead of DP. We find that incorporating supervision into the topic model leads to improvement on clustering task, as predicted. However, this is not the case for the PubMed dataset, we suspect this is because the publications in the PubMed dataset are highly related to one another so the category labels are less useful (see Table 7.3).