4.7 Simulation studies
4.7.2 Study 2
In study 2 we look at a single dataset with n = 40 complete rankings of K = 20 entities from informative (wi = 1) rankers. We also simulate the cluster allocations (for both
rankers and entities) marginally from the prior. The distinct skill parameters and cluster allocations were simulated using α = γs= 1 for s∈ N, and a = 1 so that our base distri-
bution is G0 = Ga(1, 1). The simulation gave three ranker clusters (Nr = 3) containing
24, 12 and 4 rankers, which we label as rankers 1–24, 25–36 and 37–40. Also the ranker clusters contained 8, 6 and 3 entity clusters (N1e= 8, N2e= 6, N3e= 3).
Table 4.4 shows the entity clustering (within each ranker cluster) along with the associated true values of the skill parameters on the log scale. For ease of interpretation, the entities are labelled according to the size of their “true aggregate” skill parameter, largest to
Entity cluster Cr Rankers 1 2 3 4 5 6 7 8 1 1–24 1 6 10,13 3,4,7,9,12,15 2 5,11,14,17–20 16 8 2.47 0.65 0.52 0.40 0.34 0.24 0.02 0.01 2 25–36 2–5,8 1,7,9,11 14,16 12 6,10,15,17 13,18–20 1.72 0.76 0.68 0.42 0.21 0.15 3 37–40 2,6,10 7 1,3–5,8,9,11–20 1.54 1.16 0.64
Table 4.4: True allocation of entities in to clusters, along with the corresponding true parameter value for each of the entity clusters.
smallest, so that they are labelled with the most preferred entity overall first, down to the least preferred entity overall last. Here the true aggregate values are an average of the true parameter values within each ranker cluster, weighted by the size of the ranker clusters. The complete (simulated) rankings analysed within this study can be found in Table B.5 within the appendices.
The purpose of this study is to investigate the ability of our WAND model to (correctly) identify different ranker groups and the associated preferences therein. The analysis given here uses the same base distribution and prior distribution for the entity concentration parameters as in Section 4.7.1, that is, G0 = Ga(1, 1) and γs ∼ Ga(3, 3) for s ∈ N. To
reflect the known ranker heterogeneity within these data we now take aα = bα = 3, that
is, α ∼ Ga(3, 3). We also consider the case where we have only moderate confidence in our rankers being informative by taking pi= 0.5.
Realisations from the posterior distribution were obtained using the (marginal) sampling algorithm outlined in Section 4.6.5 with mr= 2 and me= 3 auxiliary (ranker and entity) variables. The Markov chain was initialised at a random draw from the prior distribution. To obtain 10K (almost) un-autocorrelated realisations from the posterior distribution we performed a burn-in period of 10K iterations and then ran the scheme for a further 1M iterations and thinned the output by a factor of 100. The computational time required to perform inference was (approximately) 17 minutes. The mixing of the MCMC chain was assessed by inspecting trace plots and convergence was assessed by initialising numerous chains at differing starting values and verifying that the resulting posterior distributions were equivalent (up to stochastic noise).
The left plot in Figure 4.8 shows the posterior probabilities, Pr(wi = 1|D), that ranker i
is informative. The plot shows that, in general, the rankers in (true) ranker clusters 1 and 2 (rankers 1–36) are well identified to be informative. However the rankers in (true) ranker cluster 3 are identified as uninformative. The reason for this misidentification is perhaps due to the (true) entity clustering structure present within ranker cluster 3. Table 4.4 shows that this ranker cluster contains only 3 entity clusters, with one of these containing 16 out of the 20 entities, and so it is very likely that rankings in this cluster
0.0 0.2 0.4 0.6 0.8 1.0 Ranker P(w i = 1 | Data) 1 25 37 40 0.0 0.2 0.4 0.6 0.8 1.0 Ranker 25 33 29 34 32 27 28 36 30 26 31 35 40 39 38 37 16 6 21 22 9 10 5 24 7 18 12 19 15 8 2 1 14 20 23 17 3 4 13 11 25-36 37-40 1-24
Figure 4.8: Plot of the posterior probability Pr(wi= 1|D) that ranker i is informative (left), colours distinguish between the “true” ranker clusters. Dendrogram (complete linkage) computed using the dissimilarity ∆ij between rankers i and j (right).
resemble a random permutation of the K entities.
The right plot in Figure 4.8 shows the complete linkage dendrogram determined using dissimilarities ∆ij. The dendrogram suggests there are two ranker clusters (taking dis-
similarity ∈ (0.53, 1)) which separates those rankers numbered {25 − 30, 32, 33, 34, 36} from the remaining rankers. That there are two ranker clusters is supported further by the marginal posterior distribution of the number of ranker clusters: Pr(Nr = i|D) = 0.68, 0.25, 0.06, 0.01 for i = 2, 3, 4, 5. It is not surprising that the analysis has not iden- tified the third ranker cluster as this cluster only contains rankers whose rankings are virtually indistinguishable from random permutations; rather the model prefers to deem such rankers as uninformative and place them within clusters of informative rankers.
Table 4.5 gives the marginal posterior distribution for the number of entity clusters within each ranker cluster, conditional on the posterior modal number of ranker clusters. The modal number of entity clusters within ranker clusters 1 and 2 is six and four respectively (the corresponding true values are eight and six). Here the analysis has correctly identi- fied that ranker cluster 1 is the stronger cluster, in that these rankers are more able to distinguish between entities. The dendrograms in Figure 4.9 suggest that there are five entity clusters within ranker cluster 1 (taking dissimilarity ∈ (0.58, 0.83)) and three entity clusters in ranker cluster 2 (taking dissimilarity∈ (0.50, 0.69)). Notice that in ranker clus- ter 1, the most preferred entity in this cluster (entity 1) has its own cluster, and entities 16 and 8 (in true entity clusters 7 and 8) also form a single cluster; perhaps these are not sur- prising given the “true” values of the skill parameters for these entities within this ranker
Cluster 1 2 3 4 5 6 7 8 9 ≥10
1 0.00 0.00 0.00 0.07 0.20 0.25 0.21 0.14 0.08 0.05 2 0.00 0.11 0.25 0.26 0.18 0.11 0.05 0.02 0.01 0.01
0.0 0.2 0.4 0.6 0.8 1.0 Entity 16 8 1 3 20 18 19 11 17 5 2 9 15 4 13 7 12 10 14 6 0.0 0.2 0.4 0.6 0.8 1.0 Entity 10 20 17 15 18 13 19 6 14 12 16 9 7 11 1 2 3 5 8 4
Figure 4.9: Dendrograms showing the dissimilarity between entities within ranker clusters 1 (left) and 2 (right), conditional on two ranker clusters (Nr= 2).
cluster (see Table 4.4). True entity cluster 6 is fairly well identified with only entity 14 not being included and entity 3 (from true cluster 4) joining the cluster. The remaining two entity clusters identified by the dendrogram house the other entities from true entity clusters 2–5. Within ranker cluster 2 the “true” entity clustering structure from which the data were simulated is largely preserved but the inferred clusters are groups of the “true” clusters, with all entities in “true” cluster 1 being clearly identified in one cluster and those in clusters 2, 3 and 4 in another cluster and those in clusters 5 and 6 in another cluster. That these entity clusters have merged is perhaps not too surprising given the true values (see Table 4.4) and the limited number of rankings observed.
We now investigate the preference ordering of the entities within each ranker group and an overall preference ordering; see Table 4.6. Here the preference ordering within each ranker group has been determined by the posterior mean of the “skill” parameters, averaged over both the entity clustering and the allocation of rankers to each ranker group. The overall preference ordering has been further averaged over all ranker clusters. Comparing these preference orderings with the truth (in Table 4.4) we see that the WAND model has performed fairly well in recovering the true preferences expressed in ranker clusters 1 and 2, especially for those entities which are the most and least preferred within each ranker group. Not surprisingly there is an increased level of misidentification in the middle ranks of the preference ordering for both ranker clusters, and particularly so for ranker cluster 1. This is perhaps due, in part, to the true values of the skill parameters in entity clusters 2–6 within each ranker cluster being fairly similar, with those in ranker cluster 1 being the most similar; see Table 4.4.
The entities in Table 4.6 are listed in order of their overall “true” skill parameter. Even though the WAND model has allowed for differences between rankers, the inferred overall ordering is very different from the “true” order. That said, the inferred orderings within the ranker clusters are very similar to the “true” orderings and give a much better account of the heterogeneity within the model underpinning the data. This illustrates how inferring preference orderings using overall (population level) summaries of heterogeneous rankers
C1r C2r Aggregate Rank Entity Mean (SD) Entity Mean (SD) Entity Mean (SD)
1 1 3.54 (1.57) 2 1.91 (1.11) 1 2.93 (1.15) 2 7 1.02 (0.54) 8 1.77 (1.08) 7 1.13 (0.47) 3 13 1.02 (0.54) 5 1.73 (1.07) 4 1.02 (0.41) 4 10 0.91 (0.46) 4 1.71 (1.06) 2 0.97 (0.40) 5 14 0.89 (0.45) 3 1.65 (1.05) 14 0.96 (0.40) 6 6 0.87 (0.44) 16 1.51 (1.02) 12 0.95 (0.39) 7 12 0.83 (0.42) 1 1.44 (0.99) 9 0.89 (0.38) 8 15 0.76 (0.39) 7 1.41 (0.98) 3 0.82 (0.35) 9 4 0.75 (0.40) 9 1.36 (0.96) 13 0.80 (0.39) 10 9 0.70 (0.38) 11 1.31 (0.95) 5 0.78 (0.34) 11 2 0.60 (0.35) 12 1.23 (0.92) 10 0.73 (0.33) 12 3 0.49 (0.28) 14 1.13 (0.87) 6 0.69 (0.31) 13 20 0.44 (0.24) 10 0.25 (0.21) 11 0.64 (0.29) 14 18 0.44 (0.24) 17 0.22 (0.17) 15 0.62 (0.29) 15 17 0.41 (0.21) 15 0.22 (0.16) 8 0.52 (0.31) 16 5 0.41 (0.21) 20 0.22 (0.16) 16 0.47 (0.29) 17 11 0.37 (0.19) 13 0.21 (0.15) 20 0.39 (0.18) 18 19 0.37 (0.18) 19 0.21 (0.15) 18 0.38 (0.18) 19 16 0.05 (0.07) 6 0.21 (0.15) 17 0.36 (0.16) 20 8 0.03 (0.08) 18 0.20 (0.14) 19 0.33 (0.14)
Table 4.6: Posterior preference orderings within ranker clusters 1 and 2 (conditional on two ranker clusters) and the overall/aggregate ranking, with mean (and standard deviation) of their skill parameters.
can be very misleading even when knowing the skill parameters, let alone when attempting to infer their values.
4.8
Summary
In this chapter we have described the Adapted Nested Dirichlet process prior which facil- itates two-way clustering on both rankers and entities (within ranker groups). We then used this prior to form the WAND model by taking the underlying ranking distribution to be the Weighted Plackett–Luce model. Two approaches to inference for the WAND model were then considered. In Section 4.5 we appealed to a conditional sampling approach. Al- though intuitive, this approach to inference for DP mixtures comes with drawbacks when compared to marginal sampling schemes, as discussed in Chapter 3. In Section 4.6 we discussed how a marginal scheme for posterior sampling can be constructed for this adap- tation of the NDP. The marginal posterior sampling scheme we outlined in Section 4.6.5 allows for fast and efficient inference under our WAND model.
The richness of information in the posterior distribution allows us to infer many details of the structure both between ranker groups and between entity groups (within ranker groups). The high dimension of the posterior distribution can make the production of insightful but simple summaries quite difficult and we have explored different approaches, ranging from conditioning on modal number of groups to adopting a classification based on calculations from a dissimilarity matrix summary.
In the next chapter we consider two real datasets that have been analysed in the literature, and compare their conclusions with those obtained from fitting the WAND model.
Real data analyses
5.1
Roskam’s data set
In this section we consider a dataset originally collected in 1968 by Roskam, more recently studied by de Leeuw (2006). The data are available in the R package homals (de Leeuw and Mair, 2009) and are also given in Table B.6 within the appendices. The data consist of rankings obtained from n = 39 psychologists within the Psychology Department at the University of Nijmengen (Netherlands). Each ranker gives a complete ranking of K = 9 sub-areas (entities), listed according to how appropriate the sub-areas are to their work. The sub-areas are: SOC - Social Psychology, EDU - Educational and Developmental Psychology, CLI - Clinical Psychology, MAT - Mathematical Psychology and Psychological Statistics, EXP - Experimental Psychology, CUL - Cultural Psychology and Psychology of Religion, IND - Industrial Psychology, TST - Test Construction and Validation, and PHY - Physiological and Animal Psychology.
The heterogeneity within these data has been analysed by de Leeuw (2006) using a non- linear principal component analysis to detect groupings within the rankings. Their anal- ysis supported the idea that there are two groups of rankings: one group which favours the qualitative fields and the other favouring the quantitative fields of psychology. A homogeneity analysis was later performed by de Leeuw and Mair (2009) which exposed groupings of entities within the rankings. More recently Choulakian (2016) performed a Taxicab correspondence analysis to look at structure both between the rankings and the entities within ranker groups. Their results support the conclusions of de Leeuw (2006) and suggest that the psychologists comprise two homogeneous groups with 23 and 16 mem- bers respectively. Within the larger ranker group they obtain the entity clustering{MAT, EXP} {IND, TST} {PHY, SOC, EDU} CLI CUL, where means “is preferred to”, and quantitative areas of psychology appear to be preferred. The corresponding clus-
tering of entities for the other ranker group is {EDU, CLI, SOC} {CUL, MAT, EXP} {TST, IND} PHY, and here qualitative areas of psychology appear to be preferred. They also conclude that the larger ranker group is somewhat more homogeneous than the smaller group.
We now use our WAND model to investigate subgroup structure in these data and take our prior specification for the base distribution and concentration parameters to be a = 1 and aα = bα = 1, aγ = bγ = 3. These data contain orderings of individual preferences
which we believe to be informative and so take pi = 0.75. The posterior distribution is
fairly robust to this choice; a sensitivity analysis follows in Section 5.1.1. We report the results from a typical run of our MCMC scheme initialised from the prior, with a burn-in of 10K iterations and then run for a further 1M iterations and thinned by 100 to obtain 10K (almost) un-autocorrelated realisations from the posterior distribution. Convergence was assessed by using multiple starting values, inspection of traceplots of parameters and the log complete data likelihood, and standard statistics available in the R package coda (Plummer et al., 2006). The MCMC scheme runs fairly quickly, with C code on a single thread of an Intel Core i7-4790S CPU (3.20GHz clock speed) taking around 5 minutes.
Table 5.1 shows both the prior and posterior distribution for the number of ranker clusters. The data clearly have been informative and suggest that it is likely that there are between two and four ranker groups, with two groups being most plausible. Note that there is almost no posterior support to suggest there is a single (homogeneous) ranker group and so an aggregate ranking from this dataset may be misleading. Figure 5.1 shows the den- drogram of rankers along with the posterior probability that each ranker is informative. The dendrogram suggests that there are two ranker groups (taking dissimilarity > 0.60), and this is consistent with the posterior distribution in Table 5.1 and the conclusions of previous analyses. We note that the data are consistent with most rankers being infor- mative (with Pr(wi = 1|D) ≥ 0.8), an increase from their prior probabilities (pi = 0.75).
Also the rankers whose probabilities have decreased (rankers 1, 5, 8, 10, 13, 14, 15, 31) are those with (slightly) different preferences and hence late to join the right-hand cluster in the dendrogram.
We now turn to the subgroup structure of entities within the ranker clusters, and here we condition on there being two ranker clusters. Figure 5.2 shows the (marginal) posterior distribution for the number of entity clusters within each ranker cluster together with the
1 2 3 4 5 6 7 ≥ 8
Posterior 0.00 0.43 0.33 0.16 0.06 0.02 0.00 0.00 Prior 0.20 0.18 0.16 0.13 0.10 0.08 0.05 0.10
0.0 0.2 0.4 0.6 0.8 1.0 Ranker 14 26 28 37 33 17 16 18 39 25 21 38 35 27 23 19 20 24 22 5 15 10 8 13 31 1 6 7 12 9 34 11 32 36 29 3 4 30 2 0.0 0.2 0.4 0.6 0.8 1.0 Ranker Pr(w i = 1 | Data) 1 10 20 30 39
Figure 5.1: Roskam’s dataset: Dendrogram (left) showing the ranker cluster structure along with the posterior probability, Pr(wi= 1|D), for each ranker i (right).
prior distribution. The dendrograms in Figure 5.3 show the entity clustering structure in each ranker cluster. We define entity clusters at dissimilarities in ranges (0.45,0.95) and (0.63,0.89) for rankers groups 1 and 2 respectively and form a preference ordering of these entity clusters by examining the marginal posteriors for the skill parameters λcidcij
within each ranker group ci. Conditioning on these allocations to both ranker and entity
groups and ordering by posterior mean, we obtain {EXP, MAT} {TST, PHY, IND} {EDU, SOC, CLI} {CUL} (with entity cluster means 3.02, 0.72, 0.22, 0.06) in ranker cluster 1 and{SOC, EDU, CLI, MAT} {CUL, IND, EXP, TST} {PHY} (with entity cluster means 1.96, 0.82, 0.12) in ranker cluster two. These entity clusters (within ranker groups) are similar to those given by Choulakian (2016). Also if we use the average value of Pr(wi = 1|D) as a measure of homogeneity within a ranker cluster then we obtain 0.68
and 0.56 for clusters 1 and 2 respectively, which again agrees with the Choulakian (2016) conclusion that ranker cluster 1 is more homogeneous than ranker cluster 2. Note that, for this data analysis, we obtain a very similar entity ordering using marginal posterior means of the skill parameters within each ranker group (marginal over the distribution of entity clusters); see Table 5.2. Indeed the table suggests that the ranker groups almost have opposite (reverse) preferences to each other.
We looked at the sensitivity of the posterior distribution (and inferences) to modest changes to the prior distribution. The posterior distribution was fairly insensitive to
2 4 6 8 0.0 0.1 0.2 0.3 0.4
Number of entity clusters
Pos terior pr obability Ranker Cluster 1 Ranker Cluster 2 Prior
Figure 5.2: Prior and marginal posterior densities for the number of entity clusters within each ranker cluster (conditional on two ranker clusters).
0.0 0.2 0.4 0.6 0.8 1.0
CUL EDU SOC CLI MAT EXP IND TST PHY
0.0 0.2 0.4 0.6 0.8 1.0
PHY SOC EDU CLI MAT CUL IND EXP TST
Figure 5.3: Roskam’s dataset: Dendrograms showing the entity clustering structure within ranker cluster 1 and 2 (left and right respectively) conditional on two ranker clusters.
Ranker Rank
cluster 1 2 3 4 5 6 7 8 9
1 EXP MAT TST PHY IND EDU SOC CLI CUL
3.13 2.68 0.76 0.70 0.63 0.27 0.22 0.20 0.07
2 SOC EDU CLI MAT CUL IND EXP TST PHY
1.95 1.75 1.49 1.32 0.94 0.90 0.87 0.87 0.10
Table 5.2: Roskam’s dataset: entity rankings by posterior mean within ranker cluster (conditional on two ranker clusters). Rank 1 corresponds to the entity most preferred within each cluster.
changes in the index (a) of the gamma base distribution and to changes in the parame- ters (aα, bα, aγ, bγ) of the gamma prior distributions for the concentration parameters.
The posterior distribution was most sensitive to changes in the prior probabilities (pi)
of rankers being informative. Not surprisingly most affected by such changes were their posterior equivalents Pr(wi = 1|D) though the conclusion of two ranker groups and the
membership of these groups was robust. The allocation of entities to groups (within each ranker cluster) was also fairly robust, with only a minor change in the allocation in the p = 0.85 case. Section 5.1.1 contains the (ranker and entity) dendrograms and plots of Pr(wi = 1|D) for pi = 0.65 and pi = 0.85 in addition to the choice pi = 0.75 used in this
analysis.