6.5 Experiments
6.5.4 Experiments with HITS CQA
Applying HITS to asker-responder networks in CQA sites may not return optimum results due to the inconsistencies between the activity used to construct the authority networks and the underlying assumption of the algorithm. In order to decrease the effects of these inconsistencies, an adaptation is proposed to HITS algorithm, more specifically to the calculation of hubs score.
This adaptation is specifically proposed for the asker-responder networks. The corresponding experimental results are presented in Tables 6.26 and 6.27, respectively for question routing and reply ranking tasks.
Overall, using the modified HITSCQA returns small improvements (sometimes statistically significant) for both question routing and reply ranking tasks, except for the frequency-based weighted graphs for question routing task. Improvements are more observable in unweighted graphs, and interestingly the unweighted graphs work better in question routing task, while for reply ranking task weighted graphs work more effectively.
The improvements even though small, show the effectiveness of the proposed approach. Not seeing drastic improvements in scores can be expected, and probably due to the characteristics of users’ answering choices. If responders answer questions of askers with similar expertise levels, then using either the summation or average will have similar effects in overall. This proposed algorithm may work better in networks where responders answer questions from users with changing expertise or authority levels.
In order to observe the dependency of this approach on asker-responder networks, and whether it works on other authority graphs, it was also applied to reading and commenting authority graphs from blog collection. As shown in Table 6.28, using HITSCQA on reading or commenting activities causes consistent drops in performance due to inconsistencies between HITSCQAand the underlying assumptions of these interactions. For these interactions, using the original HITS is more ideal.
6.6 Summary
In this chapter, the available social networks are exploited as another source of expertise in order to estimate effective authority-based expertise scores. The authority estimation approaches developed for web pages are analyzed within these environments and the following research question is addressed:
Algorithm Weight P@5 P@10 P@20 MRR MSC@5 MSC@10 MSC@20 NDCG
HITS 7 .0432 .0364 .0270 .1429 .1960 .2720 .3840 .1230
HITSCQA 7 .0456 .0396 .0298rs00 .1501s .2000 .3080rs00 .4240rs00 .1281rs00
HITS 3 .0432 .0356 .0266 .1353 .1920 .2760 .3680 .1224
HITSCQA 3 .0424 .0352 .0282 .1389 .1800 .2600 .3960 .1257s Table 6.26: Question routing performance of using HITSCQAin authority estimation on TC graphs constructed from answering activities in StackOverflow collection.
Algorithm Weight NDCG@1 NDCG@2 NDCG@3 NDCG@4 NDCG@5 BAP
HITS 7 .5451 .6361 .6933 .7586 .8287 .2440
HITSCQA 7 .5610 .6432 .7018r0 .7677r0 .8345r .2720rs00
HITS 3 .5630 .6449 .7009 .7657 .8347 .2640
HITSCQA 3 .5650 .6469 .7072 .7681 .8366 .2760
Table 6.27: Reply ranking performance of using HITSCQAin authority estimation on TC graphs constructed from replying activities in StackOverflow collection.
Activity Algorithm Weight
HITS 7 .0400 .0844 .1075 .0915 .1675 .1228 .3312 HITSCQA 7 .0350 .0821 .1000 .0893 .1575 .1167 .3271 HITS 3 .0750 .1166 .1425 .1228 .2175 .1541 .3707 HITSCQA 3 .0600 .1062 .1250 .1132 .1925 .1443 .3585
Comment
HITS 7 .0400 .0863 .0950 .0885 .1425 .1137 .3284 HITSCQA 7 .0275 .0732 .0850 .0779 .1275 .1015 .3123 HITS 3 .0650 .1122 .1375 .1097 .2150 .1421 .3643 HITSCQA 3 .0625 .1158 .1275 .1066 .1900 .1302 .3544 Table 6.28: Expert ranking performance of using HITSCQAin authority estimation on TC graphs constructed from reading/commenting activities in blog collection.
• RQ2: Do the assumptions of topic-specific authority estimation approaches developed for web pages hold for user authority networks in social media? For the ones that do not, what kind of algorithmic modifications can be performed so that they hold, and is it possible to make additional assumptions and necessary modifications which can provide more effective and efficient topic-specific authority estimations?
Authority estimation depends on two factors: the constructed authority graph and the multi-step propagation algorithm which iterates over this graph. This dissertation focused on both of these aspects and their interactions, and more topic-specific graph construction and authority estimation approaches are proposed to improve effectiveness.
For topic-specific user related tasks, constructing more topic-specific graphs is an important initial step towards effective authority-based expertise estimations. Both topic-independent PageRank and topic-dependent HITS graphs which were developed for web pages are analyzed for user networks. Compared to web pages, the users are less topically clustered and have more mixed connections due to their diverse interests. Based on these differences between web
pages and users, the regular topic-dependent authority graph construction algorithm does not return the expected topical graphs for users. This dissertation proposed a graph construction approach which returns more topic-specific user authority graphs, called the Topic Candidate graphs. The statistically significant improvements observed with these graphs on three tasks show the general effectiveness of the proposed graph construction approach. In addition to effectiveness, the proposed TC graphs also provide significant gains in efficiency by lowering the running times. This is especially important, since these approaches are topic-dependent approaches which require real-time estimation of authority for each given query.
Other than topic-specificity, the authority graphs being weighted (or not) by the frequency of the activity are also analyzed. The experiments with these two types of graphs showed that weighted graphs can be more effective when the authority link between users is originated from a repetitive type of activity like reading or commenting to blog posts. If the authoritative activity is not very repetitive, as answering a certain user’s questions which is not likely in current popular CQAs for most topics, then the difference between authorities estimated from unweighted and weighted graphs are not very observable and consistent.
Furthermore, the connectivities of the user authority graphs are investigated. It has been observed that in graphs with more nodes with second-degree connectivity9, the multi-step prop-agation algorithms provide improvements over one-step propprop-agation approaches. Such graphs include reading or commenting authority graphs, where the authors of the posts also interact with other users’ posts. However, it is not very common to see users who both ask and an-swer questions on the same particular topic. Therefore, topic-specific graphs constructed from answering activities in CQAs may result in bipartite-like graphs with few second-degree con-nected nodes. In these asker-responder networks, one and multi-step propagation approaches return very similar results, which matched with the findings of the prior work on similar graphs.
Constructing more topic-specific authority graphs for asker-responders helps but propagating authority in these networks does not improve the performance compared to one-step propaga-tion. Since one-time propagation approaches iterates over the graph only once, they are also more efficient then multi-step approaches. As a result, connectivity of the graph is important in terms of effective and efficient estimation of authority-based expertise.
In addition to the graphs, the algorithms that iterate over these graphs are also important for effective authority estimation. Authority-based approaches developed for web-pages may not always directly fit to other entities’ authority graphs with different node and interaction types.
For instance, this dissertation showed how the assumption of HITS does not hold on asker responder networks, and so an adaptation is proposed which provided small but consistent improvements for this particular inconsistency.
Apart from checking the applicability and underlying assumptions of these authority based approaches on these networks, a new assumption is also proposed, which is whether being con-nected from topic-specific experts is an indication of being an expert. Experiments performed with using initially estimated expertise scores as influence and teleportation weights outper-formed the original approaches (statistically significant most of the time) in Blog collections. The experiments showed that if the authoritative action requires the tail node of the directed edge to have some prior knowledge or interest on the particular topic, then using expertise as influence helps more than using it with teleportation. However, if such a prior knowledge or interest is not required for the insertion of authority edge, then using expertise to weight teleportation
9Nodes which have both incoming and outgoing links, therefore can connect the nodes of the incoming edges to the nodes of the outgoing edges
probabilities is more effective.
These proposed approaches are tested on two data collections, blog10and StackOverflow, with three interaction types, reading, commenting and answering. Especially working on an intra-organizational blog collection provided the opportunity to use access logs (referred to as reading information), which are not widely available to or studied by the research community. This form of information is useful due to showing more implicit and frequent interactions between users.
The experiments on this collection with reading and commenting activities showed that both of them, whether explicit or implicit, are useful in estimation topic-specific authority.
Overall, experiments performed with these three interaction types on two data collections over three tasks provided a better understanding of the behaviors of authority-based approaches and their applications to user networks. These algorithms provided inconsistent behaviors across prior research. Analyzing these approaches and their underlying assumptions, and authority graph properties such as the nodes, type and frequency of connections between these nodes explained some of these inconsistent behaviors. Adaptations were proposed which improved the effectiveness, efficiency and consistency of these approaches.
10The authority-based expert blogger finding approaches evaluated with company employees’ assessments are also available at Appendix A. The findings from those experiments are similar to the trends we observed from the results presented in this chapter.
Chapter 7
Temporal Approaches
Social media sites, like CQAs, are dynamic environments where new users join constantly, or the level of activity or interest of existing users changes over time. Classic expertise estimation approaches, which were mostly developed for static datasets, cannot effectively model changing expertise and interest levels in these sites. However, these dynamic aspects of the environment should be taken into account for more effective expert identification, especially when the iden-tified experts are expected to take an action, like answering questions, in order to satisfy the information seekers. This thesis proposes to use the available temporal evidence in social media sites to make the existing approaches more dynamic and effective. This chapter starts with an analysis of the dynamism of these sites and then addresses research question RQ3.