RQ3: Exploiting Temporal Dynamics in Term Popularity to Improve

In this section I discuss the findings presented in Chapter 6 with respect to RQ3 of this the- sis: “Term popularity exhibits many patterns and trends over time in a time-based document collection. Can these temporal dynamics be exploited to improve IR system effectiveness over time?”

Information collections capture various temporal dynamics in past information behaviour as content authors discuss topical events and phenomena over time. In Chapter 3, I explored

7.2 Discussion and Contributions several temporal dynamics present in time-based collections, including term popularity (e.g., term and document frequency), specificity and relationships.

Although time-based collections – such as ever-evolving web, news and social media – are becoming increasingly common, conventional IR techniques typically view the collection as stationary over time. This assumption may lead to sub-optimal retrieval performance when the composition of the collection changes as new documents covering both old and new topics are added over time, and the underlying statistical distributions (e.g., term and document frequency) used in retrieval models become subject to temporal dynamics. In any case, IR approaches which ignore collection evolution do not exploit the rich temporal dimension available to characterise the meaning and structure of information beyond static distributions during retrieval.

Past work showed the temporal dynamics of term popularity in a collection can be used to estimate semantic relationships (i.e., temporal semantic similarity) between index terms over time (Radinsky et al., 2011). However, this finding had not been previously operationalised to improve IR system effectiveness in time-based collections. Consequently, I set out to exploit these temporal dynamics to improve IR system effectiveness through query expansion (QE), based on pseudo-relevance feedback (PRF).

PRF is a method of improving retrieval performance by expanding the original query with distinctive features found in the top-retrieved pseudo-relevant documents. Existing PRF ap- proaches typically rely upon the measures of index term discriminability to identify terms highly related to the query topic, in order to determine most distinctive terms for feedback via QE. I proposed the Temporal Semantic Query Expansion (TSQE) PRF approach, which se- lects QE terms based on both temporal (i.e., temporal semantic similarity between terms) and non-temporal (i.e., term frequency in PRF documents) evidence. Temporal index term seman- tic similarity is measured by the Pearson correlation between document frequency temporal dynamics of the terms. Terms selected for QE consist of terms which are strongly temporally interdependent on one another (i.e., there is a high degree of temporal semantic similarity amongst them), and also at the same time, are distinctive in PRF documents. I name these the topic’s “chronotype” terms, and hence posit they are the best terms to exploit for QE in a time-based collection. Note that while this approach is computationally demanding, I pro- posed various methods to make it more tractable. That said, future work will need to explore further techniques and trade-offs for reducing the modelling complexity necessary (in partic- ular, the size of the dense Temporal Semantic Network graph) to make the approach more practicable in an online system, yet still maximising retrieval performance. This is likely to include better candidate QE term selection, for example, focusing on more definitive entities rather than simply all terms found feedback documents, regardless of their importance.

7.2 Discussion and Contributions I conducted extensive retrieval experiments on four diverse time-based test collections (i.e., two news collections, Twitter microblogs and web blogs) to demonstrate the retrieval im- provement provided by the TSQE approach in differing scenarios. Based on results presented in Table 6.8 on page 155, I found the proposed TSQE approach, mixing both temporal and non-temporal evidence, is able to significantly outperform a Language Model with Tempo- ral Query Modelling PRF baseline for many collections and measures, particularly for the MAP retrieval performance metric. Accordingly, this increased MAP shows TSQE is able to higher rank relevant results the majority of cases. In some cases TSQE did not outper- form the baseline, or only provided statistically insignificant improvement, however there are likely inconsequential factors causing this effect. In particular, the Blogs06 collection proved problematic, likely due to its unreliable composition and associated temporal evidence is- sues. Retrieval precision metrics were less reliably improved by TSQE; precision is often adversely affected by PRF models, which often lead to query topic drift if the initial retrieval is poor (Carpineto and Romano, 2012). Additionally, the lack of statistical significance in some cases may be down the relatively small test topic sets available for some collections

following train/test splits – for example only 25 queries for the TREC Microblog collection1_.

Importantly, analysis presented in Table 6.3 on page 150 showed different retrieval goals (i.e., recall or precision) in diverse time-based collections require varying PRF and TSQE pa- rameters to achieve optimal performance, demonstrating the varying temporal nature of both queries and collections. However, through train-test validation (i.e., Table 6.8) I demonstrated that these parameters could be effectively learnt for each collection and topic set. As a result, overall I have shown incorporating temporal dynamics into QE (based on PRF) mixing both temporal (i.e., temporal dynamics) and non-temporal evidence is beneficial for improving IR system retrieval effectiveness for time-based collections.

Contributions

Addressing RQ3, this thesis makes several novel contributions for improving retrieval perfor- mance in time-based collections, based on exploiting temporal dynamics in term popularity for QE through PRF:

1. A novel network-based structure, named a Temporal Semantic Network (TSN) to cap- ture temporal and non-temporal evidence contained in PRF documents and index term temporal dynamics.

2. A novel approach for temporal semantic query expansion flexibly combining temporal and non-temporal evidence, based on network analysis of the TSN to identify a query

1_{An additional set of 50 query topics for the TREC Microblog 2011 collection has become available since}

7.3 Conclusion