It has long been known that user feedback or additional user input can substan-tially boost search quality. The problem faced by information retrieval researchers has been to create interaction mechanisms that are both powerful and that users will adopt. In this dissertation, we have focused on the study and improvement of an already adopted interaction mechanism: find-similar.
Find-similar is an example of the simple interaction mechanisms that we believe are the route to continued improvements in retrieval quality. These mechanisms can be integrated with the existing search interface paradigm of a user entering a text query and receiving a ranked list of results.
The first part of the dissertation focused on measuring the potential of find-similar using simulation. By simulating user behavior given hypothetical user interfaces, we discovered that find-similar has the potential to powerfully improve retrieval qual-ity for users. In the second part of the dissertation, we used network analysis to study find-similar. We developed measures of navigability that allowed us to study find-similar and document-to-document similarity measures in a well-defined manner that is independent of user interfaces and simulated behavior. In combination, the simulation and network analysis experiments showed how a simple search tool like find-similar can help users better find relevant documents.
In summary, our key contributions were:
1. We showed that find-similar has significant potential to improve retrieval perfor-mance. Using simulation, we found that find-similar can match multiple item
relevance feedback. While perceived as an easier to use version of relevance feedback, find-similar can be as powerful as a traditionally styled multiple item relevance feedback.
2. Find-similar is not without its issues. We showed that if users apply find-similar to already good results, they are likely to degrade the retrieval performance.
3. We also showed that an interface should help the user avoid reexamining doc-uments to maximize retrieval quality with find-similar.
4. We showed that find-similar can compensate for poor retrieval quality. In effect, find-similar can be added to existing systems and make them more robust to variations in users and queries.
5. Poorer quality retrieval systems are helped more by find-similar, and find-similar boosts performance more on easier search topics than on difficult topics.
6. We found that a query-biased document-to-document similarity outperforms a similarity measure that ignores the query.
7. We demonstrated that both a local and global measure of the cluster hypothesis should be used. Our global measure, borrowed from the field of network analysis, is new to the field of information retrieval.
8. Our measures of the cluster hypothesis allow us to quantify the value of simi-larity measures like query-biased simisimi-larity without ad-hoc assumptions of user behavior found in our simulation studies.
9. We found the cluster hypothesis to be true to a limited extent on the graph of the World Wide Web. Using both local and global measures, we showed that adding 10 content similarity links to web pages should make the web significantly more navigable.
7.1 Future Work
In this section, we conclude the dissertation by discussing two avenues of future work. First, we outline work to extend find-similar to problems requiring the novelty of information. Second, we discuss the further study of different types of similarity.
7.1.1 Novelty
One aspect of find-similar that we have not examined in this dissertation is the need for some users to find a set of unique answers or information nuggets. For example, many different news agencies will report on the same event. In many cases, there will be several documents that say the same information but simply say it in a different way. Under our present evaluation methodology, which is standard for the IR tasks we examined, such repetition of information is ignored. In other words, the documents or information found by the user for these tasks is not required to be novel compared to the information already found by the user as part of their search.
The TREC complex, interactive question answering (ciQA) track’s evaluation does require information to be novel. As part of the TREC 2007 ciQA track, we have begun studying how best to create an interactive IR system for users searching for answers to complex questions (Smucker et al., 2008). A good information retrieval system should inherently be a good question answering system. Our work to date has been to establish a baseline for human performance with a standard interactive IR system that only allows the user to reformulate the query. Shown in Figure 7.1 is the interface we created for our ciQA experiments. Future work will involve adding additional interaction mechanisms to the interface to boost performance.
Extending find-similar to ciQA would involve two significant changes. The first would be to have find-similar work at the answer level. Users would be able to ask for answers similar to a given answer. Answers are typically on the order of one sentence long compared to the many sentence long documents used in this dissertation. We
Figure 7.1. A screenshot of the interactive, IR system we used for the TREC ciQA experiments. At the top of the interface, we presented the question and a search textbox. The area below the question and search box consisted of three vertically oriented panes. The left pane showed search results. Clicking on a result showed the respective document in the middle pane. The right hand pane provided a textbox allowing the user to enter and save an answer to the question. A list of the user’s saved answers appeared below the answer entry box.
would likely utilize existing techniques for measuring the similarity of sentences (Mur-dock and Croft, 2005; Metzler et al., 2005; Mur(Mur-dock, 2006; Balasubramanian et al., 2007).
The second extension would be to address the novelty requirements of ciQA, which penalize systems for repeating information nuggets in their list of answers. We would first aim to test a version of find-similar that performs sentence retrieval and removes sentences too similar to the given answer. We could apply the techniques of Chapter 5 with the modification that only the novel answers would be used and not all answers in the calculation of the navigability metrics. If the filtering is successful, the navigability of the “answer networks” should be improved.
7.1.2 Multiple Types of Similarity
As we have seen in this dissertation, different types of similarity have better nav-igability than other types of similarity. We have measured navnav-igability for one goal:
finding relevant documents. Users may have multiple goals or needs, and for some goals, different similarity measures will produce more navigable networks. We envi-sion extending find-similar to allow for multiple types of similarity within a single button or link. As mentioned in Chapter 1, CiteSeer provides different types of simi-larity for each document in its repository. Our study would focus on the user interface and usability issues of presenting multiple types of similarity to the user. For exam-ple, while there may be better types of similarity to use to find a piece of information, certain paths can make more sense to users (Teevan et al., 2004). We would be par-ticularly interested in which types of similarity make the most sense for users. In this dissertation we have taken the stance that the primary notion of similarity that matters is the one that makes document networks cluster relevant documents better.
Even though we have a local measure of navigability to capture users’ need to have an information scent (Pirolli, 2007), some similarity measures may offer a better scent for users in the field.