Prototype “savoir”

“Savoir” is the most complete of our prototypes. It accepts all three types of relational queries and it proposes a tailored solution for each type of query. Queries are input in quasi-natutal language (see section 9.2 for untyped relational query). This querying convention allows to determine the query type through simple parsing. For each query type, we trigger the adequate solution.

To have results presented in a uniform format, we use a table which has instance names as columns and attribute names as rows. If the query is an attribute query, the user will be presented the retrieved values within a

Figure 9.8: The “pouvoir” result on the query “Pink Panther”

Figure 9.9: The “revoir” result on the query “music albums” issued through class instances

simple table headed by the instance name and the attribute name. Instance queries are answered with attributes and their values and few images. Class queries are answered similarly, but their table has multiple columns, one per

Figure 9.10: The “savoir” result on the query “president of France”

Figure 9.11: The “savoir” result on the query “Mac Book Pro”

Figure 9.12: The “savoir” result on the query “mobile phones” issued as instances separated by commas

instance.

In figure 9.10, we see how results look like for attribute queries. Figure 9.11 shows results for the instance query “Mac Book Pro”, while figure 9.12 show results for the class query “mobile phones” issued through three instances (we reduced the result on the third instance, because it did not fit into the page.). We can see that result presentation is relatively uniform and readable.

We believe that these results are encouraging.

9.5

Conclusions

In this chapter, we have shown 4 prototypes of relational aggregated search. The first three are dedicated to one type of query respectively to attribute queries, instance queries and class queries. The forth prototype allows all three types of queries and answers each type of query with the appropriate designated solution. We can see this chapter as a prototype oriented trip within relational aggregated search. During this trip, we highlight issues and possible solutions. The combination of all solutions is a relational aggregated search system with encouraging performance.

Part 4: Cross-vertical

aggregated search: Interest

and evaluation

Interest and evaluation of

cross-vertical aggregated

search

10.1

Introduction

Cross-vertical aggregated search can be seen as a special case of federated search. The interest (advantages, novel issues) of this research direction remains to be explored. The work presented in this chapter targets at the same time interest and evaluation of cross-vertical aggregated search. We consider different ways to collect relevance assessments in this context trying to determine the advantages of cross-vertical aggregated search. The goals are multiple. On one side we want to study why and how multiple and diverse sources can be useful. On the other side, we want to identify the best ways to capture this utility.

We found that there are two types of relevance assessments that are used in literature for cvAS. In [14, 104, 108], human judges (assessors) are given a query and they have to assign to it to one or more vertical intents if they find so. A query has a vertical intent is there exists a vertical search that is likely to answer the query. In this kind of setup, assessors do not know the real need behind the queries and they are not shown any concrete results from search engines. We say that we have “relevance by intent assessments” on “short text queries”.

In [166, 168], Sushmita et al. investigate some of the advantages of cross-vertical aggregated search interfaces. They show that cross-vertical AS increases the quantity and diversity of relevant results accessed by users. Here, human assessors are shown results from each source being used and queries are associated with a description of the information need. We say that we have relevance by content assessments on queries with a fixed in- formation need (fixed need queries). Until now, there exist no studies that

relate or compare the two types of relevances (by intent and by content). As well, the impact of the fixed information need with respect to the short query (free to interpret) has not been studied in the context of cross-vertical aggregated search.

Our goal in this work is to reconsider the evaluation of the interest of AS, by exploiting both relevance by intent and by content, and by using queries with or without fixed information need. Our research questions include:

∙ What is a relevant source?

∙ How realistic are relevance by intent assessments and relevance by content assessments?

∙ Depending on the evaluation setup, which is the distribution of rele- vant sources?

∙ Which is the contribution of vertical searches to traditional Web search?

∙ Why can two or more sources be complementary at the same time? Are different sources complementary to each other?

∙ How should we setup evaluation of cross-vertical aggregated search?

Some of these questions have already been examined in literature (see sections 5.6 and 5.2 for details). Our aim is to revisit interest and evaluation of cross-vertical aggregated search by exploiting two types of relevance (by intent and content) and two types of queries (fixed need and short query). For this purpose we conducted a study with many human participants. We examined four search situations where we vary the type of relevance and the type of query.

This chapter is structured as follows. Sections 10.2 and 10.3 introduce our study setup and its results. In section 10.4 we provide discussion about the results and we give some thoughts about the evaluation of cvAS.

10.2

Experimental setup

We built a study which involves relevance assessments collected through hu- man participants. The goal is to determine the advantages of cross-vertical aggregated search as well as the evaluation issues. We aim to investigate on the notion of source relevance and the ways to capture it.

Our study is composed of 4 tasks which involve real participants evalu- ating relevance across 9 different sources. We consider two types of source relevance (by intent or content) and two types queries (with or without a fixed information). The two types of relevance and the two types of queries are defined below.