Results using the experiment with real users

Besides the extensive experiments performed using simulation, we also performed an experiment with real users. The experiment consisted of 10 small tasks∗_{, where}

each task would take the user approximately one minute to complete. The tasks were generated by a script that randomly selects products and includes all properties of the product in the task description. However, for the sake of brevity, properties with multiple values (e.g., ‘Audio Formats’) were reduced to one (randomly selected) value. For each task, the user was given a set of product features. The users were instructed to find the product(s) that matched all the given properties. We evaluated two systems, where each user performed the first half of the tasks with one system and the second half of the tasks with the other system. The order of the systems was alternated among users in order to compensate for the the learning effect that may occur. One system was a faceted search interface using the algorithm proposed in this dissertation† _{and the other system was a ‘standard’ faceted search interface}‡_{. The}

‘standard’ faceted search interface has no special features other than those commonly encountered on the Web and employs a fixed facet list, which is obtained from the Web shop from which the data set is originating (Tweakers.net, 2016).

∗_{all tasks are available at https://db.tt/5DRnsIhS} †_{available at http://facet-sorting.eur.dvic.io} ‡_{available at http://std-prod-search.eur.dvic.io}

We had a total of 27 users who participated in the experiment, consisting of 17 males and 10 females. There were 19 users that were between 20 and 30 years old, 6 users that were between 31 and 40 years old, and 2 users that was between 40 and 50 years old. These users were mostly students and colleagues from our university and other universities and there was no financial reimbursement for the participation in the experiment.

Table 6.7 shows the behavior of the users who participated in the experiment, for each of the systems. We can see that most users chose to filter based on the qualitative facets (such as the brand), as indicated by the event ‘List facet select’. We notice that users needed less numeric facet changes with our approach than with the standard approach (event ‘Numeric facet change’). The results from our user study also suggest that users do not reformulate the query often. Table 6.7 shows that the filters were cleared only twice in the whole study (event ‘Clear all filters’). We can also see that the users spend more time drilling down or rolling up (events ‘List facet select’ and ‘List facet deselect’). Using a paired t-test (measured per task), we can conclude that the users significantly had less interaction (i.e., less events) with our approach than with the standard approach (p = 0.001867). We also considered the user effort in terms of how long it took the users to complete the tasks. On average, the users spent 72.4 seconds per task with our approach and 79.9 seconds with the standard approach. The standard deviation is 33.2 seconds for our approach and 33.0 seconds for the standard approach. A paired t-test shows that the difference is significant although the evidence is not very strong (p = 0.047170). This might be due to the fact that there is a large difference among users and 27 users is too little to factor out that effect.

Event type Standard approach Our approach List facet select 364 376

Toggle collapsed 182 143 Numeric facet change 198 84 List facet deselect 18 2 Boolean facet change 5 2 Numeric facet remove 2 4 Boolean facet remove 4 0 Clear all filters 2 2 Change page number 2 3

6.5

Conclusion

Nowadays most Web shops employ a form of faceted navigation in their user interface, often referred as ‘faceted search’. This type of search has many advantages over keyword-based search, as it enables the user to progressively narrow the search results in a number of steps by choosing from a list of query refinements, instead of having to provide the query at once. However, the current state of faceted search is suffering from two major issues. First, most Web shops use a ‘expert-based’ selection procedure for facets (Amazon.com, 2017a; Kieskeurig.nl, 2017; Tweakers.net, 2016), requiring significant manual work as often many facets are involved. Second, a fixed facet list does not optimally make use of the interactive nature of faceted search, i.e., the importance of facets can change drastically depending on the query.

In this work, we proposed an approach that automatically orders facets such that the user finds its desired product with the least amount of effort. The main idea of our solution is to sort properties based on their facets and then, additionally, also sort the facets themselves. We use different types of metrics to score qualitative and numerical properties. For property ordering we want to rank properties descending on their impurity, promoting more selective facets that will lead to a quick drill-down of the results. Furthermore, we employ a weighting scheme based on the number of matching products to adequately handle missing values and take into account the property product coverage.

We evaluate our solution using an extensive set of simulation experiments, com- paring it to three other approaches. While analyzing the user effort, especially in terms of the number of clicks, we can conclude that our approach gives a better per- formance than the benchmark methods and in some cases even beats the manually curated ‘Expert-Based’ approach. In addition, the relatively low computational time makes it suitable for use in real-world Web shops, making our findings also relevant to industry. These results are also confirmed by a user-based evaluation study that we additionally performed.

In future we would like to replicate our study on a different domain than cell phones, thereby addressing one of the limitations of the current evaluation. Also we would like to investigate the use of other metrics, such as facet and product popularity, for determining the order and optimal set of facets.

Approximate Search and

User Preference Ranking in

Faceted E-commerce Search∗

O

ucts are sometimes not available and Web shops fail to provide similar products due to their exclusive reliance on Boolean faceted search. User prefer- ences are also often not taken into account. In order to address these problems, we present a novel framework specifically geared towards approximate faceted search within the product catalog of a Web shop. It is based on adaptations to the p-norm extended Boolean model, to account for the domain-specific charac- teristics of faceted search in an e-commerce environment. These e-commerce specific characteristics are, for example, the use of quantitative properties and the presence of user preferences. Our approach explores the concept of facet similarity functions in order to better match products to queries. In addition, the user preferences are used to assign importance weights to the query terms. Using a large-scale experimental setup, we conclude that the proposed algorithm outperforms the considered benchmark algorithms.

∗_{This chapter is based on the article “D. Vandic, L. Nederstigt, F. Frasincar, and U. Kaymak.}

Approximate Search and User Preference Ranking for Faceted Navigation. ACM Transactions on the Web, 2017, under review.”

7.1

Introduction

E-commerce has nowadays become very popular among consumers. However, there are many Web shops with very large product catalogs, which makes it difficult for users to find their desired product. It has been shown that many users encounter this difficulty while shopping online, which can negatively impact the turnover for Web shop owners (Horrigan, 2008). This emphasizes the need for better search interfaces for browsing through product catalogs on the Web.

Throughout the years various interaction paradigms have been proposed to deal with the above problem. Faceted search is one of the most popular paradigms for browsing structured data in information systems (Hearst, 2006), especially product catalogs in e-commerce stores. This paradigm is useful for exploratory search, as users can iteratively select facets that they find important to constantly refine their query and enhance the result set (Kules et al., 2009). Furthermore, it has been shown that faceted search is perceived as easy to understand and effective at guiding people to relevant documents within catalogs (Fagan, 2013).

Despite the advantages of faceted search, there are also some serious drawbacks. Traditionally, faceted search imposes a rather strict Boolean model on whether a document matches the query terms: either it matches the query or it does not. While this model allows for a quick drill-down into the catalog, it also means that a user might miss some potentially desired products if they do not completely match the query. This results in a user having to change the query afterwards to include more documents, which increases search time and makes exploratory search more error-prone.

Such strict faceted search is not desired in e-commerce, as users usually do not have completely strict requirements. For example, a user searching for a smartphone under e200 with ‘Android’ as Operating System and ‘Samsung’ as brand, might also consider Android smartphones from other brands if they are much cheaper than similar phones from Samsung. At the same time, having Android as the Operating System could be a hard requirement for the user. Another problem with strict faceted search is that there is no adequate ranking method. This is usually resolved by letting the user sort the list on one metric, such as price or popularity, but it would be more useful if the products could be sorted on how well they match the query. For this purpose, the user should also be able to rank the specified requirements on importance (e.g., a battery life requirement might be more important than a color requirement).

Most of the currently proposed approaches that aim to tackle this problem are not specifically geared towards the domain of e-commerce. In particular, the existence of quantitative facets, such as price, might pose a problem to these algorithms, as they do not consider nearly identical numerical values as being similar to a certain degree. Furthermore, past approaches for e-commerce also do not allow the user to explicitly specify which requirements are important and which ones are not. To our knowledge, a faceted search method that unifies the ranking of products and the ranking of user requirements has not yet been proposed.

In this chapter, we propose a novel algorithm specifically geared towards ap- proximate faceted search within a product catalog of a Web shop. The main focus is on improving the retrieval of relevant products, taking into account the impor- tance of the specified user requirements. We adapt the p-norm extended Boolean model (Salton et al., 1983) to cope with the domain-specific characteristics of faceted product search. The adaptations consist of a facet similarity function, both for quan- titative and qualitative product properties. We also propose a term weighting method that takes in account user preferences. The algorithm is implemented as part of a Web application, with real-life data obtained from Tweakers.net Pricewatch (Tweak- ers.net, 2016), which is a Dutch Web shop price aggregation service. An extensive evaluation is performed by means of simulations and a battery of quality and perfor- mance metrics.

7.2

Related Work

In the literature, we can find some proposed approaches that aim to improve ex- ploratory search within a product catalog. Burke (2002) proposes a recommender system based on case-based reasoning. This system matches a user’s need with a product based on previously solved user needs that are similar to the current user’s needs. The authors augment this approach by combining it with example critiquing, which shows a product to the user and then allows the user express his/her dissat- isfaction with a particular facet of that product. This information can then be used to find another product that has a better value for this facet, possibly at the cost of having a worse value for other facets, thereby facilitating trade-off-based navigation. Similarly, in (Pu and Chen, 2005) an approach is proposed that uses example critiquing techniques. They propose an interaction paradigm in which users first declare their initial preferences, after which some matching example documents are displayed. The user subsequently examines these examples and either accepts one

document or performs critiquing on one of the examples. This process consists of improving some of the attribute values by allowing a compromise on one or more other attribute values. Afterwards, the system applies a decision strategy called the weighted additive sum rule, which multiplies each attribute value of a document with the corresponding importance weight in the query, followed by adding up these values to obtain an overall score. The documents with the highest scores are displayed to the user, after which the user can either continue to perform exploratory search or accept one of the presented documents. The authors show that users were able to improve their decision accuracy by using the trade-off support provided by this system.

In (Li et al., 2011), the authors examine an approach that employs concepts from the field of utility theory. They argue that buying a product is different from finding relevant products, therefore it is possible to calculate the expected utility of products and rank them based on the largest utility surplus. The utility surplus is defined as the difference between two components, namely the utility of the product and the utility of money. The utility of the product is defined as the sum of expected utilities for each of its characteristics, whereas the utility of money is defined as an increasing concave function. In order to obtain the expected utilities, they collect aggregated data about market shares and demand, from which they derive an estimated utility. They show that users tend to prefer their surplus-based ranking when searching for hotels in comparison to other considered rankings.

Although faceted search is regarded as a useful means to explore a catalog, care needs to be taken that the interface does not become too cluttered and ob- scured, which could result in a diminished user performance due to information overload (Sinha and Karger, 2005a). General concepts from the literature for im- plementing faceted navigation are explained and reviewed in Sacco and Tzitzikas (2009b). One of the ideas discussed for reducing the overload of information is to either filter or order the facets according to their importance.

Displaying all facets may be a solution when a small number of facets is involved, but it can overwhelm the user for larger sets of facets (Sinha and Karger, 2005b; Zheng et al., 2013). Studies such as Dash et al. (2008); Herlocker et al. (1999); Kim et al. (2014a); Koren et al. (2008a); Sacco and Tzitzikas (2009a); Vandic et al. (2013b) focus primarily on optimizing the decision of which facets to show in the user interface. These algorithms aim to reduce user effort by ordering the facets in such a way that the most important ones are listed first.

The study in Kim et al. (2014a) is based on earlier work of the authors which coined a basic idea for selecting more descriptive facets using an entropy-based mea-

sure (Zhu et al., 2013). The approach focuses on Semantic Web and Linked Data data sets, similar to the approaches in Arenas et al. (2014a,b); Cuenca Grau et al. (2016); Stolz and Hepp (2015). The reduction in user effort is achieved by calculat- ing gain ratios for each facet, which is based on conditional entropy. The conditional entropy indicates how much the value for a facet can be predicted by the value for another facet. By summing up all the gain ratios of the facets belonging to each property, the total gain ratio per property is computed. Last, the properties are ranked in decreasing order of total gain ratio and presented to the user. This process is repeated after each interaction between the user and the search engine.

Even though the previously discussed approaches are specifically geared towards e-commerce, they have various shortcomings. Approaches such as (Li et al., 2011), which rely on well-estimated user statistics, are not usable in practice. The reason for this is that some Web shops do not have the infrastructure to collect the user data or that the product catalogs are relatively large and changing rapidly. Other methods, such as (Burke, 2002; Pu and Chen, 2005), do not rely on previously collected user statistics, but they suffer from other issues. One of the main arguments against these example critiquing approaches is that they deviate too much from the currently used interaction paradigms (i.e., regular faceted navigation). We argue that an approach that lies closer to what users use on a daily basis will be more successful when deployed in practice. Therefore, our focus is to have an approach that introduces minimal changes to the classical faceted navigation interface present in current Web shops, while supporting approximate search and explicit user preferences. The only user interface adaption we make is that users can specify importance weights for the selected facets, which can be achieved in many different, non-intrusive ways.