Binary Question Generation

We generate a sufficiently small number of binary questions in four steps.

First, we use a learn-to-rank method to construct a list l₁₂ of k most impor-tant spatial objects from the two top-k lists l1 and l2. Second, we compute the similarity between l12 and l1 (l2) to decide which objects in l1(l2) to ig-nore in binary questions. Third, we use spatial object entropy to control the number of binary questions. Fourth, we generate the binary questions for the crowdsourcing platform.

A List by Learn-to-Rank

The ListNet method [9] can solve the ranking problem on the lists of objects, and the features we extract are similar to those the ListNet method requires, so we adopt the ListNet method in this work. As shown in the upper part of Figure C.3, this step has two processes, training and ranking. The train-ing accepts a set of queries Que = ⁿQue⁽¹⁾, Que⁽²⁾, ..., Que^(m)o

. Each query Que⁽ⁱ⁾ results in two ranking lists generated by the two ranking functions.

Then, a list of objects o⁽ⁱ⁾ = (o⁽ⁱ⁾₁ , o⁽ⁱ⁾₂ , ..., o_n(i)(i)) is extracted from these two ranking lists, where o⁽ⁱ⁾_j is the j-th object in the ranking list, and n⁽ⁱ⁾ is the number of objects in o⁽ⁱ⁾. Furthermore, each list of spatial objects o⁽ⁱ⁾is asso-ciated with a list of scores g⁽ⁱ⁾ = (g⁽ⁱ⁾₁ , g⁽ⁱ⁾₂ , ..., g_n(i)(i)), where g⁽ⁱ⁾_j is the score of object o⁽ⁱ⁾_j with respect to a query Que⁽ⁱ⁾. The score g⁽ⁱ⁾_j represents how relevant an object o⁽ⁱ⁾_j is to a query Que⁽ⁱ⁾. Here, we manually select several queries from different categories as our training queries and then adopt the following equation to obtain g⁽ⁱ⁾_j [9].

g⁽ⁱ⁾_j =

3. Question Model for Crowdsourcing

g⁽ⁱ⁾ = (g⁽ⁱ⁾₁ , g₂⁽ⁱ⁾, ..., g_n(i)(i)) form a training instance (x⁽ⁱ⁾, g⁽ⁱ⁾)_{, i} = 1, 2, ..., m.

The training set is given asn

(x⁽ⁱ⁾, g⁽ⁱ⁾)^om

i=1.

The goal of training is to learn a ranking function f that generates a score f(x⁽ⁱ⁾_j ) for each feature vector x⁽ⁱ⁾_j . These can be organized into a list of scores z⁽ⁱ⁾ = (f(x⁽ⁱ⁾₁ ), f(x⁽ⁱ⁾₂ ), . . . , f(x_n(i)(i))) for the list of feature vec-tors x⁽ⁱ⁾. Here, in order to minimize the total loss on the training data, a loss function L based on the two lists of scores g⁽ⁱ⁾ and z⁽ⁱ⁾ is denoted as ∑i=1^m L(g⁽ⁱ⁾, z⁽ⁱ⁾). Specifically, we adopt the method in [9] to define L:

andG_kis the top-k subgroup containing all the permutations.

In ranking, given a new query Queⁱ⁰ and its corresponding list of spatial objects oⁱ⁰, a list of feature vectors x⁽ⁱ⁰⁾is produced by using the feature map-ping function Ψ. Using this list of feature vectors and the trained ranking function, the corresponding list of scores g⁽ⁱ⁰⁾is generated. Last, top-k spatial objects oⁱ⁰ are ranked in a list l12in ascending order of the scores.

We adopt the ListNet method [9] in the learn-to-rank step. In ListNet, a cross entropy metric and gradient descent are used as the loss function and learning algorithm, respectively. The gradient of L(_g⁽ⁱ⁾_{, z}⁽ⁱ⁾)with respect to parameter Que of f can be computed as follows: ∆Que = ^{∂L(g(i),z(i)( f )} simplic-ity, we also use a linear neural network model to calculate the score of a feature vector fQue(x⁽ⁱ⁾_j ) = hQue, x⁽ⁱ⁾_j i, whereh·,·idenotes an inner product.

Similarity Based Question Reduction

When deciding the binary questions to ask, the similarity between the input ranking lists l₁ = (a₁, . . . , a_k) and l2 = (b₁, . . . , b_k) can give us indications about what not to ask. For example, if we know ai = bj, it is unnecessary to ask “Is ai better than bj?” or the opposite. Let Nc = |l1∩l2|. The origi-nal 2k×2k matrixMcan be reduced to a(2k−Nc) × (2k−Nc)matrixM⁰. In particular, matrix M⁰ is obtained by removing any column or row cor-responding to a b_j that appears in l₁. By reducing matrix M to M⁰, the total number of possible binary questions is also reduced. Now N_bqbecomes

(2k−N_c)(2k−N_c−1)

2 .

If Nc= |l₁∩l2| ≥2, we can further reduce the number of binary questions that we should ask. This is done by using the Kendall Tau Coefficient. Refer to the original l1= (a1, . . . , a_k)and l2= (b1, . . . , b_k). If ai =bjand a_i⁰=b_j⁰for 1 ≤i<i⁰ ≤k and 1≤ j< j⁰ ≤k, aiand a_i⁰ are said to be concordant in the

Paper C.

two raking lists. Consequently, it is unnecessary to ask “Is aibetter than ai⁰?”

as the two ranking lists agree on this. Accordingly, for each concordant pair a_i and a_i⁰ we can reduce N_bq by 1. Assuming that there are N_τ concordant pairs, N_bqbecomes ^{(2k−Nc)(2k−N}₂ ^c−1)−N_τ.

Entropy Based Spatial Object Selection

For the remaining cells in matrixM⁰, we want to select and ask questions that involve the most important spatial objects. For that purpose, we consider the amount of information a spatial object conveys in its keywords.

Intuitively, if a spatial object has a keyword that is more representative, this spatial object is more likely to be searched for and is thus of higher importance. Also, the more frequent a keyword occurs in spatial objects, the more representative this keyword is. With these considerations, we use entropy to compare spatial objects. The entropy takes into account the fre-quency of keywords in a certain spatial object as well as the relative pro-portion of their total distribution across objects. A spatial object will have a higher entropy if more representative keywords are associated with it.

Consider a spatial object set O and an object o. Let f be a ranking func-tion, and let κ(f , o) be a subset of o.KW that is relevant to spatial objects obtained by f . We use κ(f , O) = ^S_o∈Oκ(f , o) to denote the union of such sets over all objects in O. We intend to determine the fraction of all key-words in κ(f , O)that come from object o’s contribution. This is captured by P_{κ( f,O)}(_o) = _{|κ( f ,O)|}^{|κ( f ,o)|}_.

Here, we give an example to classify this point. Given a query q with a keyword q.keywords = (Fast food) and a ranking function f , O={o1, o2, o3, o4} is the result of top-4 query according to f , where o1.keywords = (Food, Restaurant, Burger king), o2.keywords = (Food, Burger king), o3.keywords

= (Food, Restaurant) and o₄.keywords = (Restaurant). Thus, P_{κ( f,O)}(o₁) =

3+3+2

4 =2, P_{κ( f,O)}(o₂) = ³⁺²₄ = ⁵₄, P_{κ( f,O)}(o₃) = ³⁺³₄ = ³₂, and P_{κ( f,O)}(o₄) =³₄. Then, we can find that o₁ is more relevant to q and o₄ is less relevant to q.

Further, we can filter some spatial objects in the process of generating binary questions.

Accordingly, a spatial object o’s entropy is defined as Entropy(o) = −

∑

kw∈o.KW

P_{κ( f,O)}(o)logP_{κ( f,O)}(o)

The entropy of objects will be considered when we decide which object pairs should be included in binary questions that we publish to a crowdsourcing platform.

4. Crowdsourcing Model

Generating Binary Questions

So far, we obtained a matrixM⁰ with Nbq= ^{(2k−Nc)(2k−N}₂ ^c−1)−Nτ cells that correspond to possible binary questions. Now we describe how to generate Mq <N_bqbinary questions from these cells.

The questions are generated by Algorithm C.1. The algorithm first pro-cesses each pair of objects in ranking list l12 (lines 2–3) that results from the learn-to-rank step (Section 3.2), as list l12 is regarded as having the k most important objects from lists l1 and l2. Specifically, the algorithm checks if the entropy difference between a pair of objects is smaller than a threshold σ (line 4). If so, a question is generated for the pair, as their closeness calls for human intelligence to judge whether the ranking between them is appro-priate (lines 5–9). If the question budget has been exhausted, the algorithm returns (line 11).

Otherwise, the algorithm continues similarly on the common part of lists l1and l2(lines 14–26). If there is room for additional questions, the algorithm continues to work on all remaining objects in the union of l₁and l₂(lines 27–

39). The “if-then-else” statements in the algorithm are used because our setting of binary questions is biased rather than symmetric. Having both directions of comparisons in the binary questions is expected to offset the bias to some extent.

4 Crowdsourcing Model