After the question is determined, the algorithm processes the question in a number of iterations. As shown in Algorithm A.4, in each iteration, the algorithm determines the number of assignments for the question, gets the answers from the workers, and updates the graph accordingly. The algorithm updates the PRM after it completes all the iterations for the question.
In the first iteration, the algorithm creates ina assignments for the ques-tion. In the upcoming iterations, the number of assignments is determined based on the total number of assignments in the previous iterations for the question. Specifically, if the algorithm is in the(n+1)st iteration and the pre-vious iterations had k assignments in total, the algorithm creates k additional assignments in this iteration. This ensures that the new iteration can have a significant effect for this pair of PoIs, which is tested using a significance test.
3. Proposed Method
Algorithm A.4ProcessTheQuestion Algorithm Input: graph, prm, nextQ, ina, pt, minni, maxni, pvalue Output: graph, prm
1: ni←0
2: (i, j) ←the indexes of PoIs in nextQ
3: whiletrue do
4: Get edges from graph corresponding to nextQ as currentEdges
5: Determine the number of assignments in this iteration
6: Get answers for the assignments regarding nextQ
7: Update graph with the answers
8: if ni<minni then
9: ni←ni+1
10: continue
11: end if
12: Get edges from graph corresponding to nextQ as updatedEdges
13: edgePvalue←p-value corresponding to χ2value between currentEdges and updatedEdges
14: if edgePvalue≥pvalue then .stopping criteria is reached
15: prb1←NW/NA of the edge for the answer 1
16: prb−1←NW/NA of the edge for the answer−1
17: prb0←1−prb1−prb−1
18: edgeProbability←max(prb1, prb−1, prb0)
19: answer←the answer with the maximum probability
20: if edgeProbability>pt∧answer=0 then
21: Set prm[i, j]and prm[j, i]to 0
22: else if edgeProbability> pt then
23: Set prm[i, j]to 1 and prm[j, i]to−1 or vice versa with respect to the answer.
24: UpdateMatrix(prm)
25: else
26: Set prm[i, j]and prm[j, i]to 3
27: end if
28: break
29: else if ni=maxni then .maximum number of iterations are finished but the question still has no consensus.
30: Set prm[i, j]and prm[j, i]to 3
31: break
32: end if
33: end while
34: return graph, prm
Paper A.
When the answers from the workers are received, the graph is updated ac-cordingly. If the graph does not have any edge between the PoIs, two edges are added to the graph. Otherwise, the NA and NW weights of the corre-sponding edges are updated. It should be noted that if a worker’s answer is that the PoIs are incomparable, only the NA weight is incremented on the two edges; the NW weights are not changed.
When the graph is updated, the algorithm computes a p-value with respect to the previous iteration as shown in line 13. Here, the sets that we compare in the χ2test are the NW(pi, pj)and NW(pj, pi)values in the previous itera-tion and in the current iteraitera-tion. So in this test, the degree of freedom is 1.
Then, the algorithm checks whether it has reached the stopping conditions as shown in lines 14–32. There are two types of stopping conditions:
• The edge has consensus. The test for this case is shown in line 14. When this is the case, the algorithm computes the probability values of the possible answers and gets the answer with the maximum probability as shown in lines 15–19. It then updates the cells in the pairwise relevance matrix as shown in lines 20–24. If no answer has a probability value that exceeds the probability threshold pt, the algorithm sets the cells of the pairwise relevance matrix to 3 as shown in line 26.
• The maximum number of iterations is reached, but there is no consen-sus. The test for this case is shown in line 29. When this occurs, the corresponding cells of the pairwise relevance matrix are set to 3. . The algorithm for updating the PRM in line 24 is the same as the Com-puteGain algorithm (Algorithm A.3). However, instead of incrementing the gain in lines 10,12, and 14 of Algorithm A.3, the matrix is updated with the inferred value. Note that we do not change the value of a cell to an inferred value if it is already 1, −1, or 0 (line 8 in Algorithm A.3). In other words, we choose to value the direct opinion of the crowd over inferred information, even if this involves inconsistencies. To illustrate, if question(pi, pj)is asked to the workers, their consensus answer is pi ≺ pj, and the current PRM M contains both M[j, k] = 1 (encoding pj ≺ pk) and M[i, k] = 0, the updated PRM will have M[i, k] =0 instead of the inferred M[i, k] =1.
Example. Continuing from the example in Section 3.3, we describe the processing of the question(p3, p4).
Three iterations have to be finished before looking for consensus, since minni parameter is set to 3. We assign this question to 5, 5, and 10 workers in the first, second, and third iterations, respectively, since ina =5. At the end of the third iteration, we have a total of 20 assignments. Let us assume that there is consensus after the third iteration and that 15 workers stated that p3≺p4, 3 workers stated the opposite, and 2 workers stated that the PoIs are incomparable. The algorithm updates the graph with the answers from the workers, and the updated graph is shown in Figure A.5.
3. Proposed Method
p1 p2 p3
p4 p5
18,20
2,20 6,20
13,20 6,20 2,20
2,20 4,20 16,20
14,20
3,20 5,20
15,20
5,20
Fig. A.5:Updated Graph
To update the pairwise relevance matrix, we check the probability of the answers. Since the answer with the maximum probability is 1 and its prob-ability is 0.75, which exceeds the pt parameter (0.6), we set the answer to 1.
Then, the pairwise relevance matrix is updated with respect to this answer.
This answer leads to inferring p3≺p2and p3≺p5since the matrix contains the pairwise relevances p4≺ p2and p4≺ p5. The updated PRM is shown in Table A.2, with the changed values in bold.
p1 p2 p3 p4 p5
p1 0 1 2 0 1
p2 -1 0 -1 -1 2
p3 2 1 0 1 1
p4 0 1 -1 0 1
p5 -1 2 -1 -1 0
Table A.2:Updated PRM