This section provides the methodology and the results of a user study with the votes of 130 workers towards their preferred selection criteria of tasks on the platform. The qualitative and quantitative survey among crowd workers focuses on task selection and recommendation and takes into account the demographic characteristics of the workers. The provided results support the hypothesis that the preferences of work- ers are inhomogeneous and that criteria relevant for selection are not available for filtering the tasks in the current platforms.
4.2.1 Methodology
The study focuses on the question which characteristics of tasks are most relevant to workers with respect to the recommendation of tasks. The survey first introduces the idea of task recommendation to the workers. Then the workers are supposed to se- lect their most important recommendation criteria. After selecting recommendation criteria, the workers are required to arrange all recommendation criteria according to importance. The recommendation criteria include six standard measurements, like payment, as well as additional criteria that consider the similarity of tasks. Six fac- tual criteria are considered, namely the most money, least time, payment per time, time to rate, best-rated task and best-rated requester. Additionally, three criteria considering the similarity are provided: similar task, different task, similar worker. The time to rate describes the time a worker is required to wait until the work on the task is verified
and the money transferred. The similar/different task criteria describe a task which is similar or different to the last completed task of the worker. The similar worker recom- mendation criterion describes tasks that have been completed by workers which are found to be similar to the worker receiving the recommendation. The task was made available on the PCP Microworkers. The survey features a structured questionnaire
Description of the recommender idea Demographic & personal information Vote four out of nine criteria Rank four out of nine criteria Free text answers about criteria Demographic & personal information
Main interest: Recommendation criteria
Worker‘s characteristics Worker‘s characteristics
Figure 15: Survey design: five sections [146].
including six sections, shown in Figure15. Within the questionnaire, the workers are provided with a description of the idea of task recommendation and have to answer questions about characteristic personal information considered later. The main part has the workers vote for the four most important criteria from a set of nine and asks them to order the whole set according to importance afterward, also giving them the opportunity to express further opinions given a free text field. The last part asks for more or different personal characteristics of the worker, including consistency ques- tions [73] to filter out spammers.
The survey was executed on the PCP Microworkers in April and May 2015, leading to 130 submissions that are taken into consideration for data analysis. The analy- sis focuses on the characteristics: region, gender, experience, average payment, and activity, to find differences between the workers.
4.2.2 Results
A single survey submission is filtered by comparing the answers of voting (step three) and ranking (step four), applying the methods of the consistency questions. Only such votes that are consistent in voting and ranking are considered for the data analysis. The votes for each of the criteria are weighted by the rank it was voted on (1 − 4), where a higher rank corresponds to a higher weight. For each of the criteria, the weights are summed up and divided by the overall sum of weighted votes. This average weighted ranking (awr) gives a value between 0 and 1 for each of the criteria, where a higher awr hints towards higher valued recommendation criteria. The overall votes for the criteria and the corresponding awr is given in Table8.
Analyzing the results shows, that the workers unsurprisingly value the criteria most money and highest payment per time the most. However, the criterion similar tasks comes in third place being more valuable to workers than criteria like time to rate or best-rated task. This shows that there is a requirement for identifying similar tasks in order to be able to recommend them to workers on such platforms. Additionally, different task, is the least valued of the criteria, showing that workers want to be provided with similar tasks.
Table 8: Overall votes and awr.
criteria awr rank
1 2 3 4
most money 0.242 37 17 13 12
payment per time 0.169 19 20 9 12
similar task 0.147 18 12 11 14 least time 0.124 6 19 17 7 time to rate 0.120 5 16 21 8 best-rated 0.068 5 7 8 10 similar worker 0.051 4 7 4 5 best-rated requester 0.043 6 1 4 7 different task 0.035 2 3 6 5
Table 9: Number of submissions and counted votes per region.
region countries submissions votes
Asia BD, NP, PH 37 104
EU - West FR, DE, ES, IE, IT, NL, PT, SE 45 133
Western US, UK, CA, AU 48 140
While the analysis dividing the workers by gender, experience, payment or activity did not result in significant insights, analyzing the different regions shows significant differences between workers. The regions are chosen as defined by the platform Microworkers, where a task can be created to be published in a specific region only. The submissions were divided into the different regions Asia, EU-West and Western as seen in Table9. Figure16shows the differences between the regions for each of the criteria. It can be seen, that the results of the EU-West region almost mirror the overall results. However, the opinions of the workers from the Asia region and those from the Western region differ significantly, especially when it comes to the recommendation criteria of similar tasks. In the Asia region, this is the least valued recommendation criteria, while in the Western region this is the most valued recommendation criteria. Accordingly, different task is ranked sixth out of nine in Asia compared to the last rank in EU and Western regions.
4.2.3 Conclusion
This section provides the results of a survey undergone with workers on the PCP Microworkers. Analyzing the votes of 130 workers towards their preferred recommen- dation criteria, it showed that overall, workers value the criteria of time and money most. However, criteria that are not so strongly defined as the similarity of tasks
most mone y payment per time similar least time time torate best rated similar worker best
requester differ ent 0 0.1 0.2 0.3 A verage w eighted ranking Asia Western EU
Figure 16: Results with respect to workers’ region [146].
are also valued high. Especially the similar task criteria, which is found to be most valuable for workers from the Western region and appears to be least valuable for workers from the Asia region provides interesting recommendation criteria and has to be analyzed in a more detailed study.