Amazon Mechanical Turk
Amazon Mechanical Turk Workers Can Provide Consistent and Economically Meaningful Data
... Amazon Mechanical Turk (also known as AMT or MTurk) has become an increasingly com- mon tool for researchers in social sciences (e.g., Clifford, 2014; Verkoeijen and Bouwmeester, 2014; Milita et al., ...
39
Clustering dictionary definitions using Amazon Mechanical Turk
... Vocabulary tutors need word sense disambig- uation (WSD) in order to provide exercises and assessments that match the sense of words being taught. Using expert annotators to build a WSD training set for all the words ...
9
Last Words: Amazon Mechanical Turk: Gold Mine or Coal Mine?
... on Amazon Mechanical Turk, in which non-expert evaluation of summarization systems is proved to be “not able to recover system rankings derived from ...
8
The average laboratory samples a population of 7,300 Amazon Mechanical Turk workers
... Using capture-recapture analysis we estimate the effective size of the active Amazon Mechanical Turk (MTurk) population that a typical laboratory can access to be about 7,300 workers. We also ...
16
How Gamification Affects Crowdsourcing: The Case of Amazon Mechanical Turk
... In recent years, following the spread and multiplication of online networks, new opportunities for ITO emerged. A particularly outstanding one—insofar it most likely will “increasingly cannibalize work currently ...
12
Amazon Mechanical Turk for Subjectivity Word Sense Disambiguation
... There has been recently an increasing interest in Amazon Mechanical Turk. Many researchers have utilized MTurk as a source of non-expert natural language annotation to create labeled datasets. In ...
9
An Innovative Crowdsourcing Approach for Amazon Mechanical Turk
... This paper is going to introduce an innovative approach for Amazon Mechanical Turk (AMT) crowdsourcing marketplace. In current AMT marketplace, workers especially new ones need to qualify themselves ...
6
The Language Demographics of Amazon Mechanical Turk
... Since Mechanical Turk provides financial incen- tives for participation, many workers attempt to complete tasks even if they do not have the lan- guage skills necessary to do ...
14
Using the Amazon Mechanical Turk to Transcribe and Annotate Meeting Speech for Extractive Summarization
... Amazon’s Mechanical Turk (MTurk) has been shown to produce good re- sults for some types of speech, its suitability for transcription and annotation of spontane- ous speech has not been ...
9
Exploring Normalization Techniques for Human Judgments of Machine Translation Adequacy Collected Using Amazon Mechanical Turk
... Adequacy judgments are collected from untrained Arabic-speaking annotators using Amazon’s Me- chanical Turk (MTurk) service. We create a human intelligence task (HIT) type that presents Turkers with a MT ...
5
Active Learning with Amazon Mechanical Turk
... Supervised classification is the predominant tech- nique for a large number of natural language pro- cessing (NLP) tasks. The large amount of labeled training data that supervised classification relies on is ...
11
Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk
... The workshop featured an open-ended shared task in which 35 teams were awarded $100 of credit on Amazon Mechanical Turk to spend on an annotation task of their choosing. Results of the shared task ...
14
Error Driven Paraphrase Annotation using Mechanical Turk
... an Amazon Mechanical Turk study that explores this idea, and establishes via an oracle evaluation that it holds the potential to substantially improve translation ...
5
Gender differences in altruism on mechanical turk: Expectations and actual behaviour
... In order to make data from different projects comparable, we first build a (quasi)continuous variable representing the proportion of the endowment that dictators give to recipients (0 = nothing, 1 = all). In the ...
15
Creating a Bi lingual Entailment Corpus through Translations with Mechanical Turk: $100 for a 10 day Rush
... As a complementary research problem, this paper addresses the data collection issue, focusing on the definition of a fast, cheap, and reliable methodology to create CLTE corpora. The main motivation is that, as in many ...
5
Examining the Use of Amazon’s Mechanical Turk for Edge Extraction of the Occlusal Surface of Fossilized Bovid Teeth
... In order to reconstruct environments associated with Plio-Pleistocene hominins in southern Africa, researchers frequently rely upon the animals associated with the hominins, in partic- ular, animals in the Family ...
19
Semi supervised Word Alignment with Mechanical Turk
... In this pilot experiment, we explore the possibility of using Amazon Mechanical Turk (MTurk) to collect bilin- gual word alignment data to assist automatic word align- ment. We develop a system ...
5
Collecting Semantic Data from Mechanical Turk for a Lexical Knowledge Resource in a Text to Picture Generating System
... Data collected from AMT tasks was manually filtered via removal of undesirable target item-response item pairs and classified via definition of the relations between the remaining target item-response item pairs. ...
5
Rating Computer Generated Questions with Mechanical Turk
... use Amazon Mechanical Turk to rate computer-generated reading comprehension questions about Wikipedia ...through Amazon Mechanical Turk, and show evidence that these ratings can ...
6
Emotions Evoked by Common Words and Phrases: Using Mechanical Turk to Create an Emotion Lexicon
... annotation through Amazon’s Mechanical Turk ser- vice. This dataset, which we will call EmoLex, is many times as large as the only other known emo- tion lexicon, WordNet Affect Lexicon. More impor- tantly, ...
9