[PDF] Top 20 Crowdsourcing for Evaluating Machine Translation Quality

... on crowdsourcing evalu- ations. They include translation evaluations, crowdsource evaluations, and crowdsource ...now, translation evaluations are performed by ...each machine- translated ... See full document

... in machine translation technology has also given rise to large generic machine translation systems, many of which are freely available ...lation quality they provide, in part due to ... See full document

... Callison-Burch (2009) proposed several ways to evaluate MT output on MTurk. One such method was to collect reference translations to score MT output. It was only a pilot study (50 sentences in each of several languages), ... See full document

... relative quality of Pilot 2 and Pilot 0 only based on the number of cases when Pilot 2 was judged strictly better than Pilot 0 (row a), or only based on the number of cases when Pilot 2 was judged better or same ... See full document

... The advent of neural techniques in machine trans- lation (MT) (Kalchbrenner and Blunsom, 2013; Cho et al., 2014; Sutskever et al., 2014) has led to profound improvements in MT quality. For “easy” language ... See full document

... of evaluating MT quality, and targeted transla- tion allows us to isolate a model’s performance on specific linguistic phenomena; recent work using in-depth, qualitative manual evaluation (Isabelle et ... See full document

... end-to-end Machine-Translation as part of a longer pipeline with several modules, and evaluation of MT via speech- to-speech translation has been conducted in the frame of a yearly shared task ...the ... See full document

... NIST Machine Translation Evaluation exer- cise has run annually for the past five years as part of DARPA’s TIDES ...The quality of Chinese-to-English and Arabic-to-English transla- tion systems is ... See full document

... 4.231-3 Sample of evaluators • The choice of the number of evaluators depends on the following features : - subjectivity of the criterion to be evaluated : • outstandingly subjective cri[r] ... See full document

... In this paper, we have described ParaEval, an automatic evaluation framework for measuring machine translation results. A large collection of paraphrases, extracted through an unsupervised fashion using SMT ... See full document

... Statistical Machine Translation (SMT) ...to translation crowdsourcing where sentence translation is decomposed into the following smaller tasks: (a) translation of constituent ... See full document

... neural machine translation (NMT) models provide improved translation qual- ity in an elegant framework, it is less clear what they learn about ...started evaluating the qual- ity of vector ... See full document

... similation evaluation of MT. The methods include post-editing and comparison by bilingual experts (Ginestí-Rosell et al., 2009), and multiple choice tests (Jones et al., 2007; Trosterud and Unham- mer, 2012). These ... See full document

... In many languages, a large collection of SLPT problems is available. More than 130 standard- ized tests for 32 languages are listed in the English Wikipedia page for ‘List of language proficiency tests’ as of April 30, ... See full document

... of crowdsourcing for evaluating translation quality of MT systems based on the HTER (Human-mediated translation edit rate) ...automated translation quality metrics is ... See full document

... Our crowdsourcing is based on enterprise data from employee participation in translation tasks, and conducted inside a worldwide company ( Osamuyimen Stewart ...statistical machine translation ... See full document

... Close and constant monitoring of the workers’ annotation process was crucial for identifying malicious behaviour among workers, keeping track of the workflow and ensuring quality translations. The monitoring ... See full document

... similar translation problems to those posed by “it” and “you” and in order to keep the number of pronoun tokens manage- able when it comes to manual evaluation, certain exclusions must also be made in terms of ... See full document

... Jan-Thorsten Peter, Hermann Ney, Ondˇrej Bojar, Ngoc-Quan Pham, Jan Niehues, Alex Waibel, Franck Burlot, François Yvon, M¯arcis Pinnis, Valters Sics, Joost Bast- ings, Miguel Rios, Wilker Aziz, Philip Williams, Frédéric ... See full document

... Hybrid Approaches to Improvement of Translation Quality in Web based English Korean Machine Translation Hybrid Approaches to Improvement of Translation Quality in Web based English Korean Machine Tran[.] ... See full document