[PDF] Top 20 Discriminating Similar Languages with Token Based Backoff

... The main difference between the first and the sec- ond runs is that in the first run, the language iden- tifier was optimized so that it made as few positive errors with the unknown language xx as possible. Positive ... See full document

... second Discriminating Similar Languages shared task (DSL 2015) aimed to discriminate between 15 similar languages and varieties, with an added “other” ...SVM, token-based ... See full document

... 3. The softmax score of the group classifier is concatenated again with the character and word repre- sentations’ concatenation to train a final language variety classifier based on softmax classifier. In the ... See full document

... approach based on SVM ensembles, which was also ranked first in the 2015 edition of the DSL task (Malmasi and Dras, 2015b), which confirms that SVM ensembles are a suitable method for this ... See full document

... European languages in news headlines and single unambiguous ...techniques based on embeddings to model semantics and evaluated using the HispaBlogs ... See full document

... between similar languages (DSL) on conversational texts is a challenging ...at discriminating between limited-resource languages on short conversational texts, like Uyghur and ...classifier ... See full document

... In sub-task 2 (Arabic dialects identification), we also submitted two runs for the closed track (run1 and run2) and two others for the open track (run3 and run4). In run1 and run3, we used a combination of ... See full document

... FO2, similar to AO3. Notably, our Group F submissions based on the supplied training data all performed substantially better on the dev partition of the shared task dataset than on the tst ...submissions ... See full document

... and token- based language models have proven to be efficient on short text samples, especially character n-gram frequency profiles from length 1 to 5, whose inter- est is (inter alia) to perform indirect ... See full document

... other languages such as Roma- nian (Fraser and Marcu, 2005) in order to de- crease word alignment error ...counts based on the different lev- els of ... See full document

... According to (Malmasi and Dras, 2015; C¸¨oltekin and Rama, 2016; Jauhiainen et al., 2016; Zirikly et al., 2016), high-order character n-grams and their combinations have proved to be highly discriminative for the DSL ... See full document

... We describe the systems entered by the National Research Council in the 2016 shared task on discriminating similar languages. Like previous years, we relied on character ngram features, and a ... See full document

... lar languages and ...than languages be- longing to different linguistic ...task: Discriminating be- tween Similar Languages (DSL) and German Di- alect Identification ...is based ... See full document

... guage groups. Before training, language codes are replaced with the respective group code (bs, hr, or sr becomes A, for example), sentences are tokenized, and each token gets an end mark ($). Tokens are defined as ... See full document

... Our team, Citius Ixa Imaxin, participated in all DSL sub-tasks with the following objective: to com- pare two very basic methods for language detection and observe how they behave when they are applied on the difficult ... See full document

... the Discriminating between Similar Languages shared task, I introduce an additional decision factor focusing on the token and subtoken ... See full document

... between similar languages” (DSL) shared ...predictions based on a two-stage process: we first predict the language group, then discriminate between languages or variants within the ... See full document

... very similar performance to character n-gram features when used in a probabilistic language model; Zampieri et ...features based purely on syntactic part-of-speech), when distinguishing different varieties ... See full document

... Two teams used information gain to estimate the best fea- tures for classification, UMich (King et al., 2014) and UniMelb-NLP (Lui et al., 2014b). These two teams were also the only ones teams which compiled and used ... See full document

... models based on continuous bag of word (CBOW) representations (Mikolov et ...are similar to feedforward NNs and simply take the mean vector of the input embeddings as input ... See full document