[PDF] Top 20 Exploring Methods and Resources for Discriminating Similar Languages

... closely-related languages (Ljubeˇsi´c et ...confusable languages, such as Malay/Indonesian (Ranaivo-Malancon, 2006), South-Eastern European languages (Tiedemann and Ljubeˇsi´c, 2012), as well as ... See full document

... small in our case. Still we believe that the proposed system is very simple. There are many possible improvements which can be done, for instance combining character and word-based n-grams, the use of dialectal lexicons ... See full document

... by the text length in tokens or characters respectively. We then train a single multi-class linear-kernel sup- port vector machine using LIBLINEAR (Fan et al., 2008) with the language identifiers (en-US, en-UK, hr, bs, ... See full document

... In our approach a maximum entropy classifier and SVM with different kernels were evaluated. The results show that maximum entropy delivers comparable results to SVM while it is consider- ably faster. To tackle the issue ... See full document

... The best run was ranked 8th out of 17 teams on the task A in closed training, i.e. without using external resources or past DSL data, with an accuracy of 0.879; the baseline of the first edition was 0.859 and the ... See full document

... of languages from text had been reported in studies over two decades ago (Beesley, 1988; Cavnar and Trenkle, 1994; Dunning, ...learning methods were in their ...the methods we have at hand today, the ... See full document

... s and comments on social-media platforms such as Twit- ter, Facebook and Microblog. These texts are obstacles for NLP tasks for the following reasons: (1) Each text is pretty short. Lengths of most sentences range from 3 ... See full document

... statistical methods achieve relatively high accuracy (more than 95%) for language de- ...various languages in one text, documents written in simi- lar languages – here are just some difficulties ... See full document

... Although, it still achieved an accuracy of 78.46% with very little tuning and training set, we are disap- pointed in the performance of the CSLTM. Based on the empirical study of (Zhang et al., 2015), character based ... See full document

... DSL 2017, participants were asked to develop sys- tems that could distinguish between 14 language varieties, distributed over 6 language groups. Two participation tracks were available: closed and open training. In ... See full document

... Statistical indicators such as character- and token- based language models have proven to be efficient on short text samples, especially character n-gram frequency profiles from length 1 to 5, whose inter- est is (inter ... See full document

... The main difference between the first and the sec- ond runs is that in the first run, the language iden- tifier was optimized so that it made as few positive errors with the unknown language xx as possible. Positive ... See full document

... the Discriminating between Similar Languages (DSL) shared task, which was organized as part of the VarDial’2016 workshop at ...very similar lan- guages and language varieties in newswire ... See full document

... the Discriminating between Similar Languages (DSL) shared task, held at the VarDial 2017 ...14 languages across 6 language groups using a corpus of excerpts of journalistic ...(all ... See full document

... classifies languages straight away. When exploring different combina- tions of word and character n-grams we trained the system on the 100,000 same instances and found that the highest results were achieved ... See full document

... the resources we created for and used in the 3rd edition of the Discriminating between Similar Languages (DSL) shared task, which was organized as part of the VarDial 2016 ...the ... See full document

... computational methods to identify the language a document is written ...when discriminating between very similar languages (Zampieri et ... See full document

... very similar languages in multilingual settings are the aforementioned Discriminating between Similar Languages (DSL) shared ...in similar languages and language ... See full document

... A similar system achieved the best score in the open submission of Tweet-LID shared task 2014 (Gamallo et ...achieve similar results in both in-domain and out-domain ... See full document

... nized similar languages and language varieties using the re- sults of the first two DSL shared ...of languages may need more ...difficult discriminating between similar languages ... See full document