The BilBOWA model as introduced in this chapter utilises a sampled L2 bag of words
cross-lingual loss. From a scientific point of view, we showed that this loss efficiently ap- proximates the expensive full crosslingual loss, translating into a significant speedup during training. From an engineering point of view, we showed that the asynchronous implemen- tation can speed up training, and that parallel subsampling improves the accuracy of the learned features. For the former, we found that getting asynchronous training to work re- quired clipping the updates, especially as the dimensionality of the embeddings gets larger. This had a big impact on the model training speed, and no noticeable impact on the quality of the learned embeddings. Parallel subsampling speeds up training and at the same time makes it more accurate for the frequent words. It therefore turns out to be really important both in the monolingual and parallel setting. We have motivated the reason for the crosslin- gual setting as helping to uncover a better approximation of the global alignment statistics from the observed local, sentence-level co-occurrences.
offline methods like translation matrix [82] or multilingual CCA [83,92]. However, as we show on the translation task, BilBOWA can learn much finer-grained cross-lingual relation- ships than these methods. BilBOWA is also slightly slower than purely parallel methods like BiCVM and BAEs, but it has the advantage that it can train over much larger monolin- gual datasets. If the goal is to learn high-quality general purpose bilingual embeddings, it should always be beneficial to leverage more training data, and hence a hybrid model like BilBOWA might be a better choice than a parallel-only technique.
5.8
Conclusion
In this chapter we introduced BilBOWA, a computationally-efficient model for inducing bilingual distributed word representations directly from raw text without requiring word- alignments or dictionaries. Instead, the model directly utilises a limited amount of paral- lel data to learn bilingual word representations. BilBOWA combines advances in training monolingual word embeddings with a particularly efficient novel sampled cross-lingual ob- jective. The result is that the required computations per training step scales only with the number of words in the sentences, thereby enabling efficient large-scale cross-lingual train- ing. We evaluated BilBOWA on English-German cross-lingual document classification and achieved state-of-the-art results while reducing training time to only a few minutes, rather than several days as for most previous approaches. We also evaluated on an English-Spanish word-translation task where we improve upon the previous state of the art in relative word translation error rate.
Chapter 6
Barista: Simple, Task-specific
Bilingual Embeddings
This chapter introduces Barista (“Bilingual Adaptive Reshuffling with Individual Stochastic Alternatives”), a simple and computationally efficient training technique for learning task- specific bilingual word embeddings using monolingual word-embedding algorithms. The method incorporates additional semantic information in the form of how words map into equivalence classes during the embedding process.
In this chapter we make use of cross-lingual part-of-speech classes extracted from the crowd-sourced Wiktionary, translation classes obtained using the online Google Translate system, as well as semantic supersense classes extracted from the WordNet lexical resource. The model has the advantage that it is orthogonal to the choice of embedding algorithm, does not require parallel data, and can be adapted to specific tasks by redefining the equiva- lence classes. We analyze the learned embeddings and show that the model incorporates the information during training by trading off learning the standard syntactic linguistic regular- ities that word embedding models have become known for learning, with learning to cluster according to the semantic classes. We demonstrate the utility of the approach by applying these learned embedding features to the tasks of cross-lingual part-of-speech tagging over seven language pairs, as well as cross-lingual supersense tagging.
All code can be found athttps://github.com/gouwsmeister/barista. The work in this chapter was presented at the North American Chapter of the Association for Com- putational Linguistics (NAACL 2015).
6.1
Introduction
Standard neural language models are unsupervised models that train purely on raw text by learning word features that enable the model to predict the next word in a sequence of words. In the process, the model learns to cluster words into soft equivalence classes (words
that show similar statistical behaviour), in that words that cluster together in the embedding space are words that have a high conditional likelihood of appearing in the same contexts in the training data. While there is still some controversy whether such methods are superior to older methods, there is no doubt that continuous word representations can potentially solve some of the data sparsity problems inherent in NLP, due to its high-dimensional discrete prediction problems.
However, for some tasks we might already know the equivalence classes relevant to the prediction task at hand, and would like the models to make use of this additional informa- tion during training, to hopefully learn richer, task-specific representations. For example, it is easy to extract generalized parts of speech (POS) classes [93] from crowd-sourced an- notations on Wiktionary1[94]. If one knows which words in English maps to which POS classes, and likewise, which words in French maps to which POS classes, one might like to leverage this additional information during training to learn richer, POS-specific repre- sentations. Likewise, albeit at a much finer level of granularity, if one knows which words in two languages are translation pairs (translation equivalence classes), one might like to incorporate this side information to learn fine-grained bilingual representations.
It has furthermore been shown that weakly supervised embeddings algorithms can lead to large improvements for tasks like sentiment analysis [95]. This notion of weak supervi- sion guiding the learning of word representations is relevant to the work presented in this chapter, and the Barista model is designed to efficiently utilize additional semantic infor- mation about how words in its training vocabulary map into equivalence classes. The model uses a very simple way of reshuffling the training data to learn rich, semi-supervised, task- specific bilingual embeddings over large amounts of raw text, using only a limited amount of extra class information.