[PDF] Top 20 Understanding and Improving Hidden Representations for Neural Machine Translation

... the hidden representations of NMT for better predictive performances regarding those relative tasks, in hope of achieving improved performance in terms of the target translation ...every ... See full document

... the translation process can be denoted as a deriva- tion that comprises a sequence of translation rules ...these translation rules are interpretable from a linguistic ... See full document

... The attention mechanism has been introduced as a way to learn an alignment between the source and target text, and improves encoder-decoder models significantly, while also providing a way to inter- pret the inner ... See full document

... Important words are more influential on trans- lation performance than the others. Under three different perturbations, perturbing words of top-most importance leads to lower BLEU scores than Random selected words. It ... See full document

... Both Joint-data learning and Multi-task learning improved overall translation performance. In the case of En→De, the performance of both ap- proaches is very similar. However, each has its own pros and cons. While ... See full document

... and hidden layer are respectively set as 620 and ...other hidden states, are set to ...the translation relative to a ref- erence is assessed using the BLEU metric (Pap- ineni et ... See full document

... pure neural machine translation architec- tures was first investigated by (Gülçehre et ...recurrent hidden state of the language model to the decoder state of the encoder-decoder network, with ... See full document

... for machine translation, Tu et ...the hidden layers of the translation de- ...speech translation if the speech translation is based on both the input speech and its transcrip- ... See full document

... We can make the following observations. First, using an BLSTM with max-pooling (Table 1 right) performs much better than an LSTM and us- ing the last hidden state as sentence representa- tion (Table 1 left). This ... See full document

... We use the global attentional model with gen- eral scoring function and input feeding by Lu- ong et al. (2015a). We provide only a very brief overview of this model here. It has an encoder, an attention, and a decoder. ... See full document

... model representations have been successful in a wide range of lan- guage understanding ...trained representations into sequence to se- quence models and apply it to neural ma- chine ... See full document

... We take the simple yet effective Seq2Seq model with attention mechanism proposed by Luong et al. (2015) as our baseline. Under the stan- dard encoder-decoder architecture, an encoder first maps the source-language input ... See full document

... We use a shared vocabulary BPE Model (Sen- nrich et al., 2016) for subword segmentation, with a code of 32000 merge operations. We use con- volutional (Gehring et al., 2017) encoder-decoder (15x15) architecture with the ... See full document

... and hidden state proposal matrices were initial- ized as random orthogonal matrices. Weights were optimized using the Adadelta algorithm and were updated with a mini-batch size of 32 sentences. The vocabulary ... See full document

... single hidden-to-output (h2o) layer which is used by the two separate ...decoder’s hidden state and the concatenation of the embeddings of the previous generated ... See full document

... for neural machine transla- tion (NMT) is starting to get more attention from the scientific ...and machine translation practitioners want to improve the performance of their systems on a ... See full document

... recurrent neural networks such as encoder-decoder models have gained increasing attention in machine translation owing their abil- ity to generate fluent ...German neural machine ... See full document

... generated translation and its corresponding dependency ...the translation of SMT is disfluent and ungram- matical, whereas RNNsearch is better than ...the translation of RNNsearch is locally fluent ... See full document

... and machine translation outputs as features, and combined them with baseline fea- tures to improve the system performance of ...and neural machine translation log-likelihood features ... See full document

... back translation (BT) (Sennrich et al., 2016a) with forward translation (FT) on noisy texts and find that pseudo-parallel data from forward trans- lation can help improve more ... See full document