[PDF] Top 20 What does Attention in Neural Machine Translation Pay Attention to?

... Neural machine translation (NMT) has gained a lot of attention recently due to its substantial im- provements in machine translation quality achiev- ing state-of-the-art ... See full document

... While neural machine translation (NMT) provides high-quality translation, it is still hard to interpret and analyze its behav- ...and attention and (2) provides interface to adjust ... See full document

... multi-hop attention mechanism to the Transformer and reported that the Transformer augmented with the multi-hop attention mechanism significantly outperformed the ...to neural machine ... See full document

... Table 3 shows two examples of the translated sentences which include unknown words, and for each example, its reference translation (R.) and its translation result (T.) are shown. The replaced unknown words ... See full document

... including neural ma- chine translation, language understanding, and sequence ...the attention mechanism, which con- currently processes all inputs in the ...of attention via the lens of the ... See full document

... gual shared space through optimization of a mod- ified pairwise contrastive function, where the ex- tra multilingual signal in the shared space leads to improvements in image-sentence ranking and se- mantic textual ... See full document

... sequence translation. Due to the discrete nature of the hard-attention mechanism, we design a Reinforce- ment Learning (RL) algorithm with reward shap- ing strategy (Ng et ... See full document

... how attention based techniques are improving the quality of Neural Machine Translation (NMT) ...a translation process is done from end to ...statistical machine learning to ... See full document

... of neural machine translation ...self- attention which is supposed to allow the model to independently attend to information from different representation ...different attention heads ... See full document

... Architecture Their implementation is based on the attention-based model of Luong et al. (2015), which has some differences to that of Bahdanau et al. (2015), used in our work (§2.1). Their en- coder is a ... See full document

... multi-head attention models based on the transformer ...source-to-target attention compo- ...guided translation task, where the user wants to guide translation using pre-defined dictio- nary ... See full document

... • We test the theoretical claims that architec- tures with shorter paths through networks are better at capturing long-range dependen- cies. Our experimental results on modeling subject-verb agreement over long distances ... See full document

... word units are likely to represent the word mor- phemes. The words are segmented into sub-word units, which are to some extent close to the lin- guistic word stems and suffixes. For example, the word “adventurer” is ... See full document

... Deep Neural Network has been successfully applied to machine ...end machine translation system using neural networks by introducing the encoder-decoder ...statistical machine ... See full document

... The attention mechanism evaluates the dis- tribution of to-be-translated source words in a content-based addressing fashion (Graves et ...to attention faults and generate fluent but inaccurate or inadequate ... See full document

... to-English translation examples by ...high attention weight ...under translation in NMT as the decoding process ends once a target eos is gener- ... See full document

... the attention each word can ...constrained attention transformations and propose a new one, constrained sparsemax, shown to be differ- entiable and ... See full document

... the attention-based encoder-decoder ...tional neural networks (R-CNN) (Girshick et ...multimodal attention-based NMT in Section 3, followed by re-scoring of the translation can- didates in ... See full document

... considered, does not improve ...local attention mechanism, and (Costa-juss`a and Fonollosa, 2016; Chung et ...English-German translation examples. We can see the model with dependent attention ... See full document

... new attention mechanism for NMT based on convolutional neural networks (CNNs) to leverage the structures of source sen- tences in NMT without ...the attention mechanism of ANMT. In par- ticular, the ... See full document