[PDF] Top 20 Multi Granularity Self Attention for Neural Machine Translation

... state-of-the-art neural machine trans- lation (NMT) uses a deep multi-head self- attention network with no explicit phrase in- ...tical machine translation has shown that ... See full document

... of machine translation, neural network at- tracts lots of research attention recently that the encoder-decoder framework is widely ...this neural ma- chine translation (NMT) ... See full document

... Table 5(b) shows two translation samples in- volving frequent words. For the compound word 被占领土 (beizhanlingtu, occupied territory), the baseline NMT system only partly translates the word as “occupation” and ... See full document

... on neural machine translation (NMT) and sequence prediction (SP) tasks since these two tasks are commonly cho- sen for studying Transformers (Vaswani et ...encoder self-attention, ... See full document

... how attention uses the most relevant source words to generate a target word at each step of the ...the attention mechanism or are a desired behavior of the ...of attention models in neural ... See full document

... the attention mechanism for neural machine ...the attention mechanism to address the over-translation and under-translation inherent in ...for attention to capture ... See full document

... Our work is inspired by recent efforts on attaching an external memory to neural networks, such as neural turing machines (Graves et al., 2014), memory networks (Weston et al., 2014; Meng et al., 2015) and ... See full document

... the attention-based encoder-decoder network, as the attention mecha- nism, or originally called the alignment function in (Bahdanau et ...pair-specific attention mechanism by considering only a ... See full document

... A significant amount of research has been per- formed on the use of memory within neural net- works. For instance, an RNN features an implicit memory in the form of recurring hidden states. However, a vanilla RNN ... See full document

... Hierarchical Attention model per- forms significantly better than Miculicich et ...achical Attention models are the winner in major- ity cases and our best models beat Miculicich et ...quite ... See full document

... Rather, self- attention with relative position representations en- hances the generalization to sequence lengths un- seen during training (Shaw et ...of multi-agents: a team of alternative agents ... See full document

... several machine learning ...using machine learning ...a translation memory tool which worked as a sub- system in their transfer-based MT ...their machine learning model which classifies the ... See full document

... top self- attention layer to select relevant tokens yielded better results compared to the Localness Self- Attention (Yang et ...lower self-attention ... See full document

... the machine translation literature, there are two major streams for integrating visual information: approaches that (1) employ separate attention for different (text and vision) modalities, and (2) ... See full document

... (multi-hop attention or attention-over-heads) selects important heads in the early stage of training, which results in faster convergence than the original ... See full document

... Recurrent neural networks (RNNs) (Elman, 1990) can easily deal with variable-length input sentences and thus are a natural choice for the encoder and decoder of NMT ...a neural architecture where both the ... See full document

... state-of-the-art multi-head attention models based on the transformer ...the multi-head source-to-target attention compo- ...guided translation task, where the user wants to guide ... See full document

... the multi-hop attention mechanism to the Transformer and reported that the Transformer augmented with the multi-hop attention mechanism significantly outperformed the ...to neural ... See full document

... of neural machine translation ...the multi-head self- attention which is supposed to allow the model to independently attend to information from different representation ... See full document

... The model consists of a coupled encoder and de- coder, where the encoder is a modified stack- only recurrent neural network grammar (Kuncoro et al., 2017) and the decoder is a stack-only RNNG augmented with ... See full document