Incremental language processing in a LSTM neural network

An LSTM network is a more sophisticated version of RNN which preserves the benefits of RNN as a model of incremental speech comprehension and additionally captures the long distance dependencies. In language modelling, LSTM is one of the most commonly adopted architectures for data mining and network training. Recently, Google announced a LSTM network trained on a 1 billion word benchmark which generates an accurate prediction of a following word based on the given context in a sentence (Jozefowicz et al., 2016). Note that the neural network model used in this thesis refers to this LSTM model. Here, I briefly walk through the architecture of LSTM (see also, Gers & Schmidhuber, 2000; Sundermeyer et al., 2015) and explain how it solves the vanishing gradient problem.

Instead of having a single operation in the recurrent hidden layer as in RNN, LSTM performs multiple operations, deciding which information to preserve and add inside the hidden layer. A useful analogy of this LSTM hidden layer is a memory cell with three gates in order to input, forget and output the contents of memory. First of all, it decides what to forget from the previous memory using the sigmoid function. Recall that the sigmoid function outputs a value between 0 and 1 which can be interpreted as a weight determining the strength of projection among the operators (a.k.a. gates in this analogy). Then, the vector of weights _∅(𝑡) reflects the state of the forget gate in the memory cell at a particular time _𝑡:

∅(𝑡) = 𝜎(𝑥(𝑡)𝑊𝑥∅+ 𝑠(𝑡 − 1)𝑊𝑠∅+ 𝑐(𝑡 − 1)𝑊𝑐∅+ 𝑏∅) … (13)

where _{𝜎 is a sigmoid function, 𝑥(𝑡) is a current input with associated weights 𝑊𝑥∅}, _{𝑠(𝑡 − 1)} is a previous state in the hidden layer with associated weights _𝑊𝑠∅ and _{𝑐(𝑡 − 1) is a previous} state in the memory cell with associated weights _𝑊𝑐∅. Note that the cell state term _{𝑐(𝑡 −}

74

1)𝑊𝑐∅ does not exist in the RNN architecture. Again, this vector of the forget gate state ∅(𝑡)

directly manipulates the memory content by setting 0 if it needs to be completely forgotten or setting 1 if it needs to be fully remembered.

Next, the LSTM network decides which information to add from the input and to store in the memory using sigmoid. With the same logic as above, the state of the input gate _{𝜃(𝑡) can be} expressed as:

𝜃(𝑡) = 𝜎(𝑥(𝑡)𝑊𝑥𝜃+ 𝑠(𝑡 − 1)𝑊𝑠𝜃+ 𝑐(𝑡 − 1)𝑊𝑐𝜃+ 𝑏𝜃) … (14)

Note that the weights to be trained in the input gate are different from those in the forget gate. From these weights that decide which memory contents to preserve from the previous cell state (or memory) _{∅(𝑡) and that decide which information to store from the current input 𝜃(𝑡),} we can construct new memory contents as below:

𝑐(𝑡) = 𝑐(𝑡 − 1) ⊛ ∅(𝑡) + tanh(𝑥(𝑡)𝑊𝑥𝑐+ 𝑠(𝑡 − 1)𝑊𝑠𝑐+ 𝑏𝑐) ⊛ 𝜃(𝑡) … (15)

where _{tanh is a hyperbolic tangent function described in 2.3.1 and ⊛ denotes an element-} wise product. Recall that tanh is a rescaled version of sigmoid in a scale between -1 and 1. Therefore, the input activation in the current hidden layer before passing through the memory cell is constructed through _{tanh which is, then, modified by the state of the input gate 𝜃(𝑡).} Also, note that the element-wise product _{⊛ allows a weight (a gate neuron in the input and} forget gates) to directly modify a particular feature (either from the previous memory content or from the current input) processed by the neuron via one-to-one mapping (since a number of neurons in each gate in the memory cell is same). In summary, (15) shows that the modified input representation at the input gate is combined with the modified memory representation in the forget gate to generate a new memory content.

Lastly, the network decides what it is going to output. Similar to the state of the other gates, the state of the output gate directly modulates the new memory content from (15) using sigmoid:

𝜔(𝑡) = 𝜎(𝑥(𝑡)𝑊𝑥𝜔+ 𝑠(𝑡 − 1)𝑊𝑠𝜔+ 𝑐(𝑡)𝑊𝑐𝜔+ 𝑏𝜔) … (16)

These weights are used to modify the current memory content that is going to be output: 𝑠(𝑡) = 𝜔(𝑡) ⊛ tanh(𝑐(𝑡)) … (17)

75

Similar to above, the unfiltered version of the memory content at the output gate is constructed through tanh which is, then, weighted by the state of the output gate through one-to-one mapping within every neuron in the output gate. Note that the bias term is not needed inside _{tanh of (17) because every distinct term that consists of new memory content} 𝑐(𝑡) is already adjusted; see (13), (14) and (15). The gate response 𝑠(𝑡) (equivalent to the hidden layer activation in RNN) is then projected to the output layer of the network as in RNN (see (9)):

𝑜(𝑡) = 𝜑(𝑠(𝑡)𝑊𝑔𝑜+ 𝑏2) … (18)

where _{𝜑 is the softmax function to generate a probabilistic response. Then, the BPTT}

algorithm can be applied for optimizing every weight matrix (12 in total) through the memory cell from (13) to (18); see Figure 2-6 for illustration.

Figure 2-6: A schematic illustration of LSTM architecture (see Equations (13) – (18))

To understand how this architecture effectively prevents the error gradient from vanishing as it passes through more layers, we need to see how the gradient back-propagates from _{𝑡 to} 𝑡 − 1 in the cell state. From (17), it is clear that the hidden layer activation in LSTM 𝑠(𝑡) is determined by the cell state _{𝑐(𝑡). Therefore, we just need to prove that the gradient does not}

76

necessarily diminish from 𝑐(𝑡) to 𝑐(𝑡 − 1). Using an arbitrary loss function 𝐻(𝑌, 𝑂) and a chain rule, the BPTT can simply be expressed as:

𝜕

𝜕𝑐(𝑡 − 1) 𝐻(𝑌, 𝑂) =

𝜕𝐻(𝑌, 𝑂)

𝜕𝑐(𝑡) ⊛ ∅(𝑡) … (19)

From (15), _{∅(𝑡) is a forget gate activation which controls for the rate at which the neural} network forgets its past memory. Hence, (19) simply follows from (15) defining how the new memory content at 𝑡 is constructed: note that there isn’t any non-linear activation function involved in generating this new content. In other words, the new memory content is generated from an identity function on the weighted combination of the previous cell state and the current input activation in the hidden layer. As a result, the error gradient does neither exponentially decrease (i.e. the derivative of an identity function is 1) nor explodes (i.e. the forget gate activation, which is basically a vector of sigmoid weights, is always less than 1) even if it passes through a number of previous cell states. The gradient is only linearly modulated by the forget gate activation ∅(𝑡). This is how LSTM architecture can preserve the long distance dependency information in its memory if it decides to.

2.4. Quantifying the “degree” in prediction: the information-theoretic