Notes

Long-Short-Term Memory (LSTM) Networks Long-Short-Term Memory (LSTM) is a feature of a RNN that tackles the problems arising from long sequences / deep networks by a clever memory management. A common LSTM unit is composed of a cell, an input gate i (whether to write to cell), an output gate o (how much to reveal cell), a forget gate f (whether to erase cell) and a gate gate g (how much to write cell). The cell remembers values over arbitrary time intervals and the three gates regulate the flow of information into and out of the cell. A comparison between a normal RNN cell and a LSTM cell is given in the figure (a comparison between a normal RNN cell A and a LSTM cell B).

The gate vector can be written as:

where $W = \left( W_{i},W_{f},W_{o},W_{g} \right)$. The cell state is defined as the following:

And the hidden state is a function of the cell state:

The practicality of having this particular cell structure is evident if we look at multiple cells at once, i.e., the processing over multiple sequences, as it is shown in the following figure. Training works again with Back-Propagation-Through-Time. The gradient can now be passed without being interrupted, i.e., the problems of costly weight updates, vanishing and exploding gradients should not occur anymore.

In the figure: Illustration of LSTM over many sequences. Red arrow denotes the gradient, which can flow uninterruptedly.