Notes

RNNs in Machine Learning

In the left figure: each rectangle is a vector and arrows represent functions (e.g., matrix multiply). Input vectors are in red, output vectors are in blue and green vectors hold the RNN's state. From left to right: (1) Vanilla mode of processing without RNN, from fixed-sized input to fixed-sized output (e.g., image classification). (2) Sequence output (e.g., image captioning takes an image and outputs a sentence of words). (3) Sequence input (e.g., sentiment analysis where a given sentence is classified as expressing positive or negative sentiment). (4) Sequence input and sequence output (e.g., Machine Translation: an RNN reads a sentence in English and then outputs a sentence in French). (5) Synced sequence input and output (e.g., video classification where we wish to label each frame of the video). Notice that in every case are no pre specified constraints on the lengths sequences because the recurrent transformation (green) is fixed and can be applied as many times as we like.

Recurrent Neural Networks (RNNs) add an interesting twist to basic neural networks. A vanilla neural network takes in a fixed size vector as input which limits its usage in situations that involve a "series" type input with no predetermined size. Recurrent nets allow us to operate over sequences of vectors: Sequences in the input, the output, or in the most general case both (A sequence means, that the elements can have dependency on each other and that the order matters!). A few examples that may make this more concrete, are shown in the previous figure. The size of the input or output sequence is flexible, i.e., does not change the architecture of the model. Each network state gets an indices for the sequence. Since the sequence is often related with time progression, the index is chosen to be t. The main difference in architecture compared to conventional ANNs is, that recurrent loops are allowed, i.e., inputs from previous layer states of the network. Looking at a one-to-one neural network with one hidden layer, we can write the output state y(t) and the hidden layer state h(t) as follows:

where we include the bias in the W matrix. If we want to display the network over all sequences graphically, i.e., the computational graph, we can unroll it as displayed below.

This gives us another perspective: for any fixed sequence length s, the unrolled recurrent network corresponds to a feedforward network with s hidden layers. The two main differences to a feedforward network is, that the inputs are processed and outputs produced in sequence, and that the same parameters are used for all layers/all time steps, i.e., the same functions U, V, W applied over all times steps (Not to be confused with all epochs).

Back-Propagation Through Time (BPTT) The unfolding shown in the figure above is the first step of a particular network training algorithm, which is called Back-Propagation Through Time (BPTT). The second step is applying our known backpropagation algorithm to the unrolled network to calculate all weight updates.

There are several drawbacks to BPTT:

• Costly Parameter Update: Especially for long sequences, the parameter update for a shallow layer is the same as updating a parameter in an extremely deep feedforward network. One way to fix this is using a truncated BPTT algorithm. It processes the sequence one timestep at a time, and every k1 timesteps, it runs BPTT for k2 timesteps, so a parameter update can be cheap if k2 is small. Consequently, its hidden states have been exposed to many timesteps and so may contain useful information about the far past, which would be opportunistically exploited.
• Exploding Gradients: The gradients coming from the deeper layers have to go through continuous matrix multiplications because of the chain rule, and as they approach the earlier layers. If they have large value (>1) they get larger and eventually blow up and crash the model (NaN-values!). This can be solved by gradient clipping: which places a predefined threshold on the gradients to prevent it from getting too large. Note that this only changes the length, and not the direction of the gradients.
• Vanishing Gradients: A similar problem arises if the gradients have small values (<1). They will shrink exponentially until they vanish and make it impossible for the model to learn. This issue cannot be solved as simple; hence it requires to use shorter sequences or to make fundamental change in the RNN architecture.