Notes

Attention in Deep Learning In the field of Machine Learning, attention is a mechanism used by certain types of neural networks to selectively focus on certain parts of an input when processing it. This is useful because it allows the model to automatically learn to focus on the most relevant parts of the input, which can improve its performance on a given task. For example, an attention mechanism might be used in a machine translation model to automatically focus on the words in the source sentence that are most relevant for generating the correct translation.

We'll focus here on time-series attention models.

This is a sequence-to-sequence neural network. You want your architecture to produce an output for every timestamp. And usually you do these through RNNs (LSTMs), which features hidden states and recalls previous states of the network, thus requires a sequential step-by-step calculation of the states. On the other hand we have CNNs which are great in parallelization, but the downside is their finite and fixed memory (loose flexibility). Can we combine these two architectures? That is, can we have the long dependencies and flexibility of RNNs together with the parallelization properties of CNNs? Here is where Self-Attention (SA) Models come in.

The outputs are calculate according to the formula in the right figure, however the key peculiarity is that W is not a parameter of the model that is learnt through gradient descent. It is a parameter that indicates how similar the inputs are similar between them and are calculated as shown in the following figure. These weights are passed through a Softmax such that the sum of all weights add up to 1.

• In simple self-attention $W_{ii}$, i.e., $X_{i}\ to\ Y_{i}$ usually has the most weight. Which is not a big problem, but we will allow this to change later.
• Simple self-attention has no parameters. Whatever parametrized mechanism generates $x_{i}$ (like an embedding layer) drives the self-attention.
• There is a linear operation between X and Y. Non-vanishing gradients through $Y = WX^{T}$, vanishing gradients through W = softmax($X^{T}X)$.

Intuition how Self-Attention works: The Dot Product Let's say we want to try a model to understand the sentiment of some restaurant reviews (like in the example below). The word "terrible" is something that probably we wouldn't like to have in a restaurant review, however if it comes after "not too" is probably not such a bad review. Hence, our model should pose attention also to these part of the sentence to understand the overall sentiment of the phrase, i.e., the weight between "not" and "terrible" should be a major one in our self-attention matrix. Which implies that the model should be capable of understanding the dependencies between the input "not" and "terrible" and how it modulates the meaning.

Improving Self-Attention

• Scaled Dot Product: If we have longer and longer input vectors than $W_{ij}$ gets larger and larger. In order to avoid that we scale it through a normalization factor K.
• Key, Value and Query transformations. These is the main breakthrough towards transformers architectures: the Key value is not anymore just the input value, but rather a linear transformation of the input, i.e. the input value times K plus a bias term b. In this step we are actually allowing the network to learn these attentional weights because the parameters (K, Q, V, $b_{k},b_{q},b_{v}$) can be trained through gradient descent.

• Multi-Head Attention: The input vector is projected into some low-dimensional version, you concatenate them and you obtain multiple sets of attention weights that can be learnt. This allows the network to have a self-attention that can understand the relation between words in the sentence.

Recap of Self-Attention Self-Attention: Sequence-To-Sequence layer with Parallel Computation and Perfect Long-Term Memory. Fundamentally a set-to-set layer, no access to the sequential structure of the input. A large part of the behavior comes from the parameters upstream.

Transformers Any sequence-based model that primarily uses self-attention to propagate information along the time dimension. More broadly: any model that primarily uses self-attention to propagate information between the basic units of our instances.

• Pixels Image Transformer.
• Graph Nodes Graph Transformer.

The transformer architecture is based on blocks as the one showed below that are stucked on each other as shown in the right figure.

The Auto-Regressive Transformer One problem that we still have is making sure that the network cannot access information regarding upcoming targets from the previous layers. In order to do this we calculate the Causal Self-Attention weight matrix and all future values are set to minus infinity.

Causal Self-Attention By using Causal Self-Attention we force the network to figure out which words are important in the previous sequence to generate the following letter or word. The blocks that feature such characteristic are named Causal Transformer Blocks.

The Causal Autoregressive Transformer

The Transformer: Position Embeddings The last problem we have to address is "repetition of words" or again different meanings associated with the same word (e.g., "The"). In order to overcome this limitation, Position Embeddings are used, so that same words are considered differently by the network.

The "Original Transformer" (ELMo) Adding all these features together, we obtain the very first transformer ELMo from the paper "Attention Is All You Need" that was exploited for language translation. It was a machine translation model with no recurrent layers or convolutions, but rather an encoder/decoder configuration with positional encoding. It featured 512 dims, 8 heads, 2x6 blocks. It has been trained for 3.5 days on 8 GPUs.

The GPT3 Transformer Autoregressive language model. Single stack of causal trf blocks with positional embeddings. 12288 dims, 96 heads, 96 blocks, sequence size 2048 and 175 billion total parameters. It has been trained on 10000 GPUs, likely in around 12 days for about $4,6 million.

Training Transformer Models