Lecture 17: Attention, Transformers
From recurrent attention to self-attention, Transformer blocks, causal masking, and modern Transformer architectures.
Scope. These notes primarily follow Lecture 17 on attention and Transformers
1. From RNNs to attention
Recurrent neural networks process sequential data one step at a time while carrying information forward through a hidden state. In a traditional encoder-decoder RNN, the encoder must compress information from an entire input sequence into a fixed-length representation before the decoder produces the output.
This becomes difficult for long sequences. For example, translating a word may require information from several different parts of the original sentence, including words that appeared much earlier. If all of that information must pass through one fixed representation, useful long-range information can be lost.
Attention was introduced to reduce this bottleneck. Instead of forcing every output prediction to depend on the same compressed representation, the decoder can selectively use different encoder states depending on what it is currently predicting. This idea was introduced for neural machine translation by Bahdanau, Cho, and Bengio
Suppose the encoder produces hidden states
For output position
where the coefficients
The attention weights are obtained by comparing the current decoder state with each encoder state and normalizing the resulting scores with softmax. The context vector is therefore a weighted mixture of the encoder states rather than a single fixed summary.
This is soft attention because every input position can contribute to the context vector. In contrast, hard attention makes a discrete decision about which position to use. Hard attention can be easier to interpret, but the discrete selection makes ordinary backpropagation more difficult.
The same idea can also be applied outside language translation. In image captioning, for example, different words in a caption may depend on different spatial regions of the image. Attention allows the model to change which image features it emphasizes as the caption is generated.
2. From attention to self-attention
The original attention mechanism still contains recurrent components. This raises a natural question: if attention already allows information from different positions to interact, do we still need recurrence?
Self-attention removes the recurrence and applies attention within a single sequence. Every token can use information from the other tokens to construct a new, context-aware representation.
Let
be the embeddings of the input tokens. In a simple form of self-attention, token
The scores are normalized across the sequence,
and the new representation of token
The resulting vector
This mechanism is called self-attention because the queries and the information being attended to come from the same input sequence. Unlike an RNN, information does not have to travel through every intermediate time step before reaching a distant position.
The basic construction illustrates the idea, but it contains no learnable parameters. Transformers make self-attention learnable by introducing queries, keys, and values.
3. Learnable self-attention: queries, keys, and values
Each token embedding is projected into three learned representations:
These are the query, key, and value vectors. A useful interpretation from lecture is:
Queries select keys, and keys unlock values.
The query represents what the current token is looking for. A key represents the information that another token can be matched against, while the value contains the information that will actually contribute to the output.
The compatibility between token
After the scores are normalized with softmax, the output for token
Because
The dimensions of the query and key vectors must match because they are combined using a dot product. In the original Transformer,
for each attention head.
4. Scaled dot-product and multi-head attention
Stack the
The Transformer computes scaled dot-product attention as
The matrix
so each entry represents the compatibility between one query position and one key position.
The division by
This matrix formulation also explains one major advantage of Transformers: attention for all tokens can be computed with a small number of matrix multiplications, which are highly parallelizable on modern hardware.
The tradeoff is the size of the attention matrix. Because it contains all token-to-token comparisons, standard self-attention requires
computation and memory with respect to sequence length. Doubling the context length therefore produces roughly four times as many pairwise attention scores.
Multi-head attention
A single attention operation produces only one learned pattern of token interactions. Multi-head attention applies self-attention several times in parallel.
For head
where each head uses its own learned query, key, and value projections.
The head outputs are then concatenated:
Different heads can therefore learn different ways of relating positions in the sequence. The lecture compares this idea to multiple filters or channels in a convolutional network.
The original Transformer used eight attention heads. Since each head produced
so concatenating the heads returned the representation to the model dimension of
5. Building a Transformer
The Transformer replaces sequence-aligned recurrence with stacks of attention and feed-forward layers
A modern pre-normalization Transformer block can be summarized as
followed by
The feed-forward network has the general form
The components play different roles:
- Multi-head attention mixes information across token positions.
- The feed-forward network processes each token independently using the same network at every position.
- Residual connections create direct paths for information and gradients through many layers.
- Layer normalization helps stabilize optimization.
The original Transformer used ReLU in its feed-forward network and stacked six blocks in both the encoder and decoder. Later architectures use variants such as GELU or SwiGLU and may contain many more layers.
The original paper applied layer normalization after each residual addition. Many modern Transformers instead normalize before each sublayer, as in the equations above.
6. Causal masking for next-token prediction
An unrestricted self-attention layer allows each token to attend to every other token. That is inappropriate for autoregressive language modeling, where a model predicting token
A causal mask prevents this information leak.
Scaled attention with a mask becomes
where
After softmax,
so future positions receive zero attention weight.
Token
This creates a lower-triangular attention pattern.
Causal masking gives Transformer training an important advantage over recurrent models. During training, the complete sequence is available, so predictions for all
Generation is still sequential. After one new token is generated, that token must be added to the context before the next token can be predicted.
This causal decoder-style attention is the basic mechanism used in GPT-style language models.
7. Encoding position
Self-attention and the feed-forward layers do not inherently distinguish sequence order. Without additional positional information, reordering an unmasked input simply reorders the corresponding outputs.
The original Transformer solves this using sinusoidal positional encodings, which are added to token embeddings
The same word therefore receives a slightly different input representation depending on its position.
Other approaches are also common. Learned absolute position embeddings assign a trainable vector to each position and were used by BERT and early GPT models. Modern architectures often use relative approaches such as rotary positional embeddings (RoPE) or attention biases such as ALiBi
Causal models have an additional source of positional information because their mask itself changes with position: token
8. Three families of Transformers
Transformer architectures are commonly divided into three families according to which parts of the original encoder-decoder architecture they retain.
| Architecture | Attention pattern | Typical training goal | Examples |
|---|---|---|---|
| Encoder-only | Bidirectional | Represent or recover the input | BERT |
| Encoder-decoder | Bidirectional encoder + causal decoder | Map one sequence to another | Original Transformer, T5 |
| Decoder-only | Causal | Predict the next token | GPT, Llama |
Encoder-only
Encoder-only models use bidirectional self-attention, meaning each token can use context from both its left and right.
BERT is a prominent example. During pretraining, some input tokens are masked and the model learns to predict the missing tokens from the surrounding context
Encoder-decoder
The original Transformer uses both components. The encoder builds bidirectional representations of an input sequence. The decoder uses causal self-attention while also attending to the encoder output.
This structure naturally fits sequence-to-sequence problems such as translation and summarization
Decoder-only
Decoder-only Transformers use causal attention and are trained autoregressively:
GPT-style models use this architecture. Each token can access only itself and previous tokens, allowing the same model to repeatedly predict and generate the next token.
Decoder-only Transformers have become the dominant Transformer architecture for open-ended text generation.
9. Checks for understanding
-
Why was attention introduced in encoder-decoder RNNs?
A fixed-length encoder representation can become a bottleneck. Attention lets each output selectively use different encoder states instead. -
What makes self-attention “self”?
The queries, keys, and values are all derived from the same input sequence. -
What are the roles of queries, keys, and values?
Queries represent what a token is looking for, keys determine which tokens match the query, and values contain the information combined into the output. -
Why divide attention scores by
\sqrt{d_k} ?
The scaling prevents high-dimensional query-key dot products from becoming so large that softmax saturates. -
Why is self-attention highly parallelizable?
Attention scores for every token pair can be computed simultaneously using matrix multiplication rather than processing positions sequentially. -
What is the main sequence-length cost of standard self-attention?
The attention matrix isT\times T , giving standard attentionO(T^2) computation and memory. -
What does causal masking prevent?
It prevents a token from attending to future positions, so next-token predictions cannot see their answers. -
Why can Transformer training be parallel while autoregressive generation remains sequential?
During training, the full target sequence is already known and the mask hides future information. During generation, each next token does not exist until the previous token has been produced. -
Why is positional information needed?
Ordinary unmasked self-attention does not inherently encode sequence order, so position must be introduced through an encoding, learned embedding, relative scheme, or another mechanism. -
How do encoder-only and decoder-only Transformers differ?
Encoder-only models use bidirectional attention over the full input. Decoder-only models use causal attention and are trained to predict future tokens.