Lecture 17: Attention, Transformers

From recurrent attention to self-attention, Transformer blocks, causal masking, and modern Transformer architectures.

Scope. These notes primarily follow Lecture 17 on attention and Transformers . The assigned readings are used selectively to clarify the mechanisms introduced in lecture rather than to reproduce every detail of the original papers.

1. From RNNs to attention

Recurrent neural networks process sequential data one step at a time while carrying information forward through a hidden state. In a traditional encoder-decoder RNN, the encoder must compress information from an entire input sequence into a fixed-length representation before the decoder produces the output.

This becomes difficult for long sequences. For example, translating a word may require information from several different parts of the original sentence, including words that appeared much earlier. If all of that information must pass through one fixed representation, useful long-range information can be lost.

Attention was introduced to reduce this bottleneck. Instead of forcing every output prediction to depend on the same compressed representation, the decoder can selectively use different encoder states depending on what it is currently predicting. This idea was introduced for neural machine translation by Bahdanau, Cho, and Bengio .

Suppose the encoder produces hidden states

h_1,h_2,\ldots,h_T.

For output position i, attention forms a context vector

c_i=\sum_{j=1}^{T}\alpha_{ij}h_j,

where the coefficients \alpha_{ij} describe how important encoder state h_j is for the current output.

The attention weights are obtained by comparing the current decoder state with each encoder state and normalizing the resulting scores with softmax. The context vector is therefore a weighted mixture of the encoder states rather than a single fixed summary.

This is soft attention because every input position can contribute to the context vector. In contrast, hard attention makes a discrete decision about which position to use. Hard attention can be easier to interpret, but the discrete selection makes ordinary backpropagation more difficult.

The same idea can also be applied outside language translation. In image captioning, for example, different words in a caption may depend on different spatial regions of the image. Attention allows the model to change which image features it emphasizes as the caption is generated.

2. From attention to self-attention

The original attention mechanism still contains recurrent components. This raises a natural question: if attention already allows information from different positions to interact, do we still need recurrence?

Self-attention removes the recurrence and applies attention within a single sequence. Every token can use information from the other tokens to construct a new, context-aware representation.

Let

x_1,x_2,\ldots,x_T

be the embeddings of the input tokens. In a simple form of self-attention, token i is compared with every token j using a dot product,

s_{ij}=x_i^\top x_j.

The scores are normalized across the sequence,

\alpha_{ij} = \frac{\exp(s_{ij})} {\sum_{m=1}^{T}\exp(s_{im})},

and the new representation of token i is

a_i=\sum_{j=1}^{T}\alpha_{ij}x_j.

The resulting vector a_i contains information from the entire sequence, with more relevant tokens contributing more strongly.

This mechanism is called self-attention because the queries and the information being attended to come from the same input sequence. Unlike an RNN, information does not have to travel through every intermediate time step before reaching a distant position.

The basic construction illustrates the idea, but it contains no learnable parameters. Transformers make self-attention learnable by introducing queries, keys, and values.

3. Learnable self-attention: queries, keys, and values

Each token embedding is projected into three learned representations:

q_i=W^q x_i,\qquad k_i=W^k x_i,\qquad v_i=W^v x_i.

These are the query, key, and value vectors. A useful interpretation from lecture is:

Queries select keys, and keys unlock values.

The query represents what the current token is looking for. A key represents the information that another token can be matched against, while the value contains the information that will actually contribute to the output.

The compatibility between token i and token j is measured using the query-key dot product,

q_i^\top k_j.

After the scores are normalized with softmax, the output for token i becomes a weighted sum of the value vectors:

a_i=\sum_{j=1}^{T}\alpha_{ij}v_j.

Because W^q, W^k, and W^v are trainable, the model learns which types of token relationships are useful.

Learnable self-attention showing query, key, and value projections and the weighted combination of value vectors.
Figure 1. Learnable self-attention. Each input is projected into query, key, and value representations. Query-key compatibility scores determine how strongly the corresponding value vectors contribute to the context-aware output. Source: Lecture 17 slides .

The dimensions of the query and key vectors must match because they are combined using a dot product. In the original Transformer,

d_{\text{model}}=512, \qquad d_q=d_k=64, \qquad d_v=64

for each attention head.

4. Scaled dot-product and multi-head attention

Stack the T input embeddings as rows of a matrix X. The queries, keys, and values can then be computed for every token simultaneously:

Q=XW^q,\qquad K=XW^k,\qquad V=XW^v.

The Transformer computes scaled dot-product attention as

\operatorname{Attention}(Q,K,V) = \operatorname{softmax} \left( \frac{QK^\top}{\sqrt{d_k}} \right)V.

The matrix QK^\top has shape

T\times T,

so each entry represents the compatibility between one query position and one key position.

The division by \sqrt{d_k} is important. As the dimension of the query and key vectors increases, unscaled dot products can become large in magnitude. Large values can push softmax into saturated regions with very small gradients. Scaling keeps the scores at a more useful magnitude .

This matrix formulation also explains one major advantage of Transformers: attention for all tokens can be computed with a small number of matrix multiplications, which are highly parallelizable on modern hardware.

The tradeoff is the size of the attention matrix. Because it contains all token-to-token comparisons, standard self-attention requires

O(T^2)

computation and memory with respect to sequence length. Doubling the context length therefore produces roughly four times as many pairwise attention scores.

Multi-head attention

A single attention operation produces only one learned pattern of token interactions. Multi-head attention applies self-attention several times in parallel.

For head h,

\operatorname{head}_h = \operatorname{Attention} \left( Q_h,K_h,V_h \right),

where each head uses its own learned query, key, and value projections.

The head outputs are then concatenated:

\operatorname{MultiHead}(Q,K,V) = \operatorname{Concat} \left( \operatorname{head}_1,\ldots,\operatorname{head}_H \right)W^O.

Different heads can therefore learn different ways of relating positions in the sequence. The lecture compares this idea to multiple filters or channels in a convolutional network.

The original Transformer used eight attention heads. Since each head produced 64-dimensional values,

8\times64=512,

so concatenating the heads returned the representation to the model dimension of 512 .

5. Building a Transformer

The Transformer replaces sequence-aligned recurrence with stacks of attention and feed-forward layers .

A modern pre-normalization Transformer block can be summarized as

h=x+\operatorname{MultiHead}(\operatorname{LayerNorm}(x)),

followed by

\operatorname{out} = h+\operatorname{FFN}(\operatorname{LayerNorm}(h)).

The feed-forward network has the general form

\operatorname{FFN}(h) = W_2\phi(W_1h+b_1)+b_2.

The components play different roles:

Transformer block containing layer normalization, multi-head attention, a feed-forward network, and residual connections.
Figure 2. Inside a Transformer block. Attention exchanges information across token positions, while the feed-forward network transforms each position independently. Residual connections provide direct paths around both sublayers. Source: Lecture 17 slides .

The original Transformer used ReLU in its feed-forward network and stacked six blocks in both the encoder and decoder. Later architectures use variants such as GELU or SwiGLU and may contain many more layers.

The original paper applied layer normalization after each residual addition. Many modern Transformers instead normalize before each sublayer, as in the equations above.

6. Causal masking for next-token prediction

An unrestricted self-attention layer allows each token to attend to every other token. That is inappropriate for autoregressive language modeling, where a model predicting token i+1 must not see tokens that occur later in the sequence.

A causal mask prevents this information leak.

Scaled attention with a mask becomes

A = \operatorname{softmax} \left( \frac{QK^\top}{\sqrt{d_k}}+M \right)V,

where

M_{ij} = \begin{cases} 0, & j\leq i,\\ -\infty, & j>i. \end{cases}

After softmax,

\exp(-\infty)=0,

so future positions receive zero attention weight.

Token i can therefore attend only to

1,\ldots,i.

This creates a lower-triangular attention pattern.

Lower-triangular causal attention mask where each token may attend only to itself and earlier positions.
Figure 3. Causal self-attention mask. Blue entries are available to attention, while future positions are assigned a score of -\infty before softmax and therefore receive zero weight. Source: Lecture 17 slides .

Causal masking gives Transformer training an important advantage over recurrent models. During training, the complete sequence is available, so predictions for all T positions can be computed in one parallel forward pass while the mask prevents any position from seeing its future answer.

Generation is still sequential. After one new token is generated, that token must be added to the context before the next token can be predicted.

This causal decoder-style attention is the basic mechanism used in GPT-style language models.

7. Encoding position

Self-attention and the feed-forward layers do not inherently distinguish sequence order. Without additional positional information, reordering an unmasked input simply reorders the corresponding outputs.

The original Transformer solves this using sinusoidal positional encodings, which are added to token embeddings :

PE_{(pos,2i)} = \sin \left( \frac{pos}{10000^{2i/d_{\text{model}}}} \right), PE_{(pos,2i+1)} = \cos \left( \frac{pos}{10000^{2i/d_{\text{model}}}} \right).

The same word therefore receives a slightly different input representation depending on its position.

Other approaches are also common. Learned absolute position embeddings assign a trainable vector to each position and were used by BERT and early GPT models. Modern architectures often use relative approaches such as rotary positional embeddings (RoPE) or attention biases such as ALiBi .

Causal models have an additional source of positional information because their mask itself changes with position: token i can attend to exactly i positions. However, without a mask or some other positional mechanism, an encoder remains permutation equivariant.

8. Three families of Transformers

Transformer architectures are commonly divided into three families according to which parts of the original encoder-decoder architecture they retain.

Architecture Attention pattern Typical training goal Examples
Encoder-only Bidirectional Represent or recover the input BERT
Encoder-decoder Bidirectional encoder + causal decoder Map one sequence to another Original Transformer, T5
Decoder-only Causal Predict the next token GPT, Llama

Encoder-only

Encoder-only models use bidirectional self-attention, meaning each token can use context from both its left and right.

BERT is a prominent example. During pretraining, some input tokens are masked and the model learns to predict the missing tokens from the surrounding context . This makes encoder-only models useful for tasks such as classification, search, and learned text representations.

Encoder-decoder

The original Transformer uses both components. The encoder builds bidirectional representations of an input sequence. The decoder uses causal self-attention while also attending to the encoder output.

This structure naturally fits sequence-to-sequence problems such as translation and summarization .

Decoder-only

Decoder-only Transformers use causal attention and are trained autoregressively:

p(x_{t+1}\mid x_1,\ldots,x_t).

GPT-style models use this architecture. Each token can access only itself and previous tokens, allowing the same model to repeatedly predict and generate the next token.

Decoder-only Transformers have become the dominant Transformer architecture for open-ended text generation.

9. Checks for understanding

  1. Why was attention introduced in encoder-decoder RNNs?
    A fixed-length encoder representation can become a bottleneck. Attention lets each output selectively use different encoder states instead.

  2. What makes self-attention “self”?
    The queries, keys, and values are all derived from the same input sequence.

  3. What are the roles of queries, keys, and values?
    Queries represent what a token is looking for, keys determine which tokens match the query, and values contain the information combined into the output.

  4. Why divide attention scores by \sqrt{d_k}?
    The scaling prevents high-dimensional query-key dot products from becoming so large that softmax saturates.

  5. Why is self-attention highly parallelizable?
    Attention scores for every token pair can be computed simultaneously using matrix multiplication rather than processing positions sequentially.

  6. What is the main sequence-length cost of standard self-attention?
    The attention matrix is T\times T, giving standard attention O(T^2) computation and memory.

  7. What does causal masking prevent?
    It prevents a token from attending to future positions, so next-token predictions cannot see their answers.

  8. Why can Transformer training be parallel while autoregressive generation remains sequential?
    During training, the full target sequence is already known and the mask hides future information. During generation, each next token does not exist until the previous token has been produced.

  9. Why is positional information needed?
    Ordinary unmasked self-attention does not inherently encode sequence order, so position must be introduced through an encoding, learned embedding, relative scheme, or another mechanism.

  10. How do encoder-only and decoder-only Transformers differ?
    Encoder-only models use bidirectional attention over the full input. Decoder-only models use causal attention and are trained to predict future tokens.