Lecture 16: Sequence Learning with RNNs

Turning words into vectors, recurrent neural networks and their variants, backpropagation through time and why its gradients vanish or explode, LSTMs and GRUs, and why recurrence has come back in modern sequence models.

Scope. These notes follow the November 3 lecture slides . Several diagrams on the slides come from Raschka and Mirjalili ; the figures here are redrawn or computed for these notes. Material that goes beyond the slides, mostly from the assigned readings , is marked as a supplement. Every numerical example was checked in Python.

Notation. We write h_t for the slides’ h^{\langle t\rangle}, and likewise for x_t, y_t and C_t. The input dimension is d, the hidden dimension is n, the sequence length is T, and \odot is elementwise multiplication.

1. Two Challenges with Text

In the diffusion lecture, a text prompt conditioned the image generator. That raised a question the lecture left open: how does a sentence become numbers that a network can use? Text poses two separate problems :

  1. Turning words into numbers. We need a representation of words through which activations can flow forward and gradients can flow backward.
  2. Modeling sequences. Sentences have different lengths, and a word can depend on another word far away from it. A model should accept variable-length input and carry context across long ranges.

The rest of the module maps onto these problems. This lecture handles variable-length input with RNNs. Attention and transformers (Lecture 17) address long-range dependencies. GPT architectures (Lecture 18) give a generative model of text, and the remaining lectures cover scale, alignment, reasoning and parameter-efficient fine-tuning.

Where the module is going. GPT is an autoregressive probabilistic model . It factorizes the probability of a sequence into next-token predictions, so that each token depends on all the tokens before it:

P_\theta(X) = \prod_{i}\prod_{t} P_\theta\left(X_{i,t} \mid X_{i,\lt t}\right), \qquad \max_\theta \sum_i \sum_t \log P_\theta\left(X_{i,t} \mid X_{i,\lt t}\right).

Here i indexes training sequences and t indexes positions. For example, P(\text{the sun is shining}) = P(\text{the})\,P(\text{sun}\mid\text{the})\,P(\text{is}\mid\text{the sun})\,P(\text{shining}\mid\text{the sun is}). Training maximizes the log-likelihood of observed text. The conditional P_\theta(X_{i,t}\mid X_{i,\lt t}) takes a prefix of any length as input, which is exactly the variable-length problem. The RNN is the first architecture in the course that can represent such a conditional with a fixed number of parameters.

2. Three Classic Ways to Avoid the Sequence Problem

Before RNNs, the common approaches did not solve the sequence problem; each made a simplifying assumption that removed it.

2.1 Bag-of-Words: Discard the Order

Build a vocabulary, then represent each document by how many times each word occurs. With the vocabulary and=0, is=1, one=2, shining=3, sun=4, sweet=5, the=6, two=7, weather=8, the three training sentences on the slide become rows of a design matrix:

Sentence Count vector
“The sun is shining” [0, 1, 0, 1, 1, 0, 1, 0, 0]
“The weather is sweet” [0, 1, 0, 0, 0, 1, 1, 0, 1]
“The sun is shining, the weather is sweet, and one and one is two” [2, 3, 2, 1, 1, 1, 2, 1, 1]

Any classifier, such as logistic regression or an MLP, can be trained on this 3\times 9 matrix. The cost of discarding order is that “The movie my friend has not seen is good” and “The movie my friend has seen is not good” receive the same vector, although they say opposite things.

2.2 Hidden Markov Models: Remember Only the Last State

A hidden Markov model assumes the hidden state at each step depends only on the previous state, and each observation depends only on the current state. The slide’s example has hidden weather (rainy or sunny) that follows a Markov chain, and observed activities (walk, shop, clean) that depend on the weather of that day. The Markov assumption removes the need to look further back. Information from many steps ago survives only if it is encoded in the current discrete state, which requires an exponentially large state space for rich histories.

2.3 Convolutional Networks: Assume a Maximum Length

A one-dimensional CNN slides filters over the sequence. It needs inputs of a fixed size, so every sentence is padded to a maximum length. This assumes a reasonably short maximum exists; longer inputs must be truncated, and each output only sees the positions within its receptive field.

3. Word Embeddings: From Words to Vectors

The first challenge is solved in two steps.

Step 1: map each word to an index. Add two special tokens to the vocabulary: <unk> for words not in the vocabulary and <pad> for padding. With <unk>=0, and=1, is=2, one=3, shining=4, sun=5, sweet=6, the=7, two=8, weather=9 and <pad>=10, “The sun is shining” becomes [7, 5, 2, 4, 10, 10, …], padded to the length of the longest sentence in the batch.

Step 2: map each index to a dense vector. Write index 7 as a one-hot vector e_7\in\mathbb{R}^{11} and multiply it by an embedding matrix E\in\mathbb{R}^{11\times 4}. Because e_7 has a single 1, the product e_7^\top E is just row 7 of E; on the slide that row is [7.1, 2.5, 1.5, 1.5]. Each word is now a 4-dimensional real vector, and gradients can flow into the entries of E.

The slide leaves an open question: where does E come from?

Supplement. In practice no one forms the one-hot vector. torch.nn.Embedding(num_embeddings=11, embedding_dim=4) stores E and returns rows by index, which gives the same result without the multiplication. The matrix is usually either learned from random initialization together with the rest of the network by backpropagation, or initialized from pretrained word vectors (word2vec, GloVe) and then frozen or fine-tuned. In modern language models the embedding table is simply the first layer and is trained with everything else. During training only the rows of words that appear in a batch receive a nonzero gradient.

4. Recurrent Neural Networks

4.1 Order Matters

The two “movie” sentences in Section 2.1 show that a sequence model must read words in order and remember what it has read. A feedforward network maps x\to h\to y once. An RNN adds a recurrent edge: the hidden state at time t is computed from the current input x_t and the previous hidden state h_{t-1}. Drawing one copy of the network per time step “unfolds” it into a deep feedforward network whose layers all share the same weights (Figure 1).

Left: an RNN cell with input x_t, hidden state h_t with a self-loop labeled W_hh, and output y_t. Right: the same network unrolled over three time steps, where every step uses the same W_hx, W_hh and W_yh.
Figure 1. Folded and unrolled views of a single-layer RNN. The recurrent edge becomes a chain of hidden states. The three weight matrices are shared by every time step, so the same parameters process a sequence of any length. Redrawn after the lecture's figure.

Weight sharing is what solves the variable-length problem. A five-word sentence and a five-hundred-word sentence are processed by the same parameters; the longer one simply runs the recurrence more times. It also builds in an assumption: the rule for updating the state is the same at every position.

4.2 The Equations

A single-layer RNN has three weight matrices, shared across time: W_{hx}\in\mathbb{R}^{n\times d} (input to hidden), W_{hh}\in\mathbb{R}^{n\times n} (hidden to hidden, the recurrent edge) and W_{yh} (hidden to output). At each step,

\begin{aligned} z_{h,t} &= W_{hx}\,x_t + W_{hh}\,h_{t-1} + b_h, &\qquad h_t &= \sigma_h\!\left(z_{h,t}\right),\\ z_{y,t} &= W_{yh}\,h_t + b_y, &\qquad y_t &= \sigma_y\!\left(z_{y,t}\right). \end{aligned}

The hidden activation \sigma_h is usually \tanh, and \sigma_y depends on the task, for example a softmax over the vocabulary. The initial state h_0 is usually a zero vector. The slides also write W_h=[W_{hh}\,;\,W_{hx}]: placing the two matrices side by side and multiplying by the stacked vector [h_{t-1}\,;\,x_t] gives the same z_{h,t}, which is how libraries implement it.

Parameter count. The hidden layer has nd + n^2 + n parameters, independent of T. With d=4 and n=3 that is 12+9+3=24. PyTorch’s nn.RNN(4, 3) reports 27 because it keeps two bias vectors, one for the input term and one for the recurrent term, which are redundant mathematically.

A worked example. Take one hidden unit with W_{hx}=0.5, W_{hh}=0.8, no bias, h_0=0, and the input sequence x=(1,0,0):

\begin{aligned} h_1 &= \tanh(0.5\cdot 1 + 0.8\cdot 0) = \tanh(0.5) \approx 0.462,\\ h_2 &= \tanh(0.5\cdot 0 + 0.8\cdot 0.462) = \tanh(0.370) \approx 0.354,\\ h_3 &= \tanh(0.8\cdot 0.354) = \tanh(0.283) \approx 0.276. \end{aligned}

The input seen at step 1 still affects the state at step 3, but the trace shrinks at every step because it is multiplied by 0.8 and passed through \tanh. The same shrinking reappears in Section 5 as the vanishing-gradient problem.

4.3 Multilayer RNNs

RNN layers can be stacked. The hidden state of layer 1 at time t, h^{(1)}_t, is the input of layer 2 at the same time step, and each layer keeps its own recurrent connection h^{(\ell)}_{t-1}\to h^{(\ell)}_t. Stacking adds depth at every time step, in addition to the depth across time that unrolling creates.

4.4 Bidirectional RNNs

A bidirectional RNN runs one RNN from left to right and a second, separate RNN from right to left, then concatenates their states at each position:

h_t = \left[\overrightarrow{h}_t\,;\,\overleftarrow{h}_t\right].

Every output now depends on the whole sequence. In “The movie my friend has not seen is good”, the representation of “seen” can use the words “is good” that come after it. This makes bidirectional RNNs a good fit for encoding an input that is fully available: tagging each word, classifying a sentence, or the encoder of a translation model (the attention examples in the next lecture use one). They cannot generate text, because when the model produces token t, the tokens after it have not been written, so the backward RNN has nothing to read.

4.5 Types of Sequence Tasks

Recurrence allows several input-output patterns:

Pattern Input and output Example
Many-to-one A sequence in, one fixed-size vector out Sentiment analysis: text in, class label out
One-to-many One input (not a sequence) in, a sequence out Image captioning: image in, sentence out
Many-to-many, direct One output per input step Video captioning frame by frame; tagging each word
Many-to-many, delayed Read the whole input, then produce an output sequence Machine translation

4.6 Encoder–Decoder (seq2seq)

The delayed many-to-many pattern is implemented with two RNNs . Consider translating “Where is the library” into “Dónde está la biblioteca”.

The weakness is a bottleneck: everything the decoder knows about the source must fit into one fixed-length vector c, and long sentences suffer. Attention, in the next lecture, lets the decoder look back at every encoder state instead.

Supplement. During training the decoder is usually fed the correct previous target word rather than its own prediction (“teacher forcing”), which lets all decoder steps be trained from the reference translation. At test time the decoder only has its own predictions, so early mistakes can compound. The mismatch between the two conditions is known as exposure bias.

5. Backpropagation Through Time

5.1 The Gradient

To train an RNN we unroll it over the T steps of a sequence, compute a loss L^{(t)} at every step that has an output, and add them:

L = \sum_{t=1}^{T} L^{(t)}.

The unrolled network is an ordinary computation graph, so backpropagation applies unchanged; applied to an unrolled RNN it is called backpropagation through time (BPTT) . The only new feature is weight sharing. The matrix W_{hh} is used at every step, so, as with shared convolution filters, its gradient is the sum of the contributions from every place it is used. For the loss at step t, the slides write

\frac{\partial L^{(t)}}{\partial W_{hh}} = \frac{\partial L^{(t)}}{\partial y_t}\cdot\frac{\partial y_t}{\partial h_t}\cdot \left(\sum_{k=1}^{t}\frac{\partial h_t}{\partial h_k}\cdot\frac{\partial h_k}{\partial W_{hh}}\right),

where \partial h_k/\partial W_{hh} means the “immediate” derivative at step k, treating h_{k-1} as a constant. The sum runs over every earlier step k at which W_{hh} influenced h_t. The factor that connects step k to step t is a product of one Jacobian per intervening step:

\frac{\partial h_t}{\partial h_k} = \prod_{i=k+1}^{t}\frac{\partial h_i}{\partial h_{i-1}}.

This product is straightforward to compute, and it is where the trouble starts.

5.2 Why Gradients Vanish or Explode

For a \tanh RNN, differentiating h_i=\tanh(W_{hx}x_i + W_{hh}h_{i-1}+b_h) gives

\frac{\partial h_i}{\partial h_{i-1}} = \mathrm{diag}\!\left(\tanh'(z_{h,i})\right) W_{hh} = \mathrm{diag}\!\left(1-h_i^2\right) W_{hh}.

A signal from step k must pass through t-k of these factors to reach step t. Scalars give the intuition. If every factor were 0.5, twenty steps would multiply the gradient by 0.5^{20}\approx 9.5\times 10^{-7}: the gradient vanishes, and the network gets almost no signal about how the input twenty steps back should have changed. If every factor were 1.5, twenty steps would multiply it by 1.5^{20}\approx 3.3\times 10^{3}: the gradient explodes, updates become huge and the loss can turn into NaN. The same problem appeared for deep feedforward networks in Lecture 7, but here it is worse: an RNN on a 100-word sentence is 100 layers deep with the same matrix at every layer, and dependencies across sentences require far more steps.

Supplement (Pascanu et al., this week’s reading). The scalar intuition can be made precise . Let \lambda_1 be the largest singular value of W_{hh} and let \gamma bound the derivative of the activation (\gamma=1 for \tanh, \gamma=1/4 for the sigmoid). Then each factor has norm at most \gamma\lambda_1, so

\left\lVert \frac{\partial h_t}{\partial h_k}\right\rVert \le \left(\gamma\,\lambda_1\right)^{t-k}.

If \lambda_1\lt 1/\gamma, long-range gradients are guaranteed to vanish exponentially fast; this is a sufficient condition. Conversely, \lambda_1\gt 1/\gamma is necessary for gradients to explode, but not sufficient, because saturated units have \tanh'\approx 0 and can absorb the growth. In the worked example of Section 4.2, every factor is 0.8\,\tanh'(z)\le 0.8, so the gradient shrinks at least as fast as 0.8^{t-k}. Pascanu et al. also describe the geometry: exploding gradients appear as steep “cliffs” in the loss surface, where a single gradient step can throw the parameters far away.

Figure 3 (in Section 6.4) shows these regimes in an actual randomly initialized network with 64 hidden units. With all singular values of W_{hh} equal to 0.9, the Jacobian norm falls by a factor of about 10 every 10 steps. With singular values of 3, it grows by roughly 10^{8} over 60 steps. With singular values of 1.5 it neither vanishes nor explodes, because saturation of \tanh cancels the growth of W_{hh}; keeping a plain RNN in that narrow regime during training is hard.

5.3 Fixes

Gradient clipping. If the norm of the gradient exceeds a threshold, rescale the gradient down to that norm while keeping its direction. For example, a gradient of norm 50 with threshold 5 is multiplied by 5/50=0.1. Clipping fixes exploding gradients only; it cannot create signal where the gradient has vanished. In PyTorch it is one line between the backward pass and the optimizer step:

loss.backward()
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=5.0)
optimizer.step()

Truncated backpropagation through time (TBPTT). Limit how many steps the gradient can travel back after each forward pass. A sequence may have 100 steps while gradients are propagated through only the last 20 or so. This bounds the length of the Jacobian product and the memory needed for stored activations. The cost is that the model cannot learn, through gradients, dependencies longer than the window, although the hidden state itself still carries information forward. In PyTorch, TBPTT means splitting a long sequence into chunks and calling h = h.detach() between chunks, so the value of the state is kept but the gradient stops there.

Change the architecture. The third fix, and the one that addresses vanishing gradients, is to give the memory a path along which the gradient is not multiplied by W_{hh} and squashed by \tanh at every step. That is the LSTM.

6. Long Short-Term Memory (LSTM)

6.1 Two Paths of Memory

The name is not an oxymoron. An LSTM keeps two states. The cell state C_t is the long-term memory; it is updated only by elementwise multiplication and addition. The hidden state h_t is the short-term memory and is what the cell passes to the next layer and the next time step (Figure 2).

An LSTM cell. The cell state runs along the top through an elementwise multiplication with the forget gate and an addition of the input gate times the input node. Four blocks at the bottom (sigma for forget, sigma for input, tanh for input node, sigma for output) all read the previous hidden state and the current input. The new hidden state is the output gate times tanh of the new cell state.
Figure 2. Inside an LSTM cell. The cell state (red) passes through the cell with only an elementwise product by the forget gate and an elementwise sum with new content. All four yellow blocks read [h_{t-1}, x_t] with their own weights. Redrawn after the lecture's figure.

6.2 Gates and Updates

A gate is a sigmoid layer. Its output lies between 0 and 1 and says how much of something to let through: 0 blocks everything, 1 keeps everything. A \tanh layer outputs values between -1 and 1 and proposes new content. Each of the four blocks reads x_t and h_{t-1} with its own weights:

\begin{aligned} f_t &= \sigma\!\left(W_{fx}\,x_t + W_{fh}\,h_{t-1} + b_f\right) &&\text{forget gate: how much of } C_{t-1} \text{ to keep}\\ i_t &= \sigma\!\left(W_{ix}\,x_t + W_{ih}\,h_{t-1} + b_i\right) &&\text{input gate: how much new content to write}\\ g_t &= \tanh\!\left(W_{gx}\,x_t + W_{gh}\,h_{t-1} + b_g\right) &&\text{input node: the candidate content}\\ o_t &= \sigma\!\left(W_{ox}\,x_t + W_{oh}\,h_{t-1} + b_o\right) &&\text{output gate: how much of the memory to expose} \end{aligned}

The cell state and hidden state are then updated as

C_t = \left(C_{t-1}\odot f_t\right) + \left(i_t\odot g_t\right), \qquad h_t = o_t\odot\tanh\!\left(C_t\right).

The first term of C_t is the old memory, partly erased by the forget gate. The second term is new content g_t, admitted by the input gate. The output gate decides how much of the squashed memory becomes the visible state h_t.

A worked example. For a single unit, suppose C_{t-1}=2.0 and the gates take the values f_t=0.9, i_t=0.1, g_t=0.5 and o_t=0.5. Then

C_t = 2.0\times 0.9 + 0.1\times 0.5 = 1.80 + 0.05 = 1.85, \qquad h_t = 0.5\times\tanh(1.85) \approx 0.5\times 0.952 \approx 0.476.

The cell kept 90% of its old memory and wrote only a little new content; the output gate exposed half of the result.

6.3 Why the Cell State Helps the Gradient

Differentiate the cell update along the cell-state path only, holding the gates fixed:

\frac{\partial C_t}{\partial C_{t-1}} = \mathrm{diag}(f_t) \qquad\Longrightarrow\qquad \frac{\partial C_t}{\partial C_k} \approx \prod_{i=k+1}^{t}\mathrm{diag}(f_i).

Compare this with the plain RNN’s factor \mathrm{diag}(1-h_i^2)\,W_{hh}. There is no repeated multiplication by a weight matrix and no \tanh on this path, only the forget gate, whose value the network chooses from the current input. When the network needs to remember, it can set f\approx 1. With f=0.95 at every step, twenty steps keep 0.95^{20}\approx 0.36 of the gradient, compared with 0.5^{20}\approx 10^{-6} in the scalar RNN example. The approximation sign is there because the gates themselves depend on h_{t-1}, which depends on C_{t-1}, so the full Jacobian has additional indirect terms.

6.4 Measuring It

Figure 3 measures these Jacobians directly in randomly initialized networks with 64 hidden units and random inputs. For the plain RNN we compute \partial h_t/\partial h_k exactly from the product formula. For the LSTM we compute \partial C_t/\partial C_k by finite differences, so the indirect paths through h are included. Nothing is trained; the plot shows the behavior a network starts from.

Log-scale plot of Jacobian spectral norm versus distance in time steps for three tanh RNNs and three LSTMs. The RNN with singular values 0.9 and the LSTM with forget bias 0 fall to about 1e-9 at distance 60. The RNN with singular values 3 rises to about 1e8. The RNN with singular values 1.5 and the LSTM with forget bias 3 stay within a factor of about 100 of 1. The LSTM with forget bias 1 decays slowly to about 1e-3.
Figure 3. How far back does the gradient signal reach? Spectral norm of the state-to-state Jacobian against the distance t-k, for \tanh RNNs whose recurrent matrix has all singular values equal to 0.9, 1.5 or 3 (blue), and LSTMs with forget-gate bias 0, 1 or 3 (red). Random initialization, 64 hidden units, one random input sequence (NumPy, seed 0). Computed for these notes, not taken from the lecture.

Three observations follow. First, the LSTM is not automatically immune: with forget bias 0 the forget gates sit near \sigma(0)=0.5, and the gradient decays about as fast as in the vanishing RNN. Second, raising the forget-gate bias pushes f toward 1 and keeps the signal alive over the whole 60-step window. Third, with b_f=3 the norm even grows slowly, through the indirect paths; the LSTM controls the gradient through the values of its gates rather than guaranteeing it.

Supplement. The original 1997 LSTM had no forget gate; its cell simply accumulated, C_t=C_{t-1}+i_t\odot g_t, so the gradient along the cell was exactly 1 (the “constant error carousel”). Gers et al. added the forget gate so the cell could reset itself . Initializing the forget bias to about 1, so that the network remembers by default early in training, is a common practical choice ; PyTorch initializes it like the other biases, so it must be set by hand.

As the slides note, gating slows the decay but does not remove it; long-range gradients still fade, only more slowly. This is one of the reasons discussed in Section 8.

6.5 LSTMs in PyTorch

The slide’s example:

import torch
import torch.nn as nn

rnn = nn.LSTM(10, 20, 2)            # input_size=10, hidden_size=20, num_layers=2
input = torch.randn(5, 3, 10)       # (seq_len, batch, input_size)
h0 = torch.randn(2, 3, 20)          # (num_layers, batch, hidden_size)
c0 = torch.randn(2, 3, 20)
output, (hn, cn) = rnn(input, (h0, c0))
# output: (5, 3, 20)  top layer's h_t at every step
# hn, cn: (2, 3, 20)  every layer's state at the last step

Two details cause most shape bugs. The default batch_first=False puts the sequence dimension first, so (5, 3, 10) is 5 time steps of a batch of 3. The initial states are indexed by layer first. With bidirectional=True, the last dimension of output doubles to 40 and the first dimension of h0 and c0 becomes num_layers * 2 = 4. Other options are bias, dropout (applied between stacked layers, not after the last one) and proj_size.

Parameter count. Each of the four blocks has an n\times d input matrix, an n\times n recurrent matrix and bias, so an LSTM layer has four times the parameters of a plain RNN layer of the same size. PyTorch stores the four blocks stacked, as weight_ih_l0 of shape (4n, d) and weight_hh_l0 of shape (4n, n), plus two bias vectors. For nn.LSTM(10, 20) this is 4\times(200+400+20+20)=2560 parameters. In the two-layer example, the second layer reads the 20-dimensional output of the first, so it has 4\times(400+400+40)=3360, for 5920 in total.

7. Gated Recurrent Unit (GRU)

The GRU is a simpler gated RNN, with two gates instead of three and no separate cell state:

\begin{aligned} r_t &= \sigma\!\left(W_{rx}\,x_t + W_{rh}\,h_{t-1} + b_r\right) &&\text{reset gate}\\ z_t &= \sigma\!\left(W_{zx}\,x_t + W_{zh}\,h_{t-1} + b_z\right) &&\text{update gate}\\ \tilde h_t &= \tanh\!\left(W_{hx}\,x_t + W_{hh}\left(r_t\odot h_{t-1}\right) + b_h\right) &&\text{candidate state}\\ h_t &= \left(1-z_t\right)\odot h_{t-1} + z_t\odot\tilde h_t &&\text{new hidden state} \end{aligned}

The update gate z_t decides how much of the old state to overwrite. It plays the role of the LSTM’s forget and input gates tied together, with f=1-z: whatever fraction of new content is written, the same fraction of old content is forgotten. With z_t=0.2, the new state keeps 80% of h_{t-1} and mixes in 20% of the candidate. The reset gate r_t lets the candidate ignore the past when it is not relevant; with r_t\approx 0 the candidate is computed almost entirely from x_t. The term (1-z_t)\odot h_{t-1} gives the GRU the same kind of additive path as the LSTM’s cell state.

With three weight blocks instead of four, a GRU has about 25% fewer parameters than an LSTM of the same size (1920 versus 2560 for input size 10 and hidden size 20 in PyTorch), and its accuracy is usually similar . torch.nn.GRU has the same interface as nn.LSTM without the cell state. One trap when reading the documentation: PyTorch swaps the roles of z and 1-z and writes h_t=(1-z_t)\odot n_t + z_t\odot h_{t-1}, where n_t is the candidate. The two conventions describe the same model with the gate’s meaning reversed.

8. Why RNNs Faded, and Why Recurrence Is Back

8.1 Why Transformers Replaced RNNs

The net effect is that transformers turned more data and compute into better models (BERT, GPT; from 2018 onward), and by about 2020 RNNs had mostly disappeared from frontier language modeling. The slide’s summary: transformers won on training parallelism, and they pay for it at inference time.

8.2 The Cost of Attention

When a transformer generates text, every new token attends to every previous token. Generating T tokens therefore costs O(T^2) compute in total, and the key-value cache that stores the past grows linearly with T. Going from 1,000 to 10,000 tokens multiplies the attention compute by about 100. An RNN carries a fixed-size state, so each generated token costs O(1) memory regardless of how long the context is.

8.3 Linear Recurrence: Parallel Training, Recurrent Inference

The trick that brings recurrence back is to remove the nonlinearity between time steps:

\begin{aligned} \text{classic RNN:}\quad & h_t = \tanh\!\left(W_{hh}\,h_{t-1} + W_{hx}\,x_t\right)\\ \text{linear recurrence:}\quad & h_t = A_t\,h_{t-1} + B_t\,x_t,\qquad y_t = C_t\,h_t \end{aligned}

(Here C_t is an output matrix, not the LSTM cell state.) Because the update is linear in h, all steps of a training sequence can be computed in parallel with a parallel scan. At generation time the model runs one step at a time like an RNN, with a fixed-size state. It gets the training parallelism of a transformer and the cheap inference of an RNN.

Supplement: why linearity allows a parallel scan. Consider the scalar case and write b_t for B_t x_t. Each step is the affine map h\mapsto a_t h + b_t, which we can store as the pair (a_t, b_t). Applying (a_1,b_1) and then (a_2,b_2) gives h\mapsto a_2(a_1 h + b_1)+b_2, another affine map, stored as (a_2a_1,\ a_2b_1+b_2). This combination rule is associative, so the T maps can be combined pairwise in a tree . The tree has about \log_2 T levels, so T=1024 steps need about 10 rounds of parallel work instead of 1024 sequential steps; a parallel prefix scan also returns every intermediate h_t. With \tanh between steps, the composition of two steps is no longer a map of the same simple form, and the shortcut fails.

Timeline. Linear attention → S4 structured state-space models → Mamba, in which A_t, B_t and C_t depend on the input , alongside RWKV and RetNet (2023) → xLSTM . Making the matrices input-dependent plays the same role as the LSTM’s gates: the model decides from the current token what to keep and what to overwrite.

Today: hybrids. Current models such as Jamba (2024), Nemotron-H and Qwen3-Next (2025) use mostly recurrent layers plus a few attention layers.

Why keep any attention? A fixed-size state cannot store everything. Exact recall and copying from long contexts remain hard for purely recurrent models . A few attention layers restore direct access to earlier tokens while the recurrent layers keep most of the inference cost low.

9. Checks for Understanding

  1. Why can one RNN process sentences of any length? The same weights W_{hx}, W_{hh}, W_{yh} are applied at every step; a longer sentence only runs the recurrence more times.
  2. How many parameters does the hidden layer of a plain RNN with d=10, n=20 have? nd+n^2+n = 200+400+20 = 620 (640 in PyTorch, which keeps two bias vectors). An LSTM has four blocks and a GRU three.
  3. Why can a bidirectional RNN not generate text? Its backward direction reads tokens that come later, which do not exist yet during generation.
  4. What limits a seq2seq model with a single context vector? All information about the input must pass through one fixed-length vector c, which loses detail on long inputs.
  5. Why do gradients vanish or explode in BPTT? The gradient across t-k steps is a product of t-k Jacobians \mathrm{diag}(1-h_i^2)W_{hh}; its norm behaves roughly like a number raised to the power t-k.
  6. Does gradient clipping fix vanishing gradients? No. It only rescales gradients that are too large.
  7. Why does the LSTM’s cell state help? Along the cell path, each step multiplies the gradient only by the forget gate, which the network can keep near 1, instead of by W_{hh} and a \tanh derivative.
  8. What is the trade-off between transformers and recurrent models? Transformers train in parallel but cost O(T^2) attention and a growing cache at inference; recurrent models keep an O(1) state per token. Linear recurrences get both parallel training and recurrent inference.

10. Key Takeaways

Further reading. Karpathy’s blog post trains character-level RNNs on several datasets and inspects what individual units learn. Chapter 10 of Goodfellow et al. covers the material of this lecture in more depth, including echo state networks and leaky units as other answers to long-range dependencies.