STAT 453

Lecture 09: Regularization

Bias and variance, data augmentation, early stopping, L1/L2 penalties as MAP estimation, and dropout, with the PyTorch snippets for each.

Section 1: Why We Need Regularization

Where we left off

Last lecture we saw that weight sharing and sparse connectivity tame the parameter explosion of fully-connected layers, and that backprop through a convolution is itself a convolution (with the kernel flipped). But the punchline was: CNNs can still overfit, even though they have far fewer parameters than an equivalent fully-connected layer. So today’s question is: what should we do about overfitting?

Regularization is one tool among many

Regularization is one strategy to improve generalization, and it is just one of many ways to do it. The boundary between regularization and other engineering techniques is not always clear. A rough way to think about the whole pipeline is:

  1. Dataset
  2. Architecture setup
  3. Normalization
  4. Training loop
  5. Regularization

The general strategies against overfitting are:

Note that training is basically maximum likelihood, so the model is only ever as good as the data it is fit on. That is why high-quality data comes first. Which of the other tools is best is “not entirely settled”, and it depends on the problem.

The bias-variance trade-off

The expected test error decomposes into three parts:

\mathbb{E}[\text{test error}] = \text{bias}^2 + \text{variance} + \text{irreducible noise}.

Increasing capacity (more layers or units) typically lowers bias but raises variance. This gives us a useful way to look at every regularizer in this lecture: each technique trades a little bias for a lot less variance.This is also how Goodfellow et al. put it: an effective regularizer is one that makes a profitable trade, reducing variance a lot without increasing bias too much.

A concrete example from the lecture:

Model Train acc. Test acc. Diagnosis
A: small MLP 60% 58% Both low, so high bias (underfitting)
B: large MLP 99% 71% Huge gap, so high variance (overfitting)

Model B is actually the better model, even though it overfits. A large model that we then regularize usually beats starting with a small one.

Rule of thumb: if the training accuracy is low, look at bias. If the train/test gap is big, look at variance.

This is why the advice was: first overfit a model, then regularize it, instead of starting with a smaller model and hoping it is the right size.

Section 2: More Data and Data Augmentation

More data is the cleanest way to reduce variance. Data is valuable, but often we cannot afford more of it. Data augmentation is the cheap alternative.

The idea: if we know the label should not depend on some transformation h, then for every training pair (x_i, y_i) we can generate a new pair (h(x_i), y_i). This increases the diversity of the training data.

The catch is that we must already know something our outcome does not depend on. For image classification, the label should not change under, e.g., rotation, zooming or a sepia filter. But a transformation that changes the label is harmful. For digit recognition, flipping a “6” by 180° turns it into a “9”.

In PyTorch this is done with TorchVision. The lecture example uses MNIST: the randomly augmented digits are slightly cropped and rotated versions of the originals, but still show the same digit, so the label stays the same. The key detail is that random augmentation is applied only to the training data. The test transform is deterministic (resize, then crop the center), so that the evaluation is reproducible:

import PIL import torchvision from torchvision import datasets # Note transforms.ToTensor() scales input images to 0-1 range training_transforms = torchvision.transforms.Compose([ torchvision.transforms.Resize(size=(32, 32)), torchvision.transforms.RandomCrop(size=(28, 28)), torchvision.transforms.RandomRotation(degrees=30, interpolation=PIL.Image.BILINEAR), torchvision.transforms.ToTensor(), torchvision.transforms.Normalize(mean=(0.5,), std=(0.5,)), # normalize does (x_i - mean) / std # if images are [0, 1], they will be [-1, 1] afterwards ]) test_transforms = torchvision.transforms.Compose([ torchvision.transforms.ToTensor(), torchvision.transforms.Resize(size=(32, 32)), torchvision.transforms.CenterCrop(size=(28, 28)), torchvision.transforms.Normalize(mean=(0.5,), std=(0.5,)), ]) train_dataset = datasets.MNIST(root='data', train=True, transform=training_transforms, download=True) test_dataset = datasets.MNIST(root='data', train=False, transform=test_transforms)

The two pipelines differ in exactly the random steps (RandomCrop, RandomRotation in training and CenterCrop at test time), while the final normalization to [-1, 1] is the same for both.Code from the Lecture 9 slides, which come from the linked notebook L10/code/data-augmentation.ipynb in the repository rasbt/stat453-deep-learning-ss21.

Section 3: Reducing Capacity and Early Stopping

The simplest model that fits should generalize best

Several knobs shrink the effective capacity of a network:

For recent LLMs and foundation models there is another option: take a large pretrained model and do efficient fine-tuning. We freeze most of the weights and only train (fine-tune) the remaining small number of parameters.

Three data splits

Early stopping needs us to be careful with the data splits. As always we split into three parts:

Some fields swap the names of test and validation (medicine, for example), so watch the terminology when reading papers.

Early stopping

The idea is simple: watch the gap between training and validation performance during training, and stop at the “right point”.

Early stopping. The training accuracy keeps improving, but the validation accuracy eventually peaks and then drops again. The good early stopping point (blue box) is where the validation performance is best, and we keep the weights from that epoch. Figure from the Lecture 9 slides.

In practice we keep a copy of the best weights seen so far and stop once the validation performance has not improved for a number of epochs, called the patience. The code below monitors the validation loss, where lower is better. The figure above plots accuracy, where higher is better, so if you monitor accuracy instead, just flip the comparison (val_acc > best_val) and start from -inf:

best_val, best_state, bad_epochs, patience = float("inf"), None, 0, 5 for epoch in range(max_epochs): train_one_epoch(model, train_loader) val_loss = evaluate(model, val_loader) if val_loss < best_val: best_val, bad_epochs = val_loss, 0 best_state = {k: v.clone() for k, v in model.state_dict().items()} else: bad_epochs += 1 if bad_epochs >= patience: break model.load_state_dict(best_state) # go back to the best epoch

This makes early stopping an implicit regularizer, and not just a compute-saving trick. Why does it generalize better? Deep networks are initialized with small random weights, where many weights cancel out, so the network starts out behaving almost like a linear model. Training then makes it gradually more complex. Stopping early means it never gets to the very complex solutions that overfit.

There is also a precise version of this for linear models. With a quadratic loss, gradient descent with learning rate \epsilon run for \tau steps behaves like L2 regularization with \alpha \approx 1/(\epsilon\tau). More training steps correspond to weaker regularization.

Section 4: Norm Penalties (L1 and L2)

Adding a penalty to the loss

We add a penalty on the size of the weights to the loss:

\tilde{J}(w) = J(w) + \lambda\, \Omega(w),

where J is the usual data loss and \lambda \ge 0 controls how strongly we regularize.

In neural networks we typically only penalize the weights and leave the biases alone.

Regularization is MAP estimation

In Lecture 3 we saw that MAP estimation with a Gaussian prior gives L2 and with a Laplace prior gives L1. Here is the derivation again. MAP maximizes the posterior P(w \mid D), which by Bayes’ rule is proportional to P(D \mid w)\,P(w). Taking the negative log (like we did in Lecture 6 to go from a likelihood to a loss):

-\log P(w \mid D) = \underbrace{-\log P(D \mid w)}_{\text{data loss}} \;\underbrace{-\log P(w)}_{\text{penalty from the prior}} + \text{const}.

So the penalty is nothing but the negative log of the prior:

Prior P(w) Penalty Name
Gaussian \sum_i w_i^2 L2 / ridge
Laplace \sum_i \lvert w_i \rvert L1 / LASSO

Geometric picture

Penalizing the norm is the same as constraining the weights to lie in a region around the origin: a circle (ball) for L2 and a diamond (square rotated by 45°) for L1. The solution is the point where the contours of the data loss first touch the region.

(a) L2. The penalty region is a circle.
(b) L1. The penalty region is a diamond.
Geometry of L2 and L1 regularization. The ellipses are contours of the cost function, whose minimum is the first component of the objective. The shaded region belongs to the penalty term, which is smallest at the origin. The solution is the compromise between the two. For L2 it lies on the circle with both weights non-zero, only shrunk. For L1 it lands on a corner of the diamond, on an axis, so one weight is exactly zero. Figures from the Lecture 9 slides.

The corners of the diamond lie on the axes, and that is why L1 produces sparse solutions while L2 only shrinks.

Weight decay

For multilayer networks we simply sum the penalty over all layers. In the notation of the lecture, with n training examples, the L2-regularized cost is

\text{Cost} = \frac{1}{n}\sum_{i=1}^{n} \mathcal{L}\big(y^{[i]}, \hat{y}^{[i]}\big) + \frac{\lambda}{n}\sum_{l=1}^{L} \big\lVert \mathbf{W}^{(l)} \big\rVert_F^2, \qquad \big\lVert \mathbf{W}^{(l)} \big\rVert_F^2 = \sum_i \sum_j \big(w_{i,j}^{(l)}\big)^2,

where \lVert\cdot\rVert_F is the (squared) Frobenius norm of the weight matrix of layer l. The gradient of the penalty with respect to a single weight is \frac{2\lambda}{n} w_{i,j}, so the gradient descent update with learning rate \eta changes from the regular update to the one with L2 regularization:

w_{i,j} := w_{i,j} - \eta\,\frac{\partial \mathcal{L}}{\partial w_{i,j}} \qquad\longrightarrow\qquad w_{i,j} := w_{i,j} - \eta\left(\frac{\partial \mathcal{L}}{\partial w_{i,j}} + \frac{2\lambda}{n}\, w_{i,j}\right).

Rewriting the right-hand side gives w_{i,j} := \big(1 - \tfrac{2\eta\lambda}{n}\big) w_{i,j} - \eta\,\frac{\partial \mathcal{L}}{\partial w_{i,j}}. Each step first multiplicatively shrinks the weight and then takes the usual step. This is why L2 regularization is also called weight decay. For L1 the shrinkage is a constant amount in the direction of \mathrm{sign}(w) per step (independent of the weight size), which is what pushes small weights all the way to zero.

L2 regularization in PyTorch

In PyTorch you can add the penalty to the cost manually, or let the optimizer do it with weight_decay:

# manually: only penalize the weights, not the biases L2 = 0. for name, p in model.named_parameters(): if 'weight' in name: L2 = L2 + (p**2).sum() cost = cost + 2./targets.size(0) * LAMBDA * L2 optimizer.zero_grad() cost.backward() # automatically: the optimizer adds the decay term to the gradient optimizer = torch.optim.SGD(model.parameters(), lr=0.1, weight_decay=LAMBDA) The constant in front of the penalty only rescales the regularization strength λ, so conventions like λ/n or 2λ/n differ only in the choice of λ. With weight_decay=LAMBDA, SGD adds LAMBDA * w to the gradient, so LAMBDA plays the role of the factor in front of w in the update above. For adaptive optimizers like Adam this is not exactly the same as an L2 penalty, which is why AdamW exists.

Section 5: Dropout

Dropout is a deep-learning-specific method. When propagating forward during training, we randomly drop some neurons by setting their activations to 0.

Bernoulli masks

With drop probability p, every unit gets an independent Bernoulli mask m_i:

m_i = \begin{cases} 0 & \text{with probability } p \\ 1 & \text{with probability } 1-p \end{cases} \qquad \tilde{a}_i = m_i \, a_i .

In practice we sample v \sim \mathrm{Uniform}(0,1) for each unit and drop it if v \lt p. A new mask is drawn for every minibatch.

import torch.nn as nn model = nn.Sequential( nn.Linear(784, 256), nn.ReLU(), nn.Dropout(drop_proba), # drop_proba in [0, 1] nn.Linear(256, 10), )

Why does it work?

  1. No single connection can be relied on. The network cannot depend on particular connections too heavily, so it has to use more of them. The weights end up more spread out, which may lead to smaller weights, similar to the L2 norm. (As a side note from the slides: you can use different dropout probabilities in different layers, and assigning them proportional to the number of units in a layer is not a bad idea.)
  2. Implicit ensemble. Every minibatch trains a different sub-network. With h hidden units there are M = 2^h possible masks, so over the minibatch iterations we sample from an exponentially large family of models. These models share weights, which is itself a form of regularization. At inference time we could average over all of these models, but that is extremely expensive. This is where the trick below comes in.Goodfellow et al. describe dropout as making bagging practical for exponentially many large networks.
  3. It suppresses high-order interactions. This is the view from the lecturer’s own paper on dropout and interaction effects. An interaction of order k is an effect that needs k variables at the same time (e.g. “feature A and feature B together”). Two things happen as k grows:
    • With N input variables there are \binom{N}{k} possible order-k interactions, which grows roughly like N^k for the first few orders. So there are many more high-order combinations that could fit noise in the training data.
    • An order-k interaction only survives a dropout step if all k of its variables survive. With drop rate r this has probability (1-r)^k, which decays exponentially in k.

    The slide then states that these exponential rates cancel out, and that this anti-interaction regularization happens at every layer.The slide only has this one line about the cancellation. One way to read it is that the exponentially growing number of candidate interactions and the exponentially decaying survival probability work against each other, so it is not obvious in advance which one wins. The paper is what shows that the net effect is a suppression of higher orders. The net effect is that dropout puts more pressure on higher-order effects than on low-order ones. In the experiment from the paper that is shown on the slide, the share of the normalized effect size that comes from first-order (additive) terms grows from roughly a quarter to roughly three quarters when the dropout rate goes from 0 to 0.75, while the higher orders shrink. A practical consequence: if you want to regularize away more complex interactions, increase the dropout rate.

Inverted dropout and train() vs. eval()

If we drop units during training but use all units at test time, the expected input to the next layer changes. The fix is inverted dropout: scale the surviving activations by 1/(1-p) during training:

\tilde{a}_i = \frac{m_i}{1-p}\, a_i .

Then the expected activation is unchanged, and at inference time we do nothing. This is what most frameworks actually implement. It is helpful because a model is trained once but used many times at “test” time, so we want inference to be free of extra operations.

In PyTorch the behavior depends on the mode of the model:

model.train() for x, y in train_loader: ... model.eval() with torch.no_grad(): val_acc = evaluate(model, val_loader)

Forgetting model.eval() is a classic bug: the validation numbers come out noisy and too pessimistic, because dropout is still active.

The slides compare training with and without dropout on MNIST with a multilayer perceptron. Without dropout, the training accuracy goes towards 100% while the validation accuracy plateaus at around 97 to 98%, so there is a visible gap. With 50% dropout, the training and validation curves lie almost on top of each other at roughly 97.5%, so the gap is gone. The minibatch loss is noisier and stays higher, because every step uses a randomly thinned network.

Key Takeaways