Section 1: Why We Need Regularization

Where we left off

Last lecture we saw that weight sharing and sparse connectivity tame the parameter explosion of fully-connected layers, and that backprop through a convolution is itself a convolution (with the kernel flipped). But the punchline was: CNNs can still overfit, even though they have far fewer parameters than an equivalent fully-connected layer. So today’s question is: what should we do about overfitting?

Regularization is one tool among many

Regularization is one strategy to improve generalization, and it is just one of many ways to do it. The boundary between regularization and other engineering techniques is not always clear. A rough way to think about the whole pipeline is:

  1. Dataset
  2. Architecture setup
  3. Normalization
  4. Training loop
  5. Regularization

The general strategies against overfitting are:

  • Collect more data, especially high-quality data. This is the best option and always recommended (semi-supervised learning, transfer learning and self-supervised learning are ways to get more out of the data we have).
  • Data augmentation is helpful.
  • Reducing model capacity can help.

Note that training is basically maximum likelihood, so the model is only ever as good as the data it is fit on. That is why high-quality data comes first. Which of the other tools is best is “not entirely settled”, and it depends on the problem.

The bias-variance trade-off

For regression with squared-error loss, the expected prediction error at a fixed input x decomposes into three parts. Let f(x)=\mathbb{E}[Y\mid X=x] and let \hat f_D be the predictor fitted on a random training set D. For a fresh response independent of the training set,

\mathbb{E}_{D,Y\mid x}[(Y-\hat f_D(x))^2] = (\mathbb{E}_D[\hat f_D(x)]-f(x))^2 + \operatorname{Var}_D(\hat f_D(x)) + \operatorname{Var}(Y\mid X=x).
  • Bias is the error from a model that is too simple to capture the true pattern.
  • Variance is the error from a model that is too sensitive to the specific training sample it happened to see.

Increasing capacity (more layers or units) typically lowers bias but raises variance. This gives us a useful way to look at every regularizer in this lecture: each technique trades a little bias for a lot less variance.This is also how Goodfellow et al. put it: an effective regularizer is one that makes a profitable trade, reducing variance a lot without increasing bias too much.

The accuracy example below illustrates underfitting and overfitting qualitatively. The squared-error identity above does not decompose classification accuracy into these three terms.

Model Train acc. Test acc. Diagnosis
A: small MLP 60% 58% Both low, so high bias (underfitting)
B: large MLP 99% 71% Huge gap, so high variance (overfitting)

Model B is actually the better model, even though it overfits. A large model that we then regularize usually beats starting with a small one.

Rule of thumb: if the training accuracy is low, look at bias. If the train/test gap is big, look at variance.

This is why the advice was: first overfit a model, then regularize it, instead of starting with a smaller model and hoping it is the right size.

Section 2: More Data and Data Augmentation

More data is the cleanest way to reduce variance. Data is valuable, but often we cannot afford more of it. Data augmentation is the cheap alternative.

The idea: if we know the label should not depend on some transformation h, then for every training pair (x_i, y_i) we can generate a new pair (h(x_i), y_i). This increases the diversity of the training data.

The catch is that we must already know something our outcome does not depend on. For image classification, the label should not change under, e.g., rotation, zooming or a sepia filter. But a transformation that changes the label is harmful. For digit recognition, flipping a “6” by 180° turns it into a “9”.

In PyTorch this is done with TorchVision. The lecture example uses MNIST: the randomly augmented digits are slightly cropped and rotated versions of the originals, but still show the same digit, so the label stays the same. The key detail is that random augmentation is applied only to the training data. The test transform is deterministic (resize, then crop the center), so that the evaluation is reproducible:

import PIL import torchvision from torchvision import datasets # Note transforms.ToTensor() scales input images to 0-1 range training_transforms = torchvision.transforms.Compose([ torchvision.transforms.Resize(size=(32, 32)), torchvision.transforms.RandomCrop(size=(28, 28)), torchvision.transforms.RandomRotation(degrees=30, interpolation=PIL.Image.BILINEAR), torchvision.transforms.ToTensor(), torchvision.transforms.Normalize(mean=(0.5,), std=(0.5,)), # normalize does (x_i - mean) / std # if images are [0, 1], they will be [-1, 1] afterwards ]) test_transforms = torchvision.transforms.Compose([ torchvision.transforms.ToTensor(), torchvision.transforms.Resize(size=(32, 32)), torchvision.transforms.CenterCrop(size=(28, 28)), torchvision.transforms.Normalize(mean=(0.5,), std=(0.5,)), ]) train_dataset = datasets.MNIST(root='data', train=True, transform=training_transforms, download=True) test_dataset = datasets.MNIST(root='data', train=False, transform=test_transforms)

The two pipelines differ in exactly the random steps (RandomCrop, RandomRotation in training and CenterCrop at test time), while the final normalization to [-1, 1] is the same for both.Code from the Lecture 9 slides, which come from the linked notebook L10/code/data-augmentation.ipynb in the repository rasbt/stat453-deep-learning-ss21.

Section 3: Reducing Capacity and Early Stopping

The simplest model that fits should generalize best

Several knobs shrink the effective capacity of a network:

  • Smaller architecture: fewer hidden layers and units.
  • Smaller weights: early stopping, L2 norm penalty.
  • Sparse weights or activations: ReLU with an L1 penalty prunes units.
  • Noise: dropout.

For recent LLMs and foundation models there is another option: take a large pretrained model and do efficient fine-tuning. We freeze most of the weights and only train (fine-tune) the remaining small number of parameters.

Three data splits

Early stopping needs us to be careful with the data splits. As always we split into three parts:

  • Training set: used to update the weights.
  • Validation set: used for tuning (hyperparameters, when to stop). It lives inside the training loop.
  • Test set: used once at the end, outside the loop. In the end we care about the test accuracy, so we must not touch it before.

Some fields swap the names of test and validation (medicine, for example), so watch the terminology when reading papers.

Early stopping

Track a validation metric during training and keep the checkpoint that scores best on that metric.

Early stopping. The training accuracy keeps improving, but the validation accuracy eventually peaks and then drops again. The good early stopping point (blue box) is where the validation performance is best, and we keep the weights from that epoch. Figure from the Lecture 9 slides.

In practice we keep a copy of the best weights seen so far and stop once the validation performance has not improved for a number of epochs, called the patience. The code below monitors the validation loss, where lower is better. The figure above plots accuracy, where higher is better, so if you monitor accuracy instead, just flip the comparison (val_acc > best_val) and start from -inf:

best_val, best_state, bad_epochs, patience = float("inf"), None, 0, 5 for epoch in range(max_epochs): model.train() train_one_epoch(model, train_loader) model.eval() with torch.no_grad(): val_loss = evaluate(model, val_loader) if val_loss < best_val: best_val, bad_epochs = val_loss, 0 best_state = {k: v.clone() for k, v in model.state_dict().items()} else: bad_epochs += 1 if bad_epochs >= patience: break if best_state is None: raise RuntimeError("No valid validation checkpoint was saved.") model.load_state_dict(best_state)

Early stopping acts as an implicit regularizer by restricting the selected parameters to checkpoints along the training trajectory. If validation performance deteriorates while training performance improves, restoring an earlier checkpoint can improve generalization.

For linear least squares with zero initialization and a stable step size, early stopping and ridge regression both suppress directions of low curvature. In this setting, 1/(\epsilon\tau) gives an approximate scale for the corresponding ridge strength after \tau steps of size \epsilon. The two methods generally shrink each direction by different factors.

Section 4: Norm Penalties (L1 and L2)

Adding a penalty to the loss

We add a penalty on the size of the weights to the loss:

\tilde{J}(w) = J(w) + \lambda\, \Omega(w),

where J is the usual data loss and \lambda \ge 0 controls how strongly we regularize.

  • L2 penalizes \Omega(w)=\sum_i w_i^2. This is the same idea as ridge regression, and it encourages small values.
  • L1 penalizes \Omega(w)=\sum_i \lvert w_i \rvert. This is the same idea as LASSO, and it encourages zeros, i.e. sparse solutions.

In neural networks we typically only penalize the weights and leave the biases alone.

Regularization is MAP estimation

In Lecture 3 we saw that MAP estimation with a Gaussian prior gives L2 and with a Laplace prior gives L1. Here is the derivation again. MAP maximizes the posterior P(w \mid D), which by Bayes’ rule is proportional to P(D \mid w)\,P(w). Taking the negative log (like we did in Lecture 6 to go from a likelihood to a loss):

-\log P(w \mid D) = \underbrace{-\log P(D \mid w)}_{\text{data loss}} + \underbrace{[-\log P(w)]}_{\text{penalty from the prior}} + \text{const}.

So the penalty is nothing but the negative log of the prior:

Prior P(w) Penalty Name
Gaussian \sum_i w_i^2 L2 / ridge
Laplace \sum_i \lvert w_i \rvert L1 / LASSO

Geometric picture

Penalizing the norm is the same as constraining the weights to lie in a region around the origin: a circle (ball) for L2 and a diamond (square rotated by 45°) for L1. The solution is the point where the contours of the data loss first touch the region.

(a) L2. The penalty region is a circle.
(b) L1. The penalty region is a diamond.
Geometry of L2 and L1 regularization. The ellipses are contours of the cost function, whose minimum is the first component of the objective. The shaded region belongs to the penalty term, which is smallest at the origin. The solution is the compromise between the two. For L2 it lies on the circle with both weights non-zero, only shrunk. For L1 it lands on a corner of the diamond, on an axis, so one weight is exactly zero. Figures from the Lecture 9 slides.

The corners of the diamond lie on the axes, and that is why L1 produces sparse solutions while L2 only shrinks.

Weight decay

For multilayer networks we simply sum the penalty over all layers. In the notation of the lecture, with n training examples, the L2-regularized cost is

\text{Cost} = \frac{1}{n}\sum_{i=1}^{n} \mathcal{L}\big(y^{[i]}, \hat{y}^{[i]}\big) + \frac{\lambda}{n}\sum_{l=1}^{L} \big\lVert \mathbf{W}^{(l)} \big\rVert_F^2, \qquad \big\lVert \mathbf{W}^{(l)} \big\rVert_F^2 = \sum_i \sum_j \big(w_{i,j}^{(l)}\big)^2,

where \lVert\cdot\rVert_F is the (squared) Frobenius norm of the weight matrix of layer l. The gradient of the penalty with respect to a single weight is \frac{2\lambda}{n} w_{i,j}, so the gradient descent update with learning rate \eta changes from the regular update to the one with L2 regularization:

w_{i,j} := w_{i,j} - \eta\,\frac{\partial \mathcal{L}}{\partial w_{i,j}} \qquad\longrightarrow\qquad w_{i,j} := w_{i,j} - \eta\left(\frac{\partial \mathcal{L}}{\partial w_{i,j}} + \frac{2\lambda}{n}\, w_{i,j}\right).

Rewriting the right-hand side gives w_{i,j} := \big(1 - \tfrac{2\eta\lambda}{n}\big) w_{i,j} - \eta\,\frac{\partial \mathcal{L}}{\partial w_{i,j}}. For a sufficiently small step size, the first term shrinks the weight multiplicatively. This is the weight-decay form of L2 regularization under plain gradient descent. Away from zero, an L1 subgradient step moves the weight by a fixed amount toward zero and can overshoot it. A proximal update instead uses soft thresholding, which maps sufficiently small magnitudes to exactly zero.

L2 regularization in PyTorch

In PyTorch you can add the penalty to the cost manually, or let the optimizer do it with weight_decay:

# manually: only penalize the weights, not the biases L2 = 0. for name, p in model.named_parameters(): if 'weight' in name: L2 = L2 + (p**2).sum() cost = cost + 2./targets.size(0) * LAMBDA * L2 optimizer.zero_grad() cost.backward() # automatically: apply weight decay only to the weights weight_params = [] other_params = [] for name, p in model.named_parameters(): if p.requires_grad: if 'weight' in name: weight_params.append(p) else: other_params.append(p) optimizer = torch.optim.SGD( [ {'params': weight_params, 'weight_decay': LAMBDA}, {'params': other_params, 'weight_decay': 0.0}, ], ======= weights, other_params = [], [] for name, param in model.named_parameters(): group = weights if name.rsplit(".", 1)[-1] == "weight" else other_params group.append(param) # n is the training-set size and cost is the mean minibatch data loss. # Option 1: backpropagate this loss with weight_decay=0. penalized_loss = cost + (LAMBDA / n) * sum(p.square().sum() for p in weights) # Option 2: backpropagate cost with this optimizer. optimizer = torch.optim.SGD( [{"params": weights, "weight_decay": 2 * LAMBDA / n}, {"params": other_params, "weight_decay": 0.0}], lr=0.1, )

The parameter groups apply weight decay only to parameters whose names contain ‘weight’. Biases remain trainable but receive no weight decay, matching the weight-only policy in the manual example above.

The constant in front of the penalty only rescales the regularization strength λ, so conventions like λ/n or 2λ/n differ only in the choice of λ. With weight_decay=LAMBDA, SGD adds LAMBDA * w to the gradient, so LAMBDA plays the role of the factor in front of w in the update above. For adaptive optimizers like Adam this is not exactly the same as an L2 penalty, which is why AdamW exists.

======= The factor of two comes from differentiating the squared norm. Both alternatives apply the same gradient to the same weights. Use one option per update to avoid applying the penalty twice. AdamW decouples parameter shrinkage from the adaptive gradient update, so its weight decay differs from adding an L2 penalty to Adam’s loss.

Section 5: Dropout

Dropout is a deep-learning-specific method. When propagating forward during training, we randomly drop some neurons by setting their activations to 0.

Bernoulli masks

With drop probability p, every unit gets an independent Bernoulli mask m_i:

m_i = \begin{cases} 0 & \text{with probability } p \\ 1 & \text{with probability } 1-p \end{cases} \qquad \tilde{a}_i = m_i \, a_i .

In practice we sample v \sim \mathrm{Uniform}(0,1) for each unit and drop it if v \lt p. A new mask is drawn for every minibatch.

import torch.nn as nn model = nn.Sequential( nn.Linear(784, 256), nn.ReLU(), nn.Dropout(drop_proba), # drop_proba in [0, 1] nn.Linear(256, 10), )

Why does it work?

  1. No single connection can be relied on. The network cannot depend on particular connections too heavily, so it has to use more of them. The weights end up more spread out, which may lead to smaller weights, similar to the L2 norm. (As a side note from the slides: you can use different dropout probabilities in different layers, and assigning them proportional to the number of units in a layer is not a bad idea.)
  2. Implicit ensemble. Every minibatch trains a different sub-network. With h hidden units there are M = 2^h possible masks, so over the minibatch iterations we sample from an exponentially large family of models. These models share weights, which is itself a form of regularization. At inference time we could average over all of these models, but that is extremely expensive. This is where the trick below comes in.Goodfellow et al. describe dropout as making bagging practical for exponentially many large networks.
  3. It suppresses high-order interactions. This is the view from the lecturer’s own paper on dropout and interaction effects. An interaction of order k is an effect that needs k variables at the same time (e.g. “feature A and feature B together”). Two things happen as k grows:
    • With N input variables there are \binom{N}{k} possible order-k interactions, which grows roughly like N^k for the first few orders. So there are many more high-order combinations that could fit noise in the training data.
    • An order-k interaction only survives a dropout step if all k of its variables survive. With drop rate r this has probability (1-r)^k, which decays exponentially in k.

    The slide then states that these exponential rates cancel out, and that this anti-interaction regularization happens at every layer.The slide only has this one line about the cancellation. One way to read it is that the exponentially growing number of candidate interactions and the exponentially decaying survival probability work against each other, so it is not obvious in advance which one wins. The paper is what shows that the net effect is a suppression of higher orders. The net effect is that dropout puts more pressure on higher-order effects than on low-order ones. In the experiment from the paper that is shown on the slide, the share of the normalized effect size that comes from first-order (additive) terms grows from roughly a quarter to roughly three quarters when the dropout rate goes from 0 to 0.75, while the higher orders shrink. A practical consequence: if you want to regularize away more complex interactions, increase the dropout rate.

Inverted dropout and train() vs. eval()

If we drop units during training but use all units at test time, the expected input to the next layer changes. The fix is inverted dropout: scale the surviving activations by 1/(1-p) during training:

\tilde{a}_i = \frac{m_i}{1-p}\, a_i .

For a keep probability 1-p>0, the mask has mean 1-p, so \mathbb{E}[\tilde a_i\mid a_i]=a_i. PyTorch’s dropout layer is the identity in evaluation mode. Matching the mean at this layer does not guarantee that a later nonlinear prediction equals the mean prediction across dropout masks.

In PyTorch the behavior depends on the mode of the model:

  • model.train() turns dropout on. Use this in the training loop.
  • model.eval() turns dropout off. Use this when you just want to see how good the model is.
model.train() for x, y in train_loader: ... model.eval() with torch.no_grad(): val_acc = evaluate(model, val_loader)

Forgetting model.eval() is a classic bug: the validation numbers come out noisy and too pessimistic, because dropout is still active.

The slides compare training with and without dropout on MNIST with a multilayer perceptron. Without dropout, the training accuracy goes towards 100% while the validation accuracy plateaus at around 97 to 98%, so there is a visible gap. With 50% dropout, the training and validation curves lie almost on top of each other at roughly 97.5%, so the gap is gone. The minibatch loss is noisier and stays higher, because every step uses a randomly thinned network.

Key Takeaways

  • The bias–variance decomposition above applies to squared error. Regularization can reduce variance at the cost of higher bias.
  • Low training accuracy can flag underfitting or an optimization problem. A large training–validation gap can flag overfitting.
  • More (and more diverse) data is the best fix. Augmentation only helps if we know a transformation the label does not depend on, and it is applied only to the training data.
  • Early stopping uses the validation set to pick the stopping point and acts as an implicit regularizer. The test set is touched only once.
  • L2 (ridge, Gaussian prior) encourages small weights. L1 (LASSO, Laplace prior) encourages zeros. Both are just the negative log of a prior added to the data loss.
  • Dropout randomly zeroes units during training, which prevents co-adaptation and approximates averaging over an exponential ensemble of weight-sharing networks. Use model.train() and model.eval() correctly.