STAT 453: Introduction to Deep Learning and Generative Models (Fall 2026)

Instructor: Ben Lengerich

Useful resources from previous years:

Course website http://pages.stat.wisc.edu/~sraschka/teaching/stat453-ss2020/

GitHub repository https://github.com/rasbt/stat453-deep-learning-ss20

Open In Colab

How to Submit¶

  1. Run every cell top to bottom before exporting, so your output isn't stale:
    • Jupyter: Kernel → Restart & Run All
    • Colab: Runtime → Restart and run all
  2. Export this notebook to PDF:
    • Colab: File → Print → Save as PDF
    • Local Jupyter: File → Save and Export Notebook As → PDF, or from a terminal: jupyter nbconvert --to webpdf --allow-chromium-download STAT453_hw02.ipynb
  3. Check the PDF — make sure plots aren't cut off at page edges and every cell that should have output actually shows it.
  4. Name the file <netid>_hw2.pdf (e.g. blengerich_hw2.pdf).
  5. Submit the PDF on Canvas by the deadline listed on the homework page.

HW 2: Backprop and Convolutions, By Hand and by Autograd (70 pts)¶

This assignment has two parts, tied to lecture material from 9/22 through 9/29:

  • Part A (25 pts) asks you to implement the forward pass of a small multi-layer perceptron (MLP), then derive and implement its backward pass by hand (no autograd) — extending the single-layer gradient from Lecture 6/7 one layer deeper via the chain rule. You'll check your gradients against torch.autograd, then train the MLP on a dataset a single-layer perceptron (Lecture 4, HW1) cannot solve.
  • Part B (45 pts) does the same thing one level down: implement a 1D convolution's forward pass, then derive and implement its backward pass by hand — you'll discover for yourself why it turns out to involve a flipped kernel, the fact proven in Lecture 8 — and check it against autograd. You'll then build and train a real nn.Conv2d CNN on MNIST and compare its parameter count to an equivalent fully-connected layer.

Both parts follow the same pattern as lecture: derive/implement by hand, then confirm with autograd. This is the actual mechanism autograd is automating for you, at every depth.

You may use PyTorch freely in this assignment (unlike HW1). Cells marked # <your code> are the parts you need to fill in; A.3 and B.1–B.3 end with a small self-check cell using assert statements — if it runs without an AssertionError and prints looks good!, you're on the right track (though passing the self-checks does not guarantee full credit).

Important!¶

Due Friday, October 2 at 11:59 PM. The homework assignment should be submitted via Canvas.

See the submission checklist above. Please reach out to the TA (office hours Wed & Fri, 2:00–3:00 PM via Zoom) or the instructor if you need help.

0) Imports¶

No modification required. You should execute this code, but it is recommended not to make any alterations here.

In [ ]:
import torch
import torch.nn as nn
import torch.nn.functional as F
import matplotlib.pyplot as plt
%matplotlib inline

torch.manual_seed(453)

Part A: MLP Forward & Backward, By Hand and by Autograd (25 pts)¶

A.1) A dataset a single-layer network can't solve¶

Recall from Lecture 4: a single-layer perceptron can only learn a linear decision boundary, so it cannot separate XOR-style data. The cell below generates a small 2D binary classification dataset with exactly that structure: label = 1 if the two features have the same sign, label = 0 otherwise.

No modification required. Run this cell and look at the plot.

In [ ]:
def make_xor_dataset(n_per_quadrant=60, noise=0.35, seed=453):
    g = torch.Generator().manual_seed(seed)
    centers = torch.tensor([[1., 1.], [-1., -1.], [1., -1.], [-1., 1.]])
    labels_per_center = torch.tensor([1., 1., 0., 0.])
    X, y = [], []
    for center, label in zip(centers, labels_per_center):
        pts = center + noise * torch.randn(n_per_quadrant, 2, generator=g)
        X.append(pts)
        y.append(label.repeat(n_per_quadrant))
    X = torch.cat(X, dim=0)
    y = torch.cat(y, dim=0)
    shuffle_idx = torch.randperm(X.size(0), generator=g)
    return X[shuffle_idx], y[shuffle_idx]

X, y = make_xor_dataset()
n_train = int(0.7 * X.size(0))
X_train, X_test = X[:n_train], X[n_train:]
y_train, y_test = y[:n_train], y[n_train:]

plt.scatter(X_train[y_train == 0, 0], X_train[y_train == 0, 1], label='class 0')
plt.scatter(X_train[y_train == 1, 0], X_train[y_train == 1, 1], label='class 1')
plt.legend()
plt.title('Training data (not linearly separable)')
plt.show()

A.2) Forward pass (5 pts)¶

Complete forward() below for a 2-layer MLP: one hidden layer with a ReLU activation, one output unit with a sigmoid activation — matching the notation from Lecture 7:

$$z_1 = W_1 x + b_1, \quad h = \text{ReLU}(z_1), \quad z_2 = W_2 h + b_2, \quad \hat{y} = \sigma(z_2)$$

Each column of x is one example (feature dimension first), so W1 has shape (n_hidden, n_features) and W2 has shape (1, n_hidden).

In [ ]:
class MLP():
    def __init__(self, n_features, n_hidden):
        self.W1 = torch.randn(n_hidden, n_features) * 0.5
        self.b1 = torch.zeros(n_hidden, 1)
        self.W2 = torch.randn(1, n_hidden) * 0.5
        self.b2 = torch.zeros(1, 1)

    def forward(self, x):
        # x: (n_features, batch)
        z1 = # <your code>
        h = # <your code>          # ReLU
        z2 = # <your code>
        yhat = # <your code>       # sigmoid
        self.cache = (x, z1, h, z2, yhat)
        return yhat

A.3a) Derive delta_1, the hidden-layer gradient (7 pts)¶

Let $L$ be the binary cross-entropy loss. You already have $\delta_2 = \partial L/\partial z_2 = \hat{y} - y$ (established in Lecture 6/7 — no need to re-derive it). There is only one path from any hidden unit to the loss: $h \to z_2 \to \hat{y} \to L$. Using the chain rule, first write $\partial L/\partial h$ in terms of $\delta_2$ and $W_2$:

$$\frac{\partial L}{\partial h} = \frac{\partial L}{\partial z_2}\cdot\frac{\partial z_2}{\partial h} = \; ...$$

then push that through the ReLU activation to get $\delta_1 = \partial L/\partial z_1$:

$$\delta_1 = \frac{\partial L}{\partial h} \odot \frac{\partial h}{\partial z_1} = \; ...$$

Your derivation: (replace this text with your work and the final closed-form formulas for $\partial L/\partial h$ and $\delta_1$, in terms of $W_2$, $\delta_2$, and $z_1$)

A.3b) Implement the backward pass (8 pts)¶

Complete backward() using $\delta_2 = \hat{y}-y$, the $\delta_1$ you just derived, and the weight/bias-gradient matrix form from Lecture 7 ("Backprop: From L5's Sums to Matrix Form": $\partial L/\partial W_l = \delta_l\, a_{l-1}^\top$ and $\partial L/\partial b_l = \delta_l$ summed over the batch, where $a_0 = x$ and $a_1 = h$, applied at both layers) — no torch.autograd in this method.

(y should have shape (1, batch), matching yhat.)

In [ ]:
def backward(self, y):
    x, z1, h, z2, yhat = self.cache
    delta2 = # <your code>
    dW2 = # <your code>
    db2 = # <your code>

    relu_grad = # <your code>
    delta1 = # <your code>
    dW1 = # <your code>
    db1 = # <your code>

    return dW1, db1, dW2, db2

MLP.backward = backward

Note: delta2 = yhat - y (no 1/batch_size factor) is the gradient of the summed loss over the batch. That's consistent with the single-example formula from lecture: differentiating a sum of independent per-example losses just sums each example's gradient, so delta2, delta1, etc. are all per-example values stacked into a batch, not yet averaged. Batch-averaging happens later, at the parameter-update step in A.4, not inside backward() itself.

Run the self-check below before moving on — it compares your hand-computed gradients to torch.autograd on the same weights, input, and (summed) loss.

In [ ]:
# Mean BCE loss, for reporting/plotting in Section A.4 -- NOT what backward() differentiates.
def bce_loss(yhat, y):
    eps = 1e-7
    yhat = yhat.clamp(eps, 1 - eps)
    return -(y * torch.log(yhat) + (1 - y) * torch.log(1 - yhat)).mean()

model = MLP(n_features=2, n_hidden=4)
x_batch = X_train[:8].T          # (2, 8)
y_batch = y_train[:8].view(1, -1)

yhat = model.forward(x_batch)
dW1, db1, dW2, db2 = model.backward(y_batch)

# Same computation, but let autograd do the backward pass.
# Use the SUMMED loss here (not bce_loss's mean) to match what backward() differentiates.
W1a = model.W1.clone().requires_grad_(True)
b1a = model.b1.clone().requires_grad_(True)
W2a = model.W2.clone().requires_grad_(True)
b2a = model.b2.clone().requires_grad_(True)

z1a = W1a @ x_batch + b1a
ha = torch.clamp(z1a, min=0.0)
z2a = W2a @ ha + b2a
yhat_a = torch.sigmoid(z2a)
loss_a = -(y_batch * torch.log(yhat_a) + (1 - y_batch) * torch.log(1 - yhat_a)).sum()
loss_a.backward()

assert torch.allclose(dW1, W1a.grad, atol=1e-5), "dW1 doesn't match autograd"
assert torch.allclose(db1, b1a.grad, atol=1e-5), "db1 doesn't match autograd"
assert torch.allclose(dW2, W2a.grad, atol=1e-5), "dW2 doesn't match autograd"
assert torch.allclose(db2, b2a.grad, atol=1e-5), "db2 doesn't match autograd"
print('looks good!')

A.4) Train it, and explain what happened (5 pts)¶

No modification required for the training loop below — it uses your forward/backward in plain mini-batch gradient descent (no torch.optim, no torch.autograd).

In [ ]:
def train(model, X, y, epochs=300, lr=0.5, batch_size=16, seed=453):
    g = torch.Generator().manual_seed(seed)
    losses = []
    for epoch in range(epochs):
        idx = torch.randperm(X.size(0), generator=g)
        epoch_loss = 0.0
        for start in range(0, X.size(0), batch_size):
            batch_idx = idx[start:start + batch_size]
            x_batch = X[batch_idx].T
            y_batch = y[batch_idx].view(1, -1)

            yhat = model.forward(x_batch)
            epoch_loss += bce_loss(yhat, y_batch).item() * x_batch.size(1)

            dW1, db1, dW2, db2 = model.backward(y_batch)
            model.W1 -= lr * dW1 / x_batch.size(1)
            model.b1 -= lr * db1 / x_batch.size(1)
            model.W2 -= lr * dW2 / x_batch.size(1)
            model.b2 -= lr * db2 / x_batch.size(1)
        losses.append(epoch_loss / X.size(0))
    return losses

model = MLP(n_features=2, n_hidden=8)
losses = train(model, X_train, y_train)

plt.plot(losses)
plt.xlabel('Epoch')
plt.ylabel('Binary cross-entropy loss')
plt.show()

def accuracy(model, X, y):
    yhat = model.forward(X.T)
    preds = (yhat >= 0.5).float().view(-1)
    return (preds == y).float().mean().item()

print('Train accuracy: %.1f%%' % (100 * accuracy(model, X_train, y_train)))
print('Test accuracy:  %.1f%%' % (100 * accuracy(model, X_test, y_test)))

Your answer: In 2-3 sentences: why couldn't a single-layer perceptron (Lecture 4, HW1) or logistic regression unit (Lecture 5) solve this dataset, and what specifically about stacking a second layer (with a nonlinear activation in between) makes it solvable? (replace this text with your answer)

Part B: Convolutions, By Hand and by Autograd — Then a Real CNN (45 pts)¶

B.1) 1D convolution, forward pass (5 pts)¶

Complete conv1d_forward(a, w): a 1D cross-correlation (what frameworks call "convolution," as in Lecture 8) with stride 1 and no padding, for an input a of length n and kernel w of length k. The output has length n - k + 1:

$$z_i = \sum_{j=0}^{k-1} w_j \, a_{i+j}$$

In [ ]:
def conv1d_forward(a, w):
    n, k = a.shape[0], w.shape[0]
    out_len = # <your code>
    z = torch.zeros(out_len)
    for i in range(out_len):
        z[i] = # <your code>
    return z

B.2a) Derive dL/da_i as a sum over the z_k's it feeds into (5 pts)¶

Specialize B.1's general formula to a small concrete case, to make the pattern easy to see: kernel size 2 (weights $w_1, w_2$) and input $a$ of length 4 (using 1-indexing here for readability), which gives $z_k = w_1 a_k + w_2 a_{k+1}$ for $k=1,2,3$. Let $\delta_k := \partial L/\partial z_k$ denote the upstream gradient flowing into $z_k$ (a new use of $\delta$, local to Part B — not the MLP's $\delta_1, \delta_2$ from A.3). For each $a_i$ ($i=1,2,3,4$), list which $z_k$'s it appears in and with what coefficient, then write $\partial L/\partial a_i$ as a sum using the chain rule — the same "sum over every path" argument as Lecture 5's weight-sharing slide (a value that's reused in multiple places has a gradient that sums each place's contribution). Do this for all four of $a_1, a_2, a_3, a_4$.

Your derivation: (replace this text with, for each $a_i$: which $z_k$'s it appears in, and the resulting formula for $\partial L/\partial a_i$ in terms of $\delta_1, \delta_2, \delta_3, w_1, w_2$)

B.2b) Implement the backward pass w.r.t. the input (5 pts)¶

Complete conv1d_backward_input(delta, w, n) using the formulas you just derived (generalized from length-4 to arbitrary n). Hint: line up the coefficients across your four formulas above — the pattern is exactly a cross-correlation of zero-padded delta with the flipped kernel (pad delta with (kernel size − 1) zeros on each side, then slide the flipped kernel across it). This is the flipped-kernel result from Lecture 8 — you just re-derived why it's true.

In [ ]:
def conv1d_backward_input(delta, w, n):
    k = w.shape[0]
    w_flipped = # <your code>
    padded = torch.zeros(delta.shape[0] + 2 * (k - 1))
    padded[k - 1: k - 1 + delta.shape[0]] = delta
    da = torch.zeros(n)
    for i in range(n):
        da[i] = # <your code>
    return da

B.3a) 1D convolution, backward pass w.r.t. the kernel (5 pts)¶

Complete conv1d_backward_weight(delta, a, k) — the gradient with respect to the (shared, tied) kernel weights. Because the same k weights are reused at every output position, dL/dw_j sums each position's contribution — the same argument as Lecture 5's weight-sharing slide (recalled in Lecture 8):

$$\frac{\partial L}{\partial w_j} = \sum_i \delta_i \, a_{i+j}$$

Here $\delta_i = \partial L/\partial z_i$, and the sum runs over output positions $i$, 0-indexed as in B.1.

In [ ]:
def conv1d_backward_weight(delta, a, k):
    dw = torch.zeros(k)
    for j in range(k):
        dw[j] = # <your code>
    return dw

Run the self-check below — it compares your conv1d_forward/conv1d_backward_* to F.conv1d + torch.autograd on the same numbers.

In [ ]:
a = torch.tensor([1.0, 3.0, -2.0, 0.5, 2.0])
w = torch.tensor([2.0, -1.0, 0.5])
delta = torch.tensor([1.0, 0.5, -1.0])   # pretend upstream gradient, one per output position

z = conv1d_forward(a, w)
da = conv1d_backward_input(delta, w, n=a.shape[0])
dw = conv1d_backward_weight(delta, a, k=w.shape[0])

a_t = a.clone().requires_grad_(True)
w_t = w.clone().requires_grad_(True)
z_auto = F.conv1d(a_t.view(1, 1, -1), w_t.view(1, 1, -1)).view(-1)
z_auto.backward(delta)

assert torch.allclose(z, z_auto.detach(), atol=1e-5), "forward pass doesn't match F.conv1d"
assert torch.allclose(da, a_t.grad, atol=1e-5), "input gradient doesn't match autograd"
assert torch.allclose(dw, w_t.grad, atol=1e-5), "weight gradient doesn't match autograd"
print('looks good!')

B.3b) Why the flip? (5 pts)¶

Your answer: In 2-3 sentences, based on what you just derived in B.1-B.3: why does the backward pass need to flip the kernel relative to the forward pass? (replace this text with your answer)

B.4) A real CNN on MNIST (15 pts)¶

Now switch to actual PyTorch layers (nn.Conv2d, nn.MaxPool2d, nn.Linear) — you don't need to implement backward passes by hand anymore; loss.backward() does exactly what you did above, at every layer, automatically.

No modification required for loading the data (this downloads a small subset of MNIST).

In [ ]:
import torchvision
import torchvision.transforms as transforms

transform = transforms.ToTensor()
train_full = torchvision.datasets.MNIST(root='./data', train=True, download=True, transform=transform)
test_full = torchvision.datasets.MNIST(root='./data', train=False, download=True, transform=transform)

# Subset for a fast homework training loop
train_idx = torch.randperm(len(train_full), generator=torch.Generator().manual_seed(453))[:4000]
test_idx = torch.randperm(len(test_full), generator=torch.Generator().manual_seed(453))[:1000]
train_subset = torch.utils.data.Subset(train_full, train_idx)
test_subset = torch.utils.data.Subset(test_full, test_idx)

train_loader = torch.utils.data.DataLoader(train_subset, batch_size=64, shuffle=True)
test_loader = torch.utils.data.DataLoader(test_subset, batch_size=256, shuffle=False)

images, labels = next(iter(train_loader))
fig, axes = plt.subplots(1, 6, figsize=(9, 2))
for i, ax in enumerate(axes):
    ax.imshow(images[i, 0], cmap='gray')
    ax.set_title(labels[i].item())
    ax.axis('off')
plt.show()

Complete SmallCNN below: one nn.Conv2d layer (1 input channel, 8 output channels, kernel size 3), a ReLU, a nn.MaxPool2d(2), then a nn.Linear layer mapping the flattened features to 10 class logits.

(A 28x28 input, conv with kernel 3 and no padding, gives 26x26 feature maps; after 2x2 max-pooling that's 13x13. So the flattened size going into the linear layer is 8 * 13 * 13.)

In [ ]:
class SmallCNN(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = # <your code>
        self.pool = # <your code>
        self.fc = # <your code>

    def forward(self, x):
        x = # <your code>       # conv, then ReLU
        x = # <your code>       # pool
        x = x.view(x.size(0), -1)
        return self.fc(x)

No modification required for training and evaluation.

In [ ]:
cnn = SmallCNN()
optimizer = torch.optim.Adam(cnn.parameters(), lr=1e-3)

for epoch in range(3):
    cnn.train()
    for images, labels in train_loader:
        optimizer.zero_grad()
        logits = cnn(images)
        loss = F.cross_entropy(logits, labels)
        loss.backward()
        optimizer.step()

    cnn.eval()
    correct, total = 0, 0
    with torch.no_grad():
        for images, labels in test_loader:
            preds = cnn(images).argmax(dim=1)
            correct += (preds == labels).sum().item()
            total += labels.size(0)
    print(f'epoch {epoch + 1}: test accuracy = {100 * correct / total:.1f}%')

B.5) Why not just use a fully-connected layer? (5 pts)¶

Early in Lecture 8: a 3×200×200 image implies 120,000 weights per neuron in a fully-connected first hidden layer. Let's do the analogous count for your own network.

No modification required for the parameter count below — just run it and answer the question.

In [ ]:
conv_params = sum(p.numel() for p in cnn.conv1.parameters())
n_output_units = 8 * 26 * 26  # cnn.conv1's output, before pooling

# An equally-wide FULLY-CONNECTED layer mapping the same 28x28x1 input
# to the same number of output units:
fc_equivalent_params = (28 * 28 * 1) * n_output_units + n_output_units

print(f'conv1 parameters:                            {conv_params:,}')
print(f'equivalent fully-connected layer would need: {fc_equivalent_params:,}')
print(f'ratio: {fc_equivalent_params / conv_params:,.0f}x more parameters')

Your answer: In 1-2 sentences: looking at how conv1 actually computes its output (one small filter, reused at every position, each output only seeing a small patch of the input) — what about that structure makes the parameter count so much smaller than the fully-connected alternative? (replace this text with your answer)

Attribution: Ben Lengerich