Lecture 06: Automatic Differentiation with PyTorch¶
Live walkthrough — September 22, 2026¶
We'll build up from a trivial scalar example, then go all the way from a distribution specification to a loss function to a hand derivation to what PyTorch's autograd actually computes — tying together Lecture 3 (MLE), Lecture 5 (logistic regression), and today.
import torch
import torch.nn as nn
torch.manual_seed(0)
<torch._C.Generator at 0x123769050>
1. Warm-up: a minimal scalar example¶
z = w * x + b. By hand: dz/dw = x, dz/db = 1. Let's check that autograd agrees.
x = torch.tensor(3.0)
w = torch.tensor(2.0, requires_grad=True)
b = torch.tensor(1.0, requires_grad=True)
z = w * x + b
z.backward()
print(f"z = {z.item()}")
print(f"dz/dw = {w.grad.item()} (hand-derived: x = {x.item()})")
print(f"dz/db = {b.grad.item()} (hand-derived: 1.0)")
z = 7.0 dz/dw = 3.0 (hand-derived: x = 3.0) dz/db = 1.0 (hand-derived: 1.0)
2. The full chain: distribution → likelihood → loss → derivation → autograd¶
2a. Specify the model and the distribution¶
For a single example $(x, y)$ with $y \in \{0, 1\}$: $$z = w^\top x + b, \qquad \hat y = \sigma(z), \qquad y \mid x \sim \text{Bernoulli}(\hat y).$$
PyTorch has a literal object for this: torch.distributions.Bernoulli.
x = torch.randn(4) # one example, 4 features
w = torch.randn(4, requires_grad=True)
b = torch.zeros(1, requires_grad=True)
y = torch.tensor([1.0]) # true label, shape (1,) to match z/yhat below
z = x @ w + b
z.retain_grad() # z is an intermediate tensor; ask autograd to keep its .grad
yhat = torch.sigmoid(z)
dist = torch.distributions.Bernoulli(probs=yhat)
print(f"z = {z.item():.4f}")
print(f"yhat = {yhat.item():.4f} (this IS the Bernoulli parameter above)")
z = -1.6633 yhat = 0.1593 (this IS the Bernoulli parameter above)
2b. Write the likelihood¶
By definition of the Bernoulli PMF: $$P(y \mid x) = \hat y^{\,y}(1-\hat y)^{1-y}.$$
Three ways to compute the same number — by hand, via the distribution object, and via its .log_prob:
likelihood_by_hand = yhat ** y * (1 - yhat) ** (1 - y)
likelihood_from_dist = dist.log_prob(y).exp()
print(f"likelihood, by hand = {likelihood_by_hand.item():.6f}")
print(f"likelihood, dist.log_prob(y).exp() = {likelihood_from_dist.item():.6f}")
likelihood, by hand = 0.159319 likelihood, dist.log_prob(y).exp() = 0.159319
2c. Take the negative log → the loss function¶
$$\log P(y\mid x) = y \log \hat y + (1-y)\log(1-\hat y)$$ $$\text{Loss} = -\log P(y \mid x) = -\big[y \log \hat y + (1-y)\log(1-\hat y)\big] \quad \text{(binary cross-entropy)}$$
Three more ways to compute the same loss — our own formula, PyTorch's distribution object, and PyTorch's built-in loss function:
loss_by_hand = -(y * torch.log(yhat) + (1 - y) * torch.log(1 - yhat))
loss_from_dist = -dist.log_prob(y)
loss_from_nn = nn.BCELoss()(yhat, y)
print(f"loss, by hand = {loss_by_hand.item():.6f}")
print(f"loss, -dist.log_prob = {loss_from_dist.item():.6f}")
print(f"loss, nn.BCELoss = {loss_from_nn.item():.6f}")
loss, by hand = 1.836847 loss, -dist.log_prob = 1.836847 loss, nn.BCELoss = 1.836847
2d. Derive the gradient by hand¶
Using the chain rule and $\frac{d\hat y}{dz} = \hat y(1-\hat y)$ (the sigmoid derivative): $$\frac{\partial \text{Loss}}{\partial \hat y} = -\frac{y}{\hat y} + \frac{1-y}{1-\hat y}$$ $$\frac{\partial \text{Loss}}{\partial z} = \frac{\partial \text{Loss}}{\partial \hat y}\cdot\frac{d\hat y}{dz} = \Big[-\frac{y}{\hat y} + \frac{1-y}{1-\hat y}\Big]\hat y (1-\hat y) = -y(1-\hat y) + (1-y)\hat y = \hat y - y.$$
No autograd yet — this is pure algebra.
2e. Check it against autograd¶
Now let's see if .backward() agrees with the algebra in 2d — using the loss we already built by hand in 2c.
loss_by_hand.backward()
print(f"autograd dL/dz = {z.grad.item():.6f}")
print(f"closed-form yhat-y = {(yhat - y).item():.6f}")
# -> these two numbers match: autograd re-derived Lecture 5 for us.
autograd dL/dz = -0.840681 closed-form yhat-y = -0.840681
2f. The version you'll actually use: BCEWithLogitsLoss¶
In practice we skip computing sigmoid and log separately — nn.BCEWithLogitsLoss takes the raw logit z directly and combines the sigmoid + log in a numerically stable way (it never actually computes log(0) even when z is very negative/positive). Same loss, same gradient, better numerics.
z2 = x @ w.detach().requires_grad_() + b.detach() # fresh leaf so its .grad starts at None
z2.retain_grad()
loss_stable = nn.BCEWithLogitsLoss()(z2, y)
loss_stable.backward()
print(f"loss (stable) = {loss_stable.item():.6f} (matches 2c)")
print(f"autograd dL/dz (stable) = {z2.grad.item():.6f} (matches 2d/2e)")
loss (stable) = 1.836847 (matches 2c) autograd dL/dz (stable) = -0.840681 (matches 2d/2e)
3. A classic gotcha: gradients accumulate¶
.backward() adds into .grad rather than overwriting it. If you forget to zero it out between steps, gradients from old iterations silently pile up.
w2 = torch.tensor(1.0, requires_grad=True)
for step in range(3):
loss = (w2 - 5) ** 2
loss.backward()
print(f"step {step}: w2.grad = {w2.grad.item()}")
# BUG: we never reset w2.grad, so it keeps growing across steps.
# Fix: uncomment the next line (or call optimizer.zero_grad() when using an optimizer).
w2.grad.zero_()
step 0: w2.grad = -8.0 step 1: w2.grad = -8.0 step 2: w2.grad = -8.0
4. From autograd to a training loop¶
The same three ingredients every time: zero the gradients, call backward(), step the optimizer. Note we use BCEWithLogitsLoss on raw logits, exactly as derived in 2f.
X = torch.randn(20, 4)
true_w = torch.tensor([2.0, -1.0, 0.5, 0.0])
y_data = (X @ true_w > 0).float()
model = nn.Linear(4, 1)
loss_fn = nn.BCEWithLogitsLoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.5)
for epoch in range(50):
optimizer.zero_grad()
logits = model(X).squeeze(1)
loss = loss_fn(logits, y_data)
loss.backward()
optimizer.step()
if epoch % 10 == 0:
print(f"epoch {epoch:2d}: loss = {loss.item():.4f}")
epoch 0: loss = 0.7837 epoch 10: loss = 0.2825 epoch 20: loss = 0.1987 epoch 30: loss = 0.1593 epoch 40: loss = 0.1356