No new material was taught for the midterm session, so instead of scribing one lecture, these notes collect the essentials of Lectures 1 to 11, which the midterm covers, drawing on the slides and the readings .
How to Read These Notes
Each section gives the main idea of one lecture, the formulas with the shape of every object, and the easiest mistakes to make.
Notation. As on the slides, n is the number of examples and m the number of features, so the design matrix \mathbf{X}\in\mathbb{R}^{n\times m} has one example per row. A superscript [i] indexes examples, not powers. In Lecture 7, n_l is the width of layer l. “L7 s.22” means slide 22 of Lecture 7.
Lecture 1: What Is Machine Learning?
In traditional programming a person writes the program; in machine learning the computer gets inputs and outputs and produces the program, a function with learned parameters. In Mitchell’s definition, a program learns from experience \mathcal{E} on task \mathcal{T} if its performance \mathcal{P} on \mathcal{T} improves with \mathcal{E}, where \mathcal{P} is measured on unseen data. On the slides, ML and AI only partly overlap (Deep Blue’s search is AI without ML), while deep learning lies inside ML: many-layered networks that learn their own representations.
Three fundamental questions. For n binary variables, representation by a full joint distribution needs 2^n states and 2^n-1 free parameters (256 states for n=8). Inference such as P(X_8\mid X_1)=P(X_1,X_8)/P(X_1) needs a sum over the 2^6 settings of X_2,\ldots,X_7 (marginalization). Learning picks M=\mathrm{argmax}_{M\in\mathcal{H}}F(D;M), the model in the hypothesis space with the best score; the argmax returns the model, not the score. A graphical model factorizes the joint as \prod_i P(X_i\mid \mathrm{Pa}(X_i)), which needs \sum_i 2^{\lvert \mathrm{Pa}(X_i)\rvert} parameters: 18 instead of 255 in the lecture’s example.
Kinds of learning. Supervised: labeled pairs, for regression (h:\mathbb{R}^m\to\mathbb{R}) or classification (h:\mathbb{R}^m\to\{1,\ldots,k\}). Unsupervised: no labels; find structure (clustering, PCA, autoencoders). Reinforcement: learn a policy \pi:S\to A from rewards that only say how good an action was, often with a delay, so a reward is not a label.
Semi-supervised: few labels, many unlabeled examples. Self-supervised: labels built from the data itself, such as the next word. A batch of images has shape (N,C,H,W), such as [128,1,28,28] for 128 MNIST digits.
Lecture 2: A Brief History of Deep Learning
| Year |
Milestone |
Why it matters |
| 1943 |
McCulloch–Pitts neuron |
Threshold unit with fixed \pm 1 weights: AND, OR, NOT, but no learning and no XOR |
| 1957 |
Rosenblatt’s perceptron |
First model that learns its weights from data |
| 1969 |
Minsky and Papert |
One layer cannot represent XOR; the first “winter” follows |
| 1986 |
Backpropagation |
Rumelhart, Hinton and Williams: efficient gradients for multilayer networks |
| 1998 |
LeNet-5 |
CNN for digit recognition |
| 2012 |
AlexNet |
ImageNet top-5 error 15.4% vs. 26.2% for the runner-up |
Why one perceptron cannot do XOR. We need b\lt 0 for (0,0), w_1+b\ge 0 and w_2+b\ge 0 for (1,0) and (0,1), and w_1+w_2+b\lt 0 for (1,1). Adding the middle two gives w_1+w_2+b\ge -b\gt 0, a contradiction. A monotone activation such as the sigmoid does not help, since the boundary is still a line, but a hidden layer does: A\oplus B=\bar A B + A\bar B is a two-layer circuit.
Two facts recur all semester. Without a nonlinearity, layers collapse, W_2(W_1\mathbf{x}+\mathbf{b}_1)+\mathbf{b}_2=(W_2W_1)\mathbf{x}+(W_2\mathbf{b}_1+\mathbf{b}_2). And backpropagation only computes gradients, while gradient descent updates the parameters. Deep learning is representation learning with many levels (pixels, edges, parts, objects); it took off after 2012 thanks to large datasets and GPUs.
Lecture 3: Linear Algebra, Probability, and Estimation
Tensors. The rank of a tensor is its number of indices (.ndim); PyTorch defaults to float32, NumPy to float64. A neuron computes z=\mathbf{w}^\top\mathbf{x}+b with \mathbf{w},\mathbf{x}\in\mathbb{R}^{m}, and a batch computes \mathbf{z}=\mathbf{X}\mathbf{w}+\mathbf{1}_n b\in\mathbb{R}^{n}, written \mathbf{X}\mathbf{w}+b thanks to broadcasting. Broadcasting fails silently: shape (5,) minus shape (5,1) gives (5,5). In PyTorch * is elementwise and @ is matrix multiplication.
| Fact |
Formula |
Remember |
| Bernoulli |
P(x)=\theta^x(1-\theta)^{1-x}, mean \theta, variance \theta(1-\theta) |
x\in\{0,1\} |
| Gaussian |
f(x)=\frac{1}{\sqrt{2\pi\sigma^2}}\exp\big(-\frac{(x-\mu)^2}{2\sigma^2}\big) |
Density can exceed 1; L3 s.28 drops the minus sign |
| Variance |
\mathrm{Var}(X)=E[X^2]-E[X]^2, \mathrm{Var}(aX+b)=a^2\mathrm{Var}(X) |
E[g(X)]\ne g(E[X]) in general |
| Linearity |
E[X_1+X_2]=E[X_1]+E[X_2] |
Holds without independence; variances add only if the covariance is 0 |
| Covariance |
\mathrm{Cov}(X,Y)=E[XY]-E[X]E[Y] |
Independence gives zero covariance, not the reverse: X\sim U(-1,1), Y=X^2 |
| Bayes |
P(A\mid B)=P(B\mid A)P(A)/P(B) |
Posterior is proportional to likelihood times prior |
The prior matters: with 1% prevalence, a test that detects 99% of cases with a 5% false positive rate gives P(\text{disease}\mid +)=\frac{0.99\cdot 0.01}{0.99\cdot 0.01+0.05\cdot 0.99}\approx 0.17.
Estimation. The MLE is \hat\theta=\mathrm{argmax}_\theta P(D\mid\theta). For k ones in n Bernoulli draws, \ell(\theta)=k\log\theta+(n-k)\log(1-\theta), and \ell'(\theta)=\frac{k}{\theta}-\frac{n-k}{1-\theta}=0 gives \hat\theta=k/n (0.6 for the data 1, 0, 1, 1, 0). The log does not move the argmax; it turns products into sums and avoids underflow.
The MAP estimate is \mathrm{argmax}_\theta P(D\mid\theta)P(\theta): a uniform prior gives the MLE, and a prior P(\theta)\propto e^{-\lambda R(\theta)} gives “log-likelihood minus a penalty”, so regularization is MAP.
Linear regression. With \mathbf{y}=\mathbf{X}\beta+\epsilon and \epsilon_i\sim\mathcal{N}(0,\sigma^2), the MLE is least squares, \hat{\beta}=(\mathbf{X}^\top\mathbf{X})^{-1}\mathbf{X}^\top\mathbf{y}. Ridge (L2) is MAP with a Gaussian prior and LASSO (L1) is MAP with a Laplace prior.
Lecture 4: Single-Layer Networks and the Perceptron
A unit computes the net input z=\mathbf{w}^\top\mathbf{x}+b, the activation a=\sigma(z), and the label \hat y=f(a). A threshold \theta is a bias b=-\theta, which can be folded into the weights with a constant input x_0=1. The perceptron rule, with \hat y=\mathbb{1}[\mathbf{w}^\top\mathbf{x}\gt 0] and the bias inside \mathbf{w}, visits one example at a time:
\text{err}=y^{[i]}-\hat y^{[i]}\in\{-1,0,1\},\qquad \mathbf{w}:=\mathbf{w}+\text{err}\cdot\mathbf{x}^{[i]}.
A missed positive adds \mathbf{x} and a false positive subtracts it. Convergence is guaranteed only for linearly separable data; on XOR the rule cycles forever, and on separable data it stops at the first separating line, which may generalize poorly.
Geometry. Since \mathbf{w}^\top\mathbf{x}=\lVert\mathbf{w}\rVert\lVert\mathbf{x}\rVert\cos\theta, the prediction is 1 when the angle is below 90°. So \mathbf{w} is perpendicular to the decision boundary and points to the positive side, the bias shifts the boundary, and scaling \mathbf{w} by c\gt 0 changes no prediction.
The perceptron is not an MLE: a hard threshold defines no likelihood. With a sigmoid read as P(y=1\mid\mathbf{x}) we get logistic regression, whose log-likelihood gradient \sum_i(y_i-\sigma(\mathbf{w}^\top\mathbf{x}_i))\mathbf{x}_i has the perceptron’s form but no closed-form root.
A fully-connected layer with h outputs is \mathbf{a}=\sigma(\mathbf{W}\mathbf{x}+\mathbf{b})\in\mathbb{R}^{h}, \mathbf{W}\in\mathbb{R}^{h\times m}, where w_{i,j} connects input j to output i. With examples in rows, \mathbf{A}=\sigma(\mathbf{X}\mathbf{W}^\top+\mathbf{b}^\top)\in\mathbb{R}^{n\times h}, using (\mathbf{A}\mathbf{B})^\top=\mathbf{B}^\top\mathbf{A}^\top. This is nn.Linear: weight has shape (out_features, in_features), bias has shape (out_features,), and it returns X @ W.T + b.
Lecture 5: Parameter Optimization and Gradient Descent
A step function has zero gradient almost everywhere, so it cannot guide learning. The sigmoid \sigma(z)=1/(1+e^{-z}) is smooth, with \sigma'(z)=\sigma(z)(1-\sigma(z))\le 0.25 (maximum at z=0). Reading a=\sigma(z) as a Bernoulli probability, the negative log-likelihood of one example is the binary cross-entropy \mathcal{L}=-[y\log a+(1-y)\log(1-a)], and the chain rule gives
\frac{\partial\mathcal{L}}{\partial w_j}=\underbrace{\frac{a-y}{a(1-a)}}_{\partial\mathcal{L}/\partial a}\cdot\underbrace{a(1-a)}_{da/dz}\cdot\underbrace{x_j}_{\partial z/\partial w_j}=(a-y)\,x_j.
Likewise \partial\mathcal{L}/\partial b=a-y: the step is proportional to the error. With squared error the factor a(1-a) survives, so a saturated unit that is confidently wrong barely learns. Example: \mathbf{x}=(1,2), \mathbf{w}=\mathbf{0}, b=0, y=1 give a=0.5 and gradient (-0.5,-1); one step with \eta=0.1 gives \mathbf{w}=(0.05,0.1).
Gradient descent moves against the gradient, \theta\leftarrow\theta-\eta\nabla_{\theta}\mathcal{L}. A small \eta is slow; a large one oscillates or diverges (on f(x)=x^2, x\leftarrow(1-2\eta)x diverges for \eta\gt 1). A minibatch gradient is an unbiased but noisy estimate of the full-batch gradient. Critical points are minima, maxima or saddle points; logistic regression is convex, deep networks are not. The softmax e^{z_k}/\sum_j e^{z_j} is computed on \mathbf{z}-\max_j z_j to avoid overflow.
Chain rule on a computation graph. To get \partial l/\partial w, multiply local derivatives along each path from w to the loss, then add over all paths; several paths appear when a value fans out (a fully-connected layer) or a weight is shared. For ReLU the local derivative is \mathbb{1}[v\gt 0], so no gradient passes a unit with negative input. Backpropagation applies this rule in reverse order, caching intermediate results.
Lecture 6: Automatic Differentiation with PyTorch
Numerical differentiation, (f(\mathbf{x}+h\mathbf{e}_i)-f(\mathbf{x}))/h, has O(h) error (O(h^2) for central differences) and needs O(n) evaluations, so it is only for gradient checks. Symbolic differentiation builds expressions that blow up. Automatic differentiation applies exact local rules along the recorded computation . Forward mode gives a Jacobian-vector product J\mathbf{r} per pass; reverse mode gives J^\top\mathbf{r}, so the whole gradient of a scalar loss takes one backward pass. Backpropagation is reverse-mode AD.
PyTorch builds the graph while the forward code runs: tensors with requires_grad=True are tracked, each result stores a grad_fn, and loss.backward() adds gradients into .grad. Define an nn.Module (layers in __init__, computation in forward), create the model and optimizer, then train and evaluate:
import torch
import torch.nn.functional as F
from torch import optim
opt = optim.SGD(model.parameters(), lr=0.1)
for x, y in train_loader:
loss = F.cross_entropy(model(x), y)
opt.zero_grad() # .grad accumulates
loss.backward() # reverse-mode AD
opt.step() # w := w - lr * grad
model.eval() # dropout/BN: eval mode
with torch.no_grad(): # build no graph
pred = model(x_val).argmax(dim=1)
Losses take logits: F.cross_entropy applies log-softmax itself, and BCEWithLogitsLoss is the stable form of sigmoid plus BCE. The output gradients are \hat y-y for sigmoid with BCE and \mathrm{softmax}(\mathbf{z})_k-\mathbb{1}[k=c] for softmax with cross-entropy. model.eval() changes dropout and batch norm but still builds graphs; torch.no_grad() stops that.
Lecture 7: Multilayer Perceptrons and Backpropagation
From two classes to K. A categorical model uses the softmax \hat y_k=e^{z_k}/\sum_j e^{z_j}. With a one-hot label, the cross-entropy is L=-\sum_k y_k\log\hat y_k=-\log\hat y_c, and again \partial L/\partial z_k=\hat y_k-y_k. The sigmoid is the two-class softmax: \mathrm{softmax}(z_1,z_2)_1=\sigma(z_1-z_2). Do not apply a softmax before nn.CrossEntropyLoss.
The MLP. Layer l computes \mathbf{z}^{(l)}=W^{(l)}\mathbf{a}^{(l-1)}+\mathbf{b}^{(l)}, \mathbf{a}^{(l)}=\sigma(\mathbf{z}^{(l)}), with W^{(l)}\in\mathbb{R}^{n_l\times n_{l-1}} and \sum_l(n_{l-1}+1)\,n_l parameters in total (784-100-10: 785\cdot 100+101\cdot 10=79{,}510). With nonlinear activations, the hidden layer maps the inputs to a space where the classes are linearly separable, which solves XOR.
| Activation |
Range |
Derivative |
Notes |
| Sigmoid |
(0,1) |
\sigma(1-\sigma)\le 0.25 |
Saturates; not zero-centered |
| Tanh |
(-1,1) |
1-\tanh^2\le 1 |
Zero-centered; still saturates |
| ReLU |
[0,\infty) |
0 or 1 |
Cheap; units can “die” |
| Leaky ReLU, PReLU |
\mathbb{R} |
\alpha or 1 |
Leaky ReLU fixes \alpha; PReLU learns it |
The MLP loss is not convex, so results depend on the initialization and the seed. All-zero (or all-equal) weights give every unit in a layer the same gradient, so the units stay identical; random initialization breaks the symmetry.
Backpropagation in matrix form, for one example, with \delta^{(l)}=\partial L/\partial\mathbf{z}^{(l)}\in\mathbb{R}^{n_l}:
\begin{aligned}
\delta^{(L)}&=\hat{\mathbf{y}}-\mathbf{y},\qquad \delta^{(l)}=\big(W^{(l+1)}\big)^\top\delta^{(l+1)}\odot\sigma'\big(\mathbf{z}^{(l)}\big),\\
\frac{\partial L}{\partial W^{(l)}}&=\delta^{(l)}\big(\mathbf{a}^{(l-1)}\big)^\top,\qquad \frac{\partial L}{\partial\mathbf{b}^{(l)}}=\delta^{(l)}.
\end{aligned}
The error flows back through W^\top, not W (an n_l\times n_{l+1} matrix maps an n_{l+1}-vector to an n_l-vector), and the weight gradient is an outer product shaped like W^{(l)}. In the lecture’s worked example, a ReLU unit received an upstream gradient of 0.755 but had z\le 0, so its gradient was 0. Unrolled, \delta^{(1)}=D^{(1)}W^{(2)\top}\cdots D^{(L-1)}W^{(L)\top}\delta^{(L)} with D^{(l)}=\mathrm{diag}(\sigma'(\mathbf{z}^{(l)})); a long product shrinks or grows exponentially (with sigmoids, 0.25^{10}\approx 10^{-6}). This is the vanishing or exploding gradient problem.
Practice. The universal approximation theorem only says a wide one-hidden-layer network exists; deep networks often need far fewer parameters.
Lecture 8: Convolutional Neural Networks
A fully-connected neuron on a 3\times 200\times 200 image needs 120,000 weights and must relearn each pattern at every position. CNNs use sparse connectivity (each output sees a small window, its receptive field, which grows with depth) and parameter sharing (one kernel slides over all positions).
\begin{aligned}
O&=\left\lfloor\frac{W-K+2P}{S}\right\rfloor+1,\\
\#\text{params}&=C_{\text{out}}\,C_{\text{in}}\,K_H\,K_W+C_{\text{out}}.
\end{aligned}
Here W is the input width, K the kernel size, P the padding per side, and S the stride; pooling uses the same formula. For example, W=5,K=3,P=0,S=1 gives O=3, and P=(K-1)/2 with S=1 keeps the size.
Weights have shape (C_{\text{out}},C_{\text{in}},K_H,K_W), so the count does not depend on the image size. Each kernel spans all input channels and sums them into one feature map, so there is one output channel per kernel: 64 kernels of 3\times 3 on RGB have 64\cdot 27+64=1{,}792 parameters. Pooling has no parameters, works per channel, and lowers resolution without changing the channel count.
Convolution versus cross-correlation. The name comes from probability: for independent variables, P_{X+Y}(z)=\sum_x P_X(x)P_Y(z-x). In 2D, convolution is Z[i,j]=\sum_{u,v}K[u,v]A[i-u,j-v] (flipped kernel) and cross-correlation is Z[i,j]=\sum_{u,v}K[u,v]A[i+u,j+v]. nn.Conv2d computes cross-correlation; since the kernel is learned, this only matters for hand computations with an asymmetric kernel. Convolution is translation equivariant (shift the input and the feature map shifts); pooling adds only approximate invariance to small shifts, and neither handles rotation or scaling.
Backpropagation in CNNs is the multivariable chain rule: a shared weight’s gradient is the sum over all positions where it is used. In 1D with z_k=w_1a_k+w_2a_{k+1}, a_2 feeds z_1 and z_2, so dL/da_2=w_2\delta_1+w_1\delta_2. In general the input gradient is the zero-padded \delta cross-correlated with the flipped kernel (a true convolution), and the kernel gradient is the input cross-correlated with \delta. LeNet-5 (convolution and subsampling layers, then fully-connected layers) has about 60,000 parameters.
Lecture 9: Regularization
Diagnose first. For squared loss, expected test error is \text{bias}^2+\text{variance}+\sigma^2. Low training accuracy means high bias (underfitting): add capacity or weaken regularization. A large train-test gap means high variance (overfitting); every tool below trades a little bias for much less variance. A big model at 99% train and 71% test accuracy beats a small one at 60% and 58%: first overfit, then regularize.
Data and capacity. More high-quality data is the best fix; augmentation adds pairs (h(\mathbf{x}),y) for transformations h that keep the label (rotating a “6” by 180° does not), and it is applied only to training data. Early stopping keeps the weights from the epoch with the best validation score and acts as an implicit regularizer. Hyperparameters, including the stopping point, are chosen on the validation set; the test set is used once.
Norm penalties are MAP: a Gaussian prior on the weights gives L2 (ridge), a Laplace prior gives L1 (LASSO):
\text{Cost}=\frac1n\sum_{i=1}^{n}\mathcal{L}^{[i]}+\frac{\lambda}{n}\lVert\mathbf{w}\rVert_2^2\ \ (\text{L2})\quad\text{or}\quad+\frac{\lambda}{n}\lVert\mathbf{w}\rVert_1\ \ (\text{L1}).
With \lambda/n, the prior matters less as data grow. Biases are usually not penalized; deep networks sum \lVert W^{(l)}\rVert_F^2 over layers. L2 shrinks weights proportionally and rarely to exactly 0; L1 soft-thresholds and zeroes many, because the corners of the L1 diamond lie on the axes (L1 is rarely used in deep learning). Large \lambda means high bias. For gradient descent, L2 becomes weight decay:
w:=\Big(1-\frac{2\eta\lambda}{n}\Big)w-\eta\frac{\partial\mathcal{L}}{\partial w}.
PyTorch’s weight_decay=wd corresponds to adding \frac{\mathrm{wd}}{2}\lVert\mathbf{w}\rVert^2 and also decays biases by default. For Adam, an L2 penalty is not weight decay, hence AdamW.
Dropout zeroes each hidden unit with probability p during training: m_i\sim\text{Bernoulli}(1-p), \tilde a_i=m_ia_i, so E[\tilde a_i]=(1-p)a_i, and the original method multiplies by 1-p at test time. Inverted dropout (PyTorch) instead uses \tilde a_i=m_ia_i/(1-p) in training and does nothing at test time.
PyTorch’s p is the probability of dropping; the paper’s p is the probability of keeping. Why it works: units cannot co-adapt; it trains an implicit ensemble of 2^h weight-sharing subnetworks; and it suppresses high-order interactions. model.train() turns dropout on and model.eval() off; a higher, noisier training loss is expected.
Lecture 10: Normalization and Initialization
Normalization controls the scale of signals so that deep networks train at all. Inputs are standardized, x'=(x-\mu)/\sigma, with training-set statistics, but this fixes only the first layer. Batch normalization does it inside the network, for each unit j over the minibatch:
\begin{aligned}
\mu_j&=\frac1n\sum_i z_j^{[i]}, & \sigma_j^2&=\frac1n\sum_i\big(z_j^{[i]}-\mu_j\big)^2,\\
z_j'^{[i]}&=\frac{z_j^{[i]}-\mu_j}{\sqrt{\sigma_j^2+\epsilon}}, & a_j'^{[i]}&=\gamma_j z_j'^{[i]}+\beta_j.
\end{aligned}
The learnable \gamma,\beta (one per unit) restore any mean and scale; \gamma=\sigma, \beta=\mu gives the identity. BN goes after the linear layer and before the activation in the original paper, and centering cancels that layer’s bias (bias=False).
In backprop, \partial l/\partial\beta_j=\sum_i\partial l/\partial a_j'^{[i]}, \partial l/\partial\gamma_j=\sum_i(\partial l/\partial a_j'^{[i]})\,z_j'^{[i]}, and the gradient for z has three paths: direct, through \mu, and through \sigma^2. Training uses batch statistics; model.eval() switches to running averages, which PyTorch updates as \text{running}\leftarrow(1-m)\,\text{running}+m\cdot\text{batch} with m=0.1.L10 s.24 puts the momentum on the running value instead; PyTorch puts it on the new batch.
Why BN helps is debated: the original paper says it reduces internal covariate shift, but Santurkar et al. (2018) found it still helps when shift is injected on purpose, and argue that it smooths the loss landscape. With tiny batches (about 4 or fewer) batch statistics are too noisy; freeze BN or use a batch-independent norm. LayerNorm normalizes each example over its features (Transformers), InstanceNorm each (example, channel), and GroupNorm each (example, channel group).
Initialization. All-zero weights stay symmetric, so weights are small and random. For z=\sum_{k=1}^{m}w_ka_k with independent zero-mean terms, \mathrm{Var}(z)=m\,\mathrm{Var}(W)\,\mathrm{Var}(a), so \mathrm{Var}(W) must scale like 1/\text{fan\_in} to keep the variance constant across layers:
| Scheme |
\mathrm{Var}(W) |
Use with |
| Forward only |
1/\text{fan\_in} |
Activations near-linear around 0 |
| Xavier/Glorot |
2/(\text{fan\_in}+\text{fan\_out}) |
Tanh; balances forward and backward passes |
| He/Kaiming |
2/\text{fan\_in} |
ReLU, which zeroes half its inputs |
U(-r,r)
has variance r^2/3, so Xavier-uniform uses r=\sqrt{6/(\text{fan\_in}+\text{fan\_out})}.L10 s.46 says `nn.Linear` defaults to He; its default has variance 1/(3\,\text{fan\_in}), one sixth of He.
Lecture 11: Optimization and Learning Rates
In \mathbf{w}\leftarrow\mathbf{w}-\eta\mathbf{g} we can improve the step size \eta and the direction \mathbf{g}.
Step size. In the lecture, batch sizes 64 and 1024 reached about the same accuracy. Schedules include \eta_t=\eta_0e^{-kt}, halving every T_0 epochs, \eta_t=\eta_0/(1+kt), cyclical rates and cosine annealing, often after a linear warmup. Do not decay too early. A larger batch acts like a smaller learning rate (the slides write B\propto\epsilon, \epsilon the learning rate). To resume training, save the state_dict of the model, the optimizer and the scheduler.
Momentum averages past steps, so consistent directions add up and oscillations cancel. The two conventions differ by a factor of \eta:
\begin{aligned}
\text{slides:}&\quad \Delta w_t=\alpha\Delta w_{t-1}+\eta\,\frac{\partial\mathcal{L}}{\partial w}, & w&\leftarrow w-\Delta w_t,\\
\text{PyTorch:}&\quad v_t=\beta v_{t-1}+g_t, & w&\leftarrow w-\eta\, v_t.
\end{aligned}
A constant gradient gives v_t\to g/(1-\beta), ten times g at \beta=0.9; for g=2, v goes 2, 3.8, 5.42, toward 20. A gradient that flips sign every step settles near \pm g/(1+\beta)\approx\pm 0.53g. The averaging form v=\beta v+(1-\beta)g converges to g instead.
Nesterov evaluates the gradient at the look-ahead point \mathbf{w}_t-\alpha\Delta\mathbf{w}_{t-1}. Gradient clipping sets \mathbf{g}\leftarrow\mathbf{g}\,c/\lVert\mathbf{g}\rVert when \lVert\mathbf{g}\rVert\gt c: same direction, capped step, so it fights exploding but not vanishing gradients.
Per-parameter step sizes. Newton’s method \mathbf{w}\leftarrow\mathbf{w}-H^{-1}\nabla\mathcal{L} needs O(p^2) memory and O(p^3) time for p parameters, so adaptive methods approximate it elementwise. RMSProp: r_t=\rho r_{t-1}+(1-\rho)g_t^2, w\leftarrow w-\eta\,g_t/(\sqrt{r_t}+\epsilon). Adam adds momentum and corrects the bias from starting at zero:
\begin{aligned}
m_t&=\beta_1m_{t-1}+(1-\beta_1)g_t, & \hat m_t&=\frac{m_t}{1-\beta_1^t},\\
v_t&=\beta_2v_{t-1}+(1-\beta_2)g_t^2, & \hat v_t&=\frac{v_t}{1-\beta_2^t},\\
\theta_t&=\theta_{t-1}-\alpha\,\frac{\hat m_t}{\sqrt{\hat v_t}+\epsilon}.
\end{aligned}
Defaults: \alpha=10^{-3}, \beta_1=0.9, \beta_2=0.999, \epsilon=10^{-8}. At t=1, m_1=0.1g and v_1=0.001g^2, so without correction the first step would be 0.1/\sqrt{0.001}\approx 3.2 times \alpha; with it, the step is \alpha. On L11 s.37, \alpha,\beta name the two decay rates, while in the paper and PyTorch \alpha (lr) is the step size and betas=(0.9, 0.999) the decay rates. Adam is a good default, but tuned SGD with momentum can match it.
Putting It Together
One recipe for losses. Pick a distribution for y, write the likelihood, and minimize the negative log-likelihood. Each standard pairing gives “prediction minus target” at the output:
| Output |
Distribution |
Loss |
\partial\mathcal{L}/\partial z |
| Sigmoid |
Bernoulli |
Binary cross-entropy |
\hat y-y |
| Softmax |
Categorical |
Cross-entropy |
\hat{\mathbf{y}}-\mathbf{y} |
| Identity |
Gaussian |
Squared error, times \tfrac12 |
\hat y-y |
The same chain rule runs through L5 (path sums), L6 (autograd), L7 (the \delta recursion), L8 (convolutions) and L10 (batch norm), and regularization is MAP in L3, L4 and L9.
Values for computing by hand (no calculators at the exam):
| Quantity |
Value |
Quantity |
Value |
Quantity |
Value |
| \sigma(0) |
0.5 |
\sigma(\pm 1) |
0.731 / 0.269 |
\sigma(\pm 2) |
0.881 / 0.119 |
| e^{-1} |
0.368 |
e^{-2} |
0.135 |
\ln 2 |
0.693 |
| \ln 0.1 |
−2.303 |
\ln 0.9 |
−0.105 |
0.9^2,\ 0.9^3 |
0.81, 0.729 |
| 1/(1-0.9) |
10 |
1/(1+0.9) |
0.526 |
0.25^{10}=2^{-20} |
9.5\times 10^{-7} |