Lecture 03 - Stats / Linear Algebra Review

The fundamental math skills for deep learning - tensors and PyTorch, vectors, matrices and broadcasting, probability basics, estimation (MLE and MAP), and linear regression.

1. Motivation: Why We Need This Math

Today’s roadmap is:

Tensors in deep learning
Tensors and PyTorch
Vectors, matrices, and broadcasting
Probability basics
Estimation methods
Linear regression

2. Tensors in Deep Learning

2.1 Scalars, vectors, and matrices

A scalar is an order-0 tensor, written with a lowercase symbol:

x \in \mathbb{R}, \qquad x = 1.23.

A vector is an order-1 tensor, written with a bold lowercase symbol:

\mathbf{x} \in \mathbb{R}^{n}.

In this lecture, vectors are assumed to be column vectors:

\mathbf{x} \in \mathbb{R}^{n\times 1}, \qquad \mathbf{x} = \begin{bmatrix} x_1\\ x_2\\ \vdots\\ x_n \end{bmatrix}.

Its transpose is a row vector:

\mathbf{x}^{\top} = [x_1\;x_2\;\cdots\;x_n] \in \mathbb{R}^{1\times n}.

A matrix is an order-2 tensor, written with a bold capital symbol:

\mathbf{X}\in\mathbb{R}^{m\times n}.

Each step up in tensor order adds another axis (dimension).

2.2 The design matrix convention

$\mathbf{X}$ denotes the design matrix (also called the feature matrix or input matrix), containing training examples and their features:

\mathbf{X}\in\mathbb{R}^{n\times m},

where $n$ is the number of training examples (rows) and $m$ is the number of features (columns).

A superscript in square brackets indexes the example and a subscript indexes the feature. For example,

x_2^{[1]}

is the second feature value of the first training example.

In lecture. Fluency with this notation matters as nearly every model in the course is expressed via the design matrix and operations on it.

2.3 Higher-order tensors

A 3D tensor (order-3 tensor) can be pictured as a stack of matrices:

\mathbf{X}\in\mathbb{R}^{m\times n\times p}.

Here $m,n,p$ denote the sizes of the three tensor dimensions.

In lecture. We usually say scalar, vector, or matrix for orders 0–2, and commonly use the word “tensor” once the order is 3 or higher, even though technically all of these objects are tensors.

A 3D tensor can represent a color image: three stacked matrices corresponding to the red, green, and blue channels, each storing one intensity per pixel.

A 4D tensor can represent a batch of images. Stacking 3D image tensors creates a 4D tensor whose first axis indexes the image in the batch.

2.4 For our purposes: tensor = multidimensional array

For this course, a tensor is a multidimensional array. The tensor’s dimensionality (or order) is the number of axes in its .shape.

import torch t = torch.tensor([[1, 2, 3], [4, 5, 6]]) t.shape # torch.Size([2, 3]) t.ndim # 2 -> an order-2 tensor (a matrix)

3. Tensors and PyTorch

3.1 NumPy arrays to PyTorch tensors

Both numpy.array / numpy.ndarray and torch.tensor / torch.Tensor are data-structure representations of tensors.

import numpy as np a = np.array([1., 2., 3.]) print(a.dtype) # float64 print(a.shape) # (3,) import torch b = torch.tensor([1., 2., 3.]) print(b.dtype) # torch.float32 print(b.shape) # torch.Size([3])

3.2 NumPy and PyTorch syntax is similar

a.dot(a) # 14.0 (NumPy) b.matmul(b) # tensor(14.) (PyTorch) b.dot(b) # tensor(14.) b @ b # tensor(14.) b.numpy() # array([1., 2., 3.], dtype=float32)

For two 1D vectors, matmul, dot, and @ all compute the inner product.`torch.dot` only accepts two 1D tensors, while `matmul` / `@` also handle matrix-vector, matrix-matrix, and batched products (with broadcasting).

3.3 Data types

NumPy data type	PyTorch tensor type	Note
`numpy.uint8`	`torch.ByteTensor`
`numpy.int16`	`torch.ShortTensor`
`numpy.int32`	`torch.IntTensor`
`numpy.int64`	`torch.LongTensor`	Default integer type in the lecture examples; `numpy.int` is a removed legacy alias
`numpy.float16`	`torch.HalfTensor`
`numpy.float32`	`torch.FloatTensor`	Default float in PyTorch
`numpy.float64`	`torch.DoubleTensor`	Default float in the lecture’s NumPy examples; `numpy.float` is a removed legacy alias

The dtype can be specified explicitly:

c = torch.tensor([1., 2., 3.], dtype=torch.float) # torch.float32 c = torch.tensor([1., 2., 3.], dtype=torch.double) # torch.float64 c = torch.tensor([1., 2., 3.], dtype=torch.float64) # torch.float64

In lecture. Dtypes can cause setup problems when operations mix incompatible types. Converting to lower precision throws information away, while higher precision consumes more memory.

A classic dtype bug is:

x = torch.from_numpy(np.array([1., 2.])) # keeps NumPy float64 x = x.float() # convert to float32 # or: x = x.to(torch.float32)

3.4 Why not just use NumPy?

PyTorch offers NumPy-like operations, but it is designed for deep learning:

GPU support: tensor operations can run on compatible GPUs.
Automatic differentiation (autograd): PyTorch records a computation graph and can traverse it in reverse to compute gradients automatically.
DL convenience functions: layers, loss functions, optimizers, data loaders, and related utilities.

3.5 Loading data onto a GPU

print(torch.cuda.is_available()) b = b.to(torch.device("cuda:0")) print(b) b = b.to(torch.device("cpu")) print(b)

If CUDA is installed, nvidia-smi can be used to inspect the available NVIDIA GPUs, including their memory usage, utilization, driver, and CUDA version.

3.6 Installing PyTorch

The lecture recommends using the selector on the PyTorch website to choose the operating system, package manager, language, and compute platform, then running the generated install command.The screenshot in the slide deck shows an older PyTorch selector. Use the current command generated by the PyTorch website rather than copying the old screenshot.

4. Vectors, Matrices, and Broadcasting

4.1 Vectors: the pre-activation of a neuron

For one observation,

\mathbf{w}^{\top}\mathbf{x} + b = z,

where $\mathbf{x}$ is the input feature vector, $\mathbf{w}$ is the weight vector, $b$ is the bias, and $z$ is the pre-activation (also called the net input).

Equivalently,

z = \sum_{j=1}^{m} w_jx_j + b.

4.2 Matrices: computing outputs for multiple examples

Stacking examples as rows of the design matrix lets us compute all pre-activations at once:

\mathbf{X}\mathbf{w} + b = \mathbf{z},

where

\mathbf{X}\in\mathbb{R}^{n\times m}, \qquad \mathbf{w}\in\mathbb{R}^{m\times 1}, \qquad \mathbf{z}\in\mathbb{R}^{n\times 1}.

Each entry of $\mathbf{z}$ is the pre-activation for one example.

In lecture: Big-O cost of matrix multiplication. For two $N\times N$ matrices, the standard matrix-multiplication algorithm is $O(N^3)$: there are $N^2$ output entries and each is a dot product of length $N$. Faster asymptotic algorithms exist, but in practice the lecture assumes approximately $O(N^3)$.Strassen's algorithm has complexity $O(N^{\log_2 7}) \approx O(N^{2.81})$. Faster theoretical algorithms exist but have large constants and are not the standard practical assumption for this course.

4.3 A common notational convenience

Strictly, $\mathbf{X}\mathbf{w}$ is an $n\times1$ vector and $b$ is a scalar, so the fully explicit expression is

\mathbf{X}\mathbf{w} + \mathbf{1}_n b = \mathbf{z}, \qquad \mathbf{1}_n=[1,1,\ldots,1]^{\top}\in\mathbb{R}^{n}.

In deep-learning notation, we typically write

\mathbf{X}\mathbf{w}+b=\mathbf{z}

and assume broadcasting.

4.4 Broadcasting

torch.tensor([1, 2, 3]) + 1 # tensor([2, 3, 4]) t = torch.tensor([[4, 5, 6], [7, 8, 9]]) t + torch.tensor([1, 2, 3]) # tensor([[ 5, 7, 9], # [ 8, 10, 12]])

Broadcasting can silently produce an unintended result when shapes are compatible, so tensor shapes are important when debugging.

y = torch.randn(5) # shape (5,) yhat = torch.randn(5, 1) # shape (5, 1) (y - yhat).shape # torch.Size([5, 5]) -- not (5,) # Possible fixes: yhat.squeeze(1) # or reshape y to (5, 1) if that is the intended layout

Beyond the slides: linear algebra facts used later.

Matrix product:

(AB)_{ij}=\sum_k A_{ik}B_{kj}, \qquad (p\times q)(q\times r)\rightarrow(p\times r).

Transpose:

(AB)^{\top}=B^{\top}A^{\top}, \qquad (A^{\top})^{\top}=A.

Norms:

\|\mathbf{x}\|_2=\left(\sum_jx_j^2\right)^{1/2}, \qquad \|\mathbf{x}\|_2^2=\mathbf{x}^{\top}\mathbf{x}, \qquad \|\mathbf{x}\|_1=\sum_j|x_j|.

Inverse:

A^{-1}A=I.

Gradients:

\nabla_{\mathbf{w}}(\mathbf{a}^{\top}\mathbf{w})=\mathbf{a},

and, for symmetric $A$,

\nabla_{\mathbf{w}}(\mathbf{w}^{\top}A\mathbf{w})=2A\mathbf{w}.

5. Probability Basics

5.1 Definitions

A discrete random variable takes values from a countable set, such as a coin flip.

A continuous random variable can take values over a continuous range, such as a height.

For discrete $X$, a probability mass function (PMF) is

P(X=x).

For continuous $X$, a probability density function (PDF) is

f(x).

Beyond the slides. A PMF gives actual point probabilities:

\sum_x P(X=x)=1.

A PDF gives a density. Probabilities come from areas:

P(a\leq X\leq b) = \int_a^b f(x)\,dx, \qquad \int f(x)\,dx = 1.

A density value $f(x)$ can be larger than 1, while for a continuous random variable $P(X=x)=0$ for any single point.

5.2 Key distributions

Distributions are parameterized; the lecture generally uses $\theta$ for parameters.

Bernoulli distribution:

P(X=x) = \theta^x(1-\theta)^{1-x}, \qquad x\in\{0,1\}.

For a fair coin, $\theta=0.5$. For Bernoulli $X$,

\theta=P(X=1), \qquad E[X]=\theta, \qquad \operatorname{Var}(X)=\theta(1-\theta).

Gaussian distribution: with mean $\mu$ and variance $\sigma^2$,The exponent on the lecture slide is missing its minus sign. Without the minus sign, the expression would not define a valid Gaussian density.

f(x) = \frac{1}{(2\pi\sigma^2)^{1/2}} \exp\left( -\frac{(x-\mu)^2}{2\sigma^2} \right).

The standard normal has

\mu=0,\qquad \sigma=1.

5.3 Central Limit Theorem (CLT)

Let $X_1,X_2,\ldots,X_n$ be i.i.d. random variables with mean $\mu$ and variance $\sigma^2$. Define the sample mean

\bar X_n = \frac{1}{n}\sum_{i=1}^{n}X_i.

Then, as $n\rightarrow\infty$,

\frac{\bar X_n-\mu}{\sigma/n^{1/2}} \rightarrow \mathcal{N}(0,1)

in distribution.

In lecture. Under the usual CLT assumptions, if we average many independent variables from the same distribution, subtract the mean, and divide by the standard error $\sigma/n^{1/2}$, the standardized average approaches a standard normal as $n$ becomes large.

**Figure 1. The CLT in action.** Each panel shows standardized sample means from a skewed Exponential(1) distribution. At small $n$ the shape remains skewed; by larger $n$ it is close to the standard normal density.

5.4 Joint, marginal, and conditional probabilities

Joint:

P(A,B),

the probability of two events occurring together.

Marginal:

P(A)=\sum_B P(A,B),

the sum of joint probabilities over one variable.

Conditional:

P(A\mid B) = \frac{P(A,B)}{P(B)},

the probability of $A$ given $B$.

Beyond the slides: rules that follow.

Product rule:

P(A,B) = P(A\mid B)P(B) = P(B\mid A)P(A).

Independence:

A\perp B \Longleftrightarrow P(A,B)=P(A)P(B) \Longleftrightarrow P(A\mid B)=P(A).

Law of total probability:

P(B) = \sum_a P(B\mid A=a)P(A=a).

5.5 Expectation and variance

Expectation:

E[X] = \sum_xxP(X=x) \qquad\text{(discrete)},

E[X] = \int xf(x)\,dx \qquad\text{(continuous)}.

Variance:

\operatorname{Var}(X) = E[(X-E[X])^2],

equivalently,

\operatorname{Var}(X) = E[X^2]-E[X]^2.

Beyond the slides: why the two variance formulas agree. Let $\mu=E[X]$. Expanding the square,

E[(X-\mu)^2] = E[X^2]-2\mu E[X]+\mu^2 = E[X^2]-\mu^2.

5.6 Linearity of expectation

For constants $a,b$,

E[aX+b] = aE[X]+b.

For multiple random variables,

E[X_1+X_2] = E[X_1]+E[X_2].

In lecture. Independence is not required for linearity of expectation. Independence matters for variance:

\operatorname{Var}(aX+b)=a^2\operatorname{Var}(X),

\operatorname{Var}(X_1+X_2) = \operatorname{Var}(X_1) + \operatorname{Var}(X_2) + 2\operatorname{Cov}(X_1,X_2).

Thus variances add when $\operatorname{Cov}(X_1,X_2)=0$, as for independent variables.

For the sample mean,

\operatorname{Var}(\bar X_n) = \frac{\sigma^2}{n},

which gives the standard error $\sigma/n^{1/2}$ used in the CLT.

5.7 Expectation of functions

For a function $g$,

E[g(X)] = \sum_xg(x)P(X=x) \qquad\text{(discrete)},

E[g(X)] = \int g(x)f(x)\,dx \qquad\text{(continuous)}.

Discrete example: if $X\sim\operatorname{Bernoulli}(\theta)$ and $g(X)=X^2$,

E[g(X)] = 1^2\theta+0^2(1-\theta) = \theta.

Continuous example: if $X\sim\operatorname{Uniform}(0,1)$ and $g(X)=X^2$,

E[g(X)] = \int_0^1x^2\,dx = \frac{1}{3}.

5.8 Variance of functions

\operatorname{Var}(g(X)) = E[(g(X)-E[g(X)])^2] = E[g(X)^2]-(E[g(X)])^2.

Beyond the slides: finishing the two examples.

For Bernoulli $X$, since $X^4=X$ when $X\in{0,1}$,

\operatorname{Var}(X^2) = \theta-\theta^2 = \theta(1-\theta).

For $X\sim\operatorname{Uniform}(0,1)$,

E[X^4] = \int_0^1x^4\,dx = \frac15,

\operatorname{Var}(X^2) = \frac15-\frac19 = \frac{4}{45} \approx0.089.

5.9 Covariance and correlation

Covariance:

\operatorname{Cov}(X,Y) = E[(X-E[X])(Y-E[Y])] = E[XY]-E[X]E[Y].

Properties:

\operatorname{Cov}(X,X)=\operatorname{Var}(X).

If $X,Y$ are independent,

\operatorname{Cov}(X,Y)=0.

Correlation:

\rho(X,Y) = \frac{\operatorname{Cov}(X,Y)} {\left(\operatorname{Var}(X)\operatorname{Var}(Y)\right)^{1/2}} \in[-1,1].

$\rho=1$: perfect positive linear relationship.
$\rho=0$: no linear correlation.
$\rho=-1$: perfect negative linear relationship.

Beyond the slides: the converse is false. Zero covariance does not imply independence. If $X\sim\operatorname{Uniform}(-1,1)$ and $Y=X^2$, then $Y$ is completely determined by $X$, yet

\operatorname{Cov}(X,Y) = E[X^3]-E[X]E[X^2] = 0.

Correlation detects linear dependence, not every kind of dependence.

5.10 Bayes’ rule

P(A\mid B) = \frac{P(B\mid A)P(A)}{P(B)} = \frac{\text{likelihood}\times\text{prior}}{\text{evidence}}.

For a medical test,

P(\text{disease}\mid\text{positive}) = \frac{ P(\text{positive}\mid\text{disease})P(\text{disease}) }{ P(\text{positive}) }.

Beyond the slides: plugging in numbers. Suppose 1% of people have the disease, the test catches 99% of cases, and the false-positive rate is 5%.

P(+) = 0.99(0.01)+0.05(0.99) = 0.0594.

Therefore,

P(D\mid+) = \frac{0.99\times0.01}{0.0594} \approx0.167.

The prior/base rate matters here: MAP incorporates prior information, while MLE uses the likelihood alone.

6. Estimation Methods

6.1 Introduction to estimation

The goal of estimation is to infer unknown parameters $\theta$ from observed data.

Types of estimation:

Point estimation: a single value, such as an MLE.
Interval estimation: a range of plausible values, such as a confidence interval.

Common methods:

Maximum Likelihood Estimation (MLE)
Maximum A Posteriori (MAP)
Method of Moments

Beyond the slides: the two methods not covered in detail.

For a large-sample mean, an approximate 95% confidence interval is

\bar x \pm 1.96\frac{s}{n^{1/2}},

using the sample standard deviation $s$ when $\sigma$ is unknown.

For the method of moments, set theoretical moments equal to sample moments and solve for the parameter. For Bernoulli data,

E[X]=\theta \quad\Rightarrow\quad \hat\theta_{\mathrm{MoM}}=\bar x.

6.2 Maximum Likelihood Estimation (MLE)

The MLE is

\hat{\theta}_{\mathrm{MLE}} = \arg\max_{\theta}L(\theta),

where $L(\theta)$ is the probability mass or probability density of the observed data given $\theta$, viewed as a function of $\theta$.

Bernoulli example

For the observed data

\{x_i\}_{i=1}^{5}=\{1,0,1,1,0\},

modeled as i.i.d. Bernoulli observations with $P(X=1\mid\theta)=\theta$,

L(\theta) = \prod_i \theta^{x_i}(1-\theta)^{1-x_i}.

The log-likelihood is

\ell(\theta) = \log L(\theta) = \sum_{i=1}^{n} \left[ x_i\log\theta + (1-x_i)\log(1-\theta) \right].

Let

k=\sum_i x_i.

Then

\ell(\theta) = k\log\theta+(n-k)\log(1-\theta),

and

\frac{d\ell}{d\theta} = \frac{k}{\theta} - \frac{n-k}{1-\theta}.

Setting the derivative to zero gives

\hat\theta_{\mathrm{MLE}} = \frac{k}{n}.

For the dataset above, $k=3$ and $n=5$, so

\hat\theta_{\mathrm{MLE}} = \frac35 = 0.6.

Beyond the slides: why take the log, and why it matters for DL.

log is strictly increasing, so it does not change the location of the maximum.
It converts products into sums, which are easier to differentiate.
Products of many probabilities can numerically underflow, while sums of log-probabilities are more stable.
For a Bernoulli model, minimizing negative log-likelihood gives binary cross-entropy. Many common deep-learning losses have negative-log-likelihood interpretations.

The lecture also notes that the MLE:

does not always exist,
is not necessarily unique,
is not necessarily admissible."Admissible" means that there is no other estimator with lower or equal risk for every parameter value and strictly lower risk for at least one.

6.3 Maximum A Posteriori (MAP) estimation

MAP selects the parameter that maximizes the posterior:

\hat\theta_{\mathrm{MAP}} = \arg\max_{\theta} P(\theta\mid\text{data}) = \arg\max_{\theta} P(\text{data}\mid\theta)P(\theta).

The evidence $P(\text{data})$ can be dropped from the argmax because it does not depend on $\theta$.

$P(\text{data}\mid\theta)$ is the likelihood.
$P(\theta)$ is the prior.

MLE ignores $P(\theta)$; MAP incorporates prior information.

Beyond the slides: MAP for the Bernoulli example. With a Beta$(\alpha,\beta)$ prior,

P(\theta) \propto \theta^{\alpha-1}(1-\theta)^{\beta-1}.

The posterior is proportional to

\theta^{k+\alpha-1}(1-\theta)^{n-k+\beta-1},

and, when the posterior has an interior mode, the MAP estimate is

\hat\theta_{\mathrm{MAP}} = \frac{k+\alpha-1}{n+\alpha+\beta-2}.

With Beta$(2,2)$ and $k=3,n=5$,

\hat\theta_{\mathrm{MAP}} = \frac47 \approx0.571.

**Figure 2. MLE vs. MAP for the dataset {1, 0, 1, 1, 0}.** The likelihood peaks at the MLE, while the Beta(2,2) prior shifts the posterior mode toward 0.5.

6.4 Regularization is MAP

A regularized maximum-likelihood objective adds a penalty:

\hat\theta_{\mathrm{reg}} = \arg\max_{\theta} \left[ \log L(\theta)-\lambda R(\theta) \right].

P(\theta)\propto e^{-\lambda R(\theta)},

then

\hat\theta_{\mathrm{MAP}} = \arg\max_{\theta} \left[ \log L(\theta)+\log P(\theta) \right] = \arg\max_{\theta} \left[ \log L(\theta)-\lambda R(\theta) \right].

Thus regularization can be interpreted as MAP estimation under a corresponding prior.

7. Linear Regression

7.1 Model definition

Linear regression is written as

\mathbf{y} = \mathbf{X}\boldsymbol{\beta} + \boldsymbol{\epsilon},

where

$\mathbf{y}$ is the response variable, $n\times1$;
$\mathbf{X}$ is the design matrix, $n\times m$;
$\boldsymbol{\beta}$ is the coefficient vector, $m\times1$;
$\boldsymbol{\epsilon}$ is the error term, often modeled with Gaussian noise.

The goal is to estimate $\boldsymbol{\beta}$. An intercept can be included by adding a column of ones to $\mathbf{X}$.

7.2 Evaluation metrics

Coefficient of determination:

R^2 = 1- \frac{SS_{\mathrm{residual}}}{SS_{\mathrm{total}}},

where

SS_{\mathrm{residual}} = \sum_i(y_i-\hat y_i)^2

and

SS_{\mathrm{total}} = \sum_i(y_i-\bar y)^2.

It measures the proportion of variance in $y$ explained by the model.

Mean Squared Error (MSE):

\mathrm{MSE} = \frac1n \sum_i(y_i-\hat y_i)^2.

Mean Absolute Error (MAE):The lecture slide labels the absolute-error formula "MSE"; the expression itself is the MAE.

\mathrm{MAE} = \frac1n \sum_i|y_i-\hat y_i|.

Beyond the slides. MSE penalizes large errors more heavily because errors are squared, while MAE is more robust to large outliers. RMSE,

\mathrm{RMSE}=(\mathrm{MSE})^{1/2},

has the same units as $y$.

7.3 Ordinary Least Squares (OLS)

The OLS objective is

\hat{\boldsymbol{\beta}}_{\mathrm{OLS}} = \arg\min_{\boldsymbol{\beta}} \|\mathbf{y}-\mathbf{X}\boldsymbol{\beta}\|_2^2.

Residuals are

e_i = y_i-\hat y_i.

When the inverse exists, the familiar closed-form expression is

\hat{\boldsymbol{\beta}}_{\mathrm{OLS}} = (\mathbf{X}^{\top}\mathbf{X})^{-1} \mathbf{X}^{\top}\mathbf{y}.

Beyond the slides: deriving the OLS solution.

Expand the objective:

\|\mathbf{y}-\mathbf{X}\boldsymbol{\beta}\|^2 = \mathbf{y}^{\top}\mathbf{y} - 2\boldsymbol{\beta}^{\top}\mathbf{X}^{\top}\mathbf{y} + \boldsymbol{\beta}^{\top}\mathbf{X}^{\top}\mathbf{X}\boldsymbol{\beta}.

Taking the gradient and setting it to zero gives the normal equations:

-2\mathbf{X}^{\top}\mathbf{y} + 2\mathbf{X}^{\top}\mathbf{X}\boldsymbol{\beta} = 0,

\mathbf{X}^{\top}\mathbf{X}\boldsymbol{\beta} = \mathbf{X}^{\top}\mathbf{y}.

In code, use a least-squares solver such as np.linalg.lstsq or torch.linalg.lstsq rather than explicitly forming a matrix inverse.

OLS is MLE under independent Gaussian noise. If

y_i = \mathbf{x}_i^{\top}\boldsymbol{\beta} + \epsilon_i, \qquad \epsilon_i\sim\mathcal{N}(0,\sigma^2),

then

\log L(\boldsymbol{\beta}) = -\frac{1}{2\sigma^2} \sum_i (y_i-\mathbf{x}_i^{\top}\boldsymbol{\beta})^2 + \text{const}.

Maximizing the likelihood is therefore equivalent to minimizing squared error.

7.4 Regularization in linear regression (MAP)

Ridge regression (L2 regularization):

\hat{\boldsymbol{\beta}}_{\mathrm{ridge}} = \arg\min_{\boldsymbol{\beta}} \|\mathbf{y}-\mathbf{X}\boldsymbol{\beta}\|_2^2 + \lambda\|\boldsymbol{\beta}\|_2^2.

A Gaussian prior on the coefficients gives the MAP interpretation:

\boldsymbol{\beta} \sim \mathcal{N} \left( 0, \frac{\sigma^2}{\lambda}I \right),

with

P(\boldsymbol{\beta}\mid\mathbf{y}) \propto P(\mathbf{y}\mid\boldsymbol{\beta}) P(\boldsymbol{\beta}).

Lasso regression (L1 regularization):

\hat{\boldsymbol{\beta}}_{\mathrm{lasso}} = \arg\min_{\boldsymbol{\beta}} \|\mathbf{y}-\mathbf{X}\boldsymbol{\beta}\|_2^2 + \lambda\|\boldsymbol{\beta}\|_1.

This corresponds to a Laplace prior on each coefficient, with the exact scale depending on the chosen objective parameterization.

**Figure 3. Ridge and lasso priors and penalties.** A Gaussian prior leads to a quadratic L2 penalty, while a Laplace prior leads to an L1 penalty with a sharp corner at zero.

8. Summary and Practice Questions

Key takeaways

Tensors are multidimensional arrays; order = number of axes = .ndim.
In the design matrix $\mathbf{X}$, rows are examples and columns are features.
PyTorch is NumPy-like but adds GPU support, autograd, and deep-learning utilities.
$\mathbf{X}\mathbf{w}+b$ computes pre-activations for a whole batch, with $b$ handled by broadcasting.
Linearity of expectation does not require independence.
Independent variables have zero covariance, but zero covariance does not in general imply independence.
MLE maximizes the likelihood; MAP maximizes likelihood times prior.
L2 regularization corresponds to a Gaussian-prior MAP interpretation; L1 corresponds to a Laplace-prior interpretation.
OLS is MLE under independent Gaussian noise; ridge and lasso have corresponding MAP interpretations.

Check your understanding

A batch of 64 grayscale $28\times28$ images: what is the tensor’s order and typical PyTorch shape?
Order 4, shape (64, 1, 28, 28).
If $\mathbf{X}\in\mathbb{R}^{100\times5}$, what shapes must $\mathbf{w}$ and $\mathbf{z}$ have in $\mathbf{X}\mathbf{w}+b=\mathbf{z}$?
$\mathbf{w}$ has shape $5\times1$ and $\mathbf{z}$ has shape $100\times1$.
You flip a coin 10 times and get 10 heads. What is the Bernoulli MLE? What is the MAP estimate with a Beta$(2,2)$ prior?
\hat\theta_{\mathrm{MLE}}=1,
and
\hat\theta_{\mathrm{MAP}} = \frac{10+2-1}{10+2+2-2} = \frac{11}{12} \approx0.917.