STAT 453

Lecture 05: Parameter Optimization and Gradient Descent

From the perceptron's hard threshold to a smooth sigmoid unit, maximum likelihood, stochastic gradient descent, and backpropagation on computation graphs.

1. Today’s Roadmap

The thread running through all four: a hard threshold gives us no way to learn from data, so we swap in a smooth activation function, turn the model into a likelihood we can maximize, and then walk up that likelihood with gradient descent. Applying gradient descent to a deep model is backpropagation.

2. Review of the Perceptron from Last Class

2.1 McCulloch and Pitts’ neuron model (1943)

McCulloch and Pitts made a neuron model that had inputs feeding a linear combiner, which passes through a hard limiter to produce an output. The hard limiter fires only when the weighted sum exceeds a threshold \theta.

2.2 Rosenblatt’s perceptron (1957)

Rosenblatt built on this by creating the perceptron, which added continuous weighting and an activation (threshold) function. The net input is

\text{net} = \sum_{i=0}^{n} w_i x_i, \qquad o = \sigma(\text{net}),

where x_0 = 1 carries the bias weight w_0. Perceptrons generalize MP neurons a bit, since the weights are now real-valued rather than fixed.

2.3 The perceptron learning algorithm

The algorithm assumes a binary classification task, and it finds a decision boundary if the classes are separable. For a dataset

\mathcal{D} = \left(\langle \mathbf{x}^{[1]}, y^{[1]}\rangle, \ldots, \langle \mathbf{x}^{[n]}, y^{[n]}\rangle\right) \in \left(\mathbb{R}^m \times \{0,1\}\right)^n,

the pseudocode is:

  1. Initialize \mathbf{w} := \mathbf{0}^m (assume the weight vector includes the bias).
  2. For every training epoch:
    • For every \langle \mathbf{x}^{[i]}, y^{[i]} \rangle \in \mathcal{D}:
      • \hat{y}^{[i]} := \sigma\left(\mathbf{x}^{[i]T}\mathbf{w}\right)

        , which is only 0 or 1

      • \textit{err} := \left(y^{[i]} - \hat{y}^{[i]}\right)

        , which is only -1, 0, or 1

      • \mathbf{w} := \mathbf{w} + \textit{err} \times \mathbf{x}^{[i]}

Because the error can only be -1, 0, or 1, the update is just adding or subtracting the input vector to the weight vector. There is no notion of how wrong a prediction was, which is the limitation the sigmoid unit fixes later.

2.4 Geometric intuition

The weight vector is perpendicular to the decision boundary. To see why, recall that

\hat{y} = \begin{cases} 0, & \mathbf{w}^\top \mathbf{x} \leq 0 \\ 1, & \mathbf{w}^\top \mathbf{x} > 0 \end{cases} \qquad\text{and}\qquad \mathbf{w}^\top \mathbf{x} = \lVert \mathbf{w} \rVert \cdot \lVert \mathbf{x} \rVert \cdot \cos(\theta).

At the boundary \mathbf{w}^\top \mathbf{x} needs to be 0, and \cos(\theta) is zero at 90^\circ, so the weight vector sits at a right angle to the boundary.

Figure 1. Perceptron geometric intuition. Points on one side of the boundary are labeled 1 and points on the other are labeled 0. Since \mathbf{w}^\top\mathbf{x} = \lVert\mathbf{w}\rVert\lVert\mathbf{x}\rVert\cos(\theta) must vanish on the boundary, the weight vector points perpendicular to it.

When a point with label 1 lands on the wrong side, we add its input vector to the weight vector. The new weight vector is rotated toward that example, which puts it on the correct side.

3. Beyond Rosenblatt’s Perceptron: Choosing an Activation Function

With a hard threshold function we need some way to learn from data, and the threshold gives us no gradient to work with. Everything beyond the classic perceptron comes from swapping out the activation function:

Activation function What the unit becomes
Threshold function (perceptron, 1950+) Classic Rosenblatt perceptron
Sigmoid function (before 2000) Deep learning “perceptron” / sigmoid unit
Identity Linear regression
ReLU (popular since CNNs), plus variants such as leaky ReLU and GeLU Modern deep networks

If we swap out the activation function for something smooth like the sigmoid, we get a deep learning perceptron, or sigmoid unit. If we use the identity as the activation function we get linear regression.

In lecture. Depth only buys us something when the activation is nonlinear. Stacking units with identity activations composes linear maps into one linear map, so the stack collapses back to a single linear model. Nonlinear activations are what let a stack of units represent functions a single unit cannot.

Making the activation function smooth is the key move. Once it is differentiable, we can apply gradient descent, and applying gradient descent to a deep learning model is backpropagation.

4. The Sigmoid Unit: Logistic Regression Gives Us an Optimizer

4.1 The forward pass

For binary classes y \in \{0, 1\}, the weights come together to form the net input, which is put into the activation function (the logistic sigmoid), which forms the output:

z = \sum_i w_i x_i + b, \qquad a = \sigma(z) = \frac{1}{1 + e^{-z}}, \qquad h(\mathbf{x}) = \sigma(\mathbf{w}^\top \mathbf{x} + b).
Figure 2. The sigmoid unit. Inputs x_1, \ldots, x_m are weighted and summed into the net input z, which is squashed by the logistic sigmoid into the activation a.

4.2 A probabilistic model

Because the output lies in (0,1), we can read it as a probability:

P(y \mid \mathbf{x}) = \begin{cases} h(\mathbf{x}) & \text{if } y = 1 \\ 1 - h(\mathbf{x}) & \text{if } y = 0 \end{cases}

Recalling the Bernoulli distribution, the two cases collapse into one expression:

P(y \mid \mathbf{x}) = a^{y}(1-a)^{(1-y)}.

Once we have a probabilistic model we have a likelihood function, and we can optimize it in different ways. Optimizing it by gradient descent is what gives us backpropagation.

4.3 Likelihood and log-likelihood

Under MLE we would like to maximize the multi-sample likelihood:

P\left(y^{[i]}, \ldots, y^{[n]} \mid \mathbf{x}^{[1]}, \ldots, \mathbf{x}^{[n]}\right) = \prod_{i=1}^{n} P\left(y^{[i]} \mid \mathbf{x}^{[i]}\right) = \prod_{i=1}^{n}\left(\sigma(z^{(i)})\right)^{y^{(i)}}\left(1 - \sigma(z^{(i)})\right)^{1 - y^{(i)}}.

Since we are going to optimize via gradient descent, we apply the logarithm to separate the components. The maximizer of the likelihood is also the maximizer of the log-likelihood, so nothing is lost:

l(\mathbf{w}) = \log L(\mathbf{w}) = \sum_{i=1}^{n}\left[y^{(i)}\log\left(\sigma(z^{(i)})\right) + \left(1 - y^{(i)}\right)\log\left(1 - \sigma(z^{(i)})\right)\right].

The log turns a product into a sum, which makes the derivative of each component much easier to compute.

4.4 Deriving the gradient piece by piece

Dividing the equation into parts makes deriving each part easier. By the chain rule,

\frac{\partial \mathcal{L}}{\partial w_j} = \frac{\partial \mathcal{L}}{\partial a}\frac{da}{dz}\frac{\partial z}{\partial w_j},

and the three pieces are

\frac{\partial \mathcal{L}}{\partial a} = \frac{a - y}{a - a^2}, \qquad \frac{da}{dz} = \frac{e^{-z}}{(1 + e^{-z})^2} = a \cdot (1 - a), \qquad \frac{\partial z}{\partial w_j} = x_j.

Multiplying them through, the messy denominator cancels:

\frac{\partial \mathcal{L}}{\partial z} = a - y \qquad \Longrightarrow \qquad \frac{\partial \mathcal{L}}{\partial w_j} = (a - y)x_j.

In lecture. If the post-activation value is very far away from the target, we make a bigger update to our weight. If it is very close, we make a very little weight update. The perceptron could not do this, since its error was only -1, 0, or 1.

5. Gradient Descent

5.1 Why we take steps instead of solving

If we want the best weights for our data, we want to maximize the likelihood. If we could set the derivative to zero and solve, we would, but for most deep learning functions we can’t. Instead we take steps in the direction of the gradient (the slope) to reach a local maximum. We can’t guarantee a global maximum, but we can find a local one by walking up the gradient.

So the recipe is: take the derivative first, then walk up the gradient to find the maximizer.

5.2 Stochastic gradient descent

Stochastic gradient descent is a greedy algorithm. At each point it takes one step in the direction that moves us best toward the goal. The gradient tells us which direction to move in, and the weight vector is updated by some step size times that gradient.

  1. Initialize \mathbf{w} := \mathbf{0} \in \mathbb{R}^m, b := 0.
  2. For every training epoch:
    • For every \langle \mathbf{x}^{[i]}, y^{[i]} \rangle \in \mathcal{D}:
      • (a) \hat{y}^{[i]} := \sigma\left(\mathbf{x}^{[i]T}\mathbf{w} + b\right)
      • (b) \nabla_{\mathbf{w}}\mathcal{L} = -\left(y^{[i]} - \hat{y}^{[i]}\right)\mathbf{x}^{[i]} and \nabla_{b}\mathcal{L} = -\left(y^{[i]} - \hat{y}^{[i]}\right)
      • (c) \mathbf{w} := \mathbf{w} + \eta \times \left(-\nabla_{\mathbf{w}}\mathcal{L}\right) and b := b + \eta \times \left(-\nabla_{b}\mathcal{L}\right)

Here \eta is the learning rate and -\nabla\mathcal{L} is the negative gradient. This is the same quantity we derived above, just with the sign flipped:

a - y \iff -\left(y^{[i]} - \hat{y}^{[i]}\right).

Using the negative versus the positive gradient is what changes whether we are descending or ascending. We iteratively take steps in the direction of the gradient, which is the slope, and that steps us toward the optimum in a greedy fashion.

6. Multilayer Perceptron

Gradient descent applies to large models too, not just a single unit. A multilayer perceptron is a computation graph with multiple fully-connected layers, and stacking layers allows us to model more functions than a single unit can.

Figure 3. Multilayer perceptron. Inputs x_1, x_2 feed a hidden layer a^{(1)}, then a second hidden layer a^{(2)}, then the outputs o_1, o_2, o_3, which are compared against the targets by the loss \mathcal{L}(\mathbf{y}, \mathbf{o}).

Use softmax at the output if this is a multi-class problem with mutually exclusive classes.

7. Backpropagation and Computation Graphs

7.1 The chain rule is the whole thing

The chain rule is basically the essence of training deep neural networks, and it is the basis for all neural networks and deep learning. For a nested function decomposed into an inner and an outer part,

F(x) = f(g(x)) = z \qquad \Longrightarrow \qquad F'(x) = f'(g(x))\,g'(x) = z'.

Neural networks are function compositions that can be represented as computation graphs. By applying the chain rule and working in reverse order, we get

\frac{\partial f_n}{\partial x} = \sum_{i_1 \in \pi(n)} \frac{\partial f_n}{\partial f_{i_1}}\frac{\partial f_{i_1}}{\partial x} = \sum_{i_1 \in \pi(n)} \frac{\partial f_n}{\partial f_{i_1}} \sum_{i_2 \in \pi(i_1)} \frac{\partial f_{i_1}}{\partial f_{i_2}}\frac{\partial f_{i_1}}{\partial x} = \cdots

where \pi(n) denotes the parents of node n in the graph. In practice: take the gradient of the output, then move back and take the gradient with respect to those inputs, and keep moving back.

7.2 Quick calculus refresher

The derivative of a function is its rate of change, or its slope:

f'(x) = \frac{df}{dx} = \lim_{\Delta x \to 0}\frac{f(x + \Delta x) - f(x)}{\Delta x}.

Conceptually we are approximating the slope (the tangent) by a secant between two points and then shrinking the gap. Two worked examples from lecture: for f(x) = 2x the difference quotient is 2\Delta x / \Delta x, so df/dx = 2; for f(x) = x^2 it is (2x\Delta x + (\Delta x)^2)/\Delta x, so df/dx = 2x.

The rules we lean on are the standard ones: sum, difference, product, quotient, reciprocal, and above all the chain rule, \frac{d}{dx}f(g(x)) = f'(g(x))g'(x).

7.3 PyTorch does this automatically

PyTorch can usually do all of this automatically for us. It literally keeps a computation graph in the background, so when you define f(x) and g(x) it already has what it needs to compute f'(x) and g'(x), and it can compute the derivatives of most differentiable functions automatically.

7.4 Worked example: ReLU

ReLU is the Rectified Linear Unit, probably the most commonly used activation function in deep learning. Every input that is positive is sent straight to the output, and every input that is negative is set to 0:

\text{relu}(z) = \begin{cases} z & \text{if } z > 0 \\ 0 & \text{otherwise} \end{cases} \qquad\text{equivalently}\qquad \text{relu}(z) = \max(0, z).

Take the activation a(x, w, b) = \text{relu}(w \cdot x + b) and break it into intermediate nodes: u = wx, then v = u + b, then a = \text{relu}(v).

Figure 4. Computation graph for a ReLU unit, with w = 2 and b = 1. The forward pass runs left to right through u = wx, v = u + b, and a = \text{relu}(v). The red arrows are the backward pass: first da/dv, then \partial v/\partial b, and so on back to each input.

Then we can work backwards with the chain rule, doing da/dv first and then \partial v/\partial b. The local derivatives are simple:

\frac{da}{dv} = \begin{cases} 1 & v > 0 \\ 0 & \text{otherwise}\end{cases}, \qquad \frac{\partial v}{\partial b} = 1, \qquad \frac{\partial v}{\partial u} = 1, \qquad \frac{\partial u}{\partial w} = x.

7.5 Single-path graphs

For a single path, we take the product of each of these derivatives, which is the univariate chain rule. For \mathcal{L}\left(y, \sigma_1(w_1 \cdot x_1)\right):

\frac{\partial l}{\partial w_1} = \frac{\partial l}{\partial o}\cdot\frac{\partial o}{\partial a_1}\cdot\frac{\partial a_1}{\partial w_1}.

7.6 Fully-connected layers

For a fully-connected layer we have to sum the different paths of derivatives from the input weight to the output, because a single weight influences the loss through every downstream unit it feeds. Backprop traces how much a weight affects the output using the chain rule along each of those paths:

\frac{\partial l}{\partial w_{1,1}^{(1)}} = \frac{\partial l}{\partial o}\cdot\frac{\partial o}{\partial a_1^{(2)}}\cdot\frac{\partial a_1^{(2)}}{\partial a_1^{(1)}}\cdot\frac{\partial a_1^{(1)}}{\partial w_{1,1}^{(1)}} + \frac{\partial l}{\partial o}\cdot\frac{\partial o}{\partial a_2^{(2)}}\cdot\frac{\partial a_2^{(2)}}{\partial a_1^{(1)}}\cdot\frac{\partial a_1^{(1)}}{\partial w_{1,1}^{(1)}}.
Figure 5. Computation graph for a fully-connected layer. The weight w_{1,1}^{(1)} reaches the loss through both units of the second hidden layer, so its gradient is the sum of the two path products.

7.7 Weight sharing

Weight sharing happens in convolutional neural networks. CNNs are really good for image recognition because a fully-connected network would not work at that scale, since that is far too many paths connecting every pixel to every other pixel. So CNNs reuse the same weight in many places, and it works really well.

When weights are shared across different paths, we have to use the multivariable chain rule and sum up the contributions of each of the paths the shared weight appears on. For \mathcal{L}\left(y, \sigma_3\left[\sigma_1(w_1 \cdot x_1), \sigma_2(w_1 \cdot x_1)\right]\right), where the same w_1 appears on both paths:

\frac{\partial l}{\partial w_1} = \underbrace{\frac{\partial l}{\partial o}\cdot\frac{\partial o}{\partial a_1}\cdot\frac{\partial a_1}{\partial w_1}}_{\text{upper path}} + \underbrace{\frac{\partial l}{\partial o}\cdot\frac{\partial o}{\partial a_2}\cdot\frac{\partial a_2}{\partial w_1}}_{\text{lower path}}.
Figure 6. Weight sharing. The same weight w_1 is used on two different paths from x_1 to the output o, so its gradient is the sum of the contributions from the upper and lower paths.

8. Summary