Lecture 04: Single-layer networks

From inputs and weights to the perceptron, decision boundaries, the Perceptron Learning Algorithm, and multi-neuron networks.

1. Today’s Roadmap

A single-layer network combines inputs using weights and bias, applies an activation function, and produces an output. The perceptron creates a linear decision boundary and the Perceptron Learning Algorithm updates the model for misclassifications.


2. Terminology & Notation

A single-layer network takes an input and computes a net input, also called the pre-activation or weighted input:

z = \sum_{i=0}^{n} w_i x_i.

The net input is passed through an activation function to produce the activation:

a = a(z).

The activation function determines how the neuron transforms its net input. For example, a perceptron uses a threshold function, while linear regression uses the identity function. A non-linear activation function such as the sigmoid can be written as $\sigma(z)$.

The predicted label output is then given by

\hat{y} = f(a).

The bias is included as $w_0$, with the corresponding input $x_0 = 1$.

3. Perceptrons

3.1 Continuous Weighting

A perceptron combines many inputs using continuous weights before applying an activation function. The weighted inputs are summed to produce the net input, which is then passed through an activation function to determine the output of the perceptron.

net = \sum_{i=0}^{n} w_i x_i.

3.2 Activation Function

The activation function is applied to the net input of the perceptron. Different activation functions result in different outputs.

In Deep Learning, the outputs are continuous and can take multiple values that signify the strength of a signal, which is why the sigmoid function (see below) applied on the activation makes the perceptron DL.

The sigmoid function:

\sigma(net) = \frac{1}{1+e^{-net}}.

Replacing the perceptron’s threshold with a sigmoid activation turns the model into logistic regression. Instead of outputting a hard class label, it outputs a probability.

3.3 Separation Boundary

An optimal estimate for a good separation boundary (with parameters θ) can be found using gradient ascent (MLE).

4. Threshold Functions

Threshold functions are used for discrete classification models and lead to the classic perceptron. Instead of continuous, the threshold function determines class by whether the net input reaches a threshold.

4.1 Classic Perceptron Limitations

The classic perceptron has several limitations:

5. Perceptron Learning Algorithm

5.1 Pseudocode

The Perceptron Learning Algorithm (PLA) updates weights based on if the perceptron is correct in classifying each training example.

Let the dataset be

D = (\langle x^{[1]}, y^{[1]}\rangle, \langle x^{[2]}, y^{[2]}\rangle, \ldots, \langle x^{[n]}, y^{[n]}\rangle) \in (\mathbb{R}^m \times \{0,1\})^n.

Initialize the weight vector to zero:

\mathbf{w} \leftarrow \mathbf{0}^m.

For every training epoch, the algorithm considers each training example $(\mathbf{x}^{[i]},y^{[i]})$.

The predicted label is calculated with the threshold function. Note that in the perceptron, $\sigma$ denotes the step function, not the sigmoid from Section 3.2: it outputs 1 when the net input is non-negative and 0 otherwise, so $\hat{y}^{[i]}$ is always 0 or 1.

\hat{y}^{[i]} = \sigma(\mathbf{x}^{[i]T}\mathbf{w}).

The error is calculated as follows. Since both $y^{[i]}$ and $\hat{y}^{[i]}$ are 0 or 1, the error can only be $-1$, $0$, or $1$:

\mathrm{err} = y^{[i]}-\hat{y}^{[i]}.

Finally, the weight vector is set to:

\mathbf{w} \leftarrow \mathbf{w}+\mathrm{err}\times\mathbf{x}^{[i]}.

5.2 Geometric Intuition

The weight vector is perpendicular to the decision boundary.

The update $\mathbf{w} \leftarrow \mathbf{w} + \mathrm{err} \times \mathbf{x}^{[i]}$ has a geometric reading. If the perceptron outputs 0 for a point whose target is 1, the error is $+1$ and the input vector is added to the weights, rotating $\mathbf{w}$ toward the point. If it outputs 1 for a point whose target is 0, the error is $-1$ and the input vector is subtracted, rotating $\mathbf{w}$ away from it. Correctly classified points give an error of 0 and leave the weights unchanged.

Because the bias is folded into the weight vector as $w_0$ with $x_0 = 1$, the same update adds or subtracts 1 from the bias, shifting the boundary so that the misclassified point moves toward the correct side.

If this is done for all points, eventually we reach a boundary that satisfies all points if the data is linearly separable.

5.3 PLA, MLE & MAP

The perceptron learning algorithm involves neither MLE nor MAP. This is because:

6. Notational Conventions for Neural Networks

6.1 From one neuron to many training examples

So far we have a single neuron. For one training example $\mathbf{x} \in \mathbb{R}^{m}$, the net input is

z = \mathbf{x}^\top \mathbf{w} + b.

If we have $n$ training examples stacked as the rows of the design matrix $\mathbf{X} \in \mathbb{R}^{n \times m}$, the same computation for all examples at once is

\mathbf{z} = \mathbf{X}\mathbf{w} + b, \qquad \mathbf{z} \in \mathbb{R}^{n \times 1},

where the scalar $b$ is added to every entry. Computing the prediction for one example is what is now often called “inference” in deep learning.

6.2 A fully-connected layer

Next we put $h$ neurons side by side. Every input feeds every neuron, which is why the layer is called fully connected. The inputs are weighted differently for each neuron: each neuron has its own weight vector, so even though all neurons see the same inputs, they can compute different outputs. Collecting the $h$ weight vectors as the rows of a matrix gives

\mathbf{x} = \begin{bmatrix} x_1 \\ x_2 \\ \vdots \\ x_m \end{bmatrix}, \qquad \mathbf{W} = \begin{bmatrix} w_{1,1} & w_{1,2} & \cdots & w_{1,m} \\ w_{2,1} & w_{2,2} & \cdots & w_{2,m} \\ \vdots & \vdots & \ddots & \vdots \\ w_{h,1} & w_{h,2} & \cdots & w_{h,m} \end{bmatrix} \in \mathbb{R}^{h \times m}.

Note the indexing convention: $w_{i,j}$ is the weight connecting the $j$-th input to the $i$-th output. Row $i$ of $\mathbf{W}$ is the weight vector of neuron $i$.

For one training example, the layer activations are

\mathbf{a} = \sigma\left(\mathbf{W}\mathbf{x} + \mathbf{b}\right), \qquad \mathbf{a} \in \mathbb{R}^{h \times 1},

where $\mathbf{b} \in \mathbb{R}^{h}$ holds one bias per neuron and $\sigma$ is applied elementwise.

For $n$ training examples stored in $\mathbf{X} \in \mathbb{R}^{n \times m}$, we transpose $\mathbf{X}$ so the examples become columns, and transpose the result back so that each row is again one example:

\mathbf{A} = \sigma\left(\left[\mathbf{W}\mathbf{X}^\top + \mathbf{b}\right]^\top\right), \qquad \mathbf{A} \in \mathbb{R}^{n \times h}.

Here $\mathbf{W}\mathbf{X}^\top$ is $h \times n$ and $\mathbf{b}$ is added to every column. Stacking several such layers, where the activations $\mathbf{a}^{(1)}$ of one layer become the inputs of the next, gives the multilayer networks of the coming lectures.

6.3 Why the $\mathbf{W}\mathbf{x}$ notation is intuitive

A matrix-vector product is a linear transformation of the input vector. The identity matrix leaves the input unchanged:

\begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix} \begin{bmatrix} x_1 \\ x_2 \end{bmatrix} = \begin{bmatrix} x_1 \\ x_2 \end{bmatrix}.

In general, the product is a weighted sum of the columns of the matrix:

\begin{bmatrix} a & b \\ c & d \end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix} = x \begin{bmatrix} a \\ c \end{bmatrix} + y \begin{bmatrix} b \\ d \end{bmatrix}.

The diagonal entries $a$ and $d$ scale the $x$ and $y$ coordinates, while the off-diagonal entries move $x$ into the $y$ direction ($c$) and $y$ into the $x$ direction ($b$). For example, the diagonal matrices below stretch the axes:

\begin{bmatrix} 3 & 0 \\ 0 & 1 \end{bmatrix}\begin{bmatrix} x \\ y \end{bmatrix} = \begin{bmatrix} 3x \\ y \end{bmatrix}, \qquad \begin{bmatrix} 1 & 0 \\ 0 & 2 \end{bmatrix}\begin{bmatrix} x \\ y \end{bmatrix} = \begin{bmatrix} x \\ 2y \end{bmatrix}, \qquad \begin{bmatrix} 3 & 0 \\ 0 & 2 \end{bmatrix}\begin{bmatrix} x \\ y \end{bmatrix} = \begin{bmatrix} 3x \\ 2y \end{bmatrix}.

So a layer $\mathbf{W}\mathbf{x}$ maps the input space to a new space, one output coordinate per neuron.

7. A Fully-Connected (Linear) Layer in PyTorch

In PyTorch a fully-connected layer is torch.nn.Linear. The lecture example uses 10 training examples with 5 features each and a layer with 3 outputs:

import torch

X = torch.arange(50, dtype=torch.float).view(10, 5)   # .view() and .reshape() are equivalent here
fc_layer = torch.nn.Linear(in_features=5, out_features=3)

print('X dim:', X.size())                  # torch.Size([10, 5])
print('W dim:', fc_layer.weight.size())    # torch.Size([3, 5])
print('b dim:', fc_layer.bias.size())      # torch.Size([3])

A = fc_layer(X)                            # .size() is equivalent to .shape
print('A dim:', A.size())                  # torch.Size([10, 3])

The weight matrix is stored as (out_features, in_features), exactly the $h \times m$ shape of $\mathbf{W}$ above, and the weights and bias are initialized randomly with requires_grad=True. Since the input follows the design-matrix convention (one row per example), PyTorch computes

\mathbf{A} = \mathbf{X}\mathbf{W}^\top + \mathbf{b},

which is the same as $\left[\mathbf{W}\mathbf{X}^\top + \mathbf{b}\right]^\top$ from Section 6.2 before the activation. Note that nn.Linear is only the affine part; the activation function is a separate layer.

7.1 About notation

ML culture is a mix of implementation and model development, and these do not always suggest the same notation. Theory usually writes $\mathbf{W}\mathbf{x}$ with column vectors, while code works with batches stored as rows. A useful tip is to think about how the dot products are computed when writing and implementing matrix multiplication. When switching between theory and code, these rules help:

\mathbf{A}\mathbf{B} = \left(\mathbf{B}^\top\mathbf{A}^\top\right)^\top, \qquad \left(\mathbf{A}\mathbf{B}\right)^\top = \mathbf{B}^\top\mathbf{A}^\top.

Be cautious of this when debugging, since it is very easy to make mistakes with shapes. Checking that the inner dimensions match ($n \times m$ times $m \times h$) is usually the fastest way to find the bug.

8. Summary