STAT 453

Lecture Notes from Lecture 2

Introduction to Deep Learning and Generative Models

Brief History of Machine Learning

Machine Learning is the process of creating programs that improve their performance on a task through experience. Here is the current timeline of machine learning and neural networks:

Artificial Neurons

McCulloch & Pitts Neuron

In 1943, Warren McCulloch and Walter Pitts introduced a mathematical model of a neuron. The model used inputs, weights, and a threshold to determine whether the neuron should activate. The weights were limited to $+1$ and $-1$.

These neurons could represent simple logical operations such as:

However, a single McCulloch & Pitts neuron could not represent XOR.

McCulloch & Pitts neuron model

Figure 1: McCulloch & Pitts neuron model.

The Perceptron

The perceptron, developed by Frank Rosenblatt in 1957, generalized the earlier artificial neuron.

Unlike the McCulloch & Pitts neuron, a perceptron uses continuous weights and an activation function.

For a binary classification problem, the perceptron attempts to find a decision boundary that separates the two classes.

If the classes are linearly separable, the perceptron can find a boundary that correctly separates them.

The perceptron learning algorithm updates its weights based on whether the predicted value agrees with the actual value. McCulloch & Pitts neuron model Figure 2: Perceptron architecture

The XOR Problem

A major limitation of a single perceptron is that it cannot represent the XOR logical operation.

For XOR:

The problem is that there is no single linear decision boundary that separates these two classes.

Therefore, a single perceptron cannot solve XOR.

This limitation helped motivate the development of multilayer neural networks, which can represent more complicated relationships.

Multi-layer Neural Networks

A multilayer perceptron (MLP), also called a multilayer neural network, combines multiple neurons into multiple layers.

The idea is to stack neuron models on top of each other so that the network can learn more complicated representations of the input. A single perceptron can only find a linear decision boundary, so it cannot solve problems such as XOR. However, multiple perceptrons can be combined to create the multiple decision boundaries needed for XOR.

The layers between the input and output are called hidden layers because their values are not directly observed in the training data. Multilayer neural network

Figure 3: Example of a multilayer neural network.

Backpropagation

A challenge with multilayer networks is that there are no direct target values for the hidden nodes. The network therefore needs a way to determine how much each parameter contributed to the final error.

Backpropagation provides this method. Neural networks can be represented as computation graphs, and the chain rule can be used to work backward through the graph and calculate gradients for the different parameters.

These gradients are then used to update the network’s parameters during training, allowing the network to learn the appropriate weights needed to represent more complicated functions such as XOR.

Deep Learning

Deep learning refers to using neural networks with multiple layers to learn increasingly complex representations of data.

The main idea is that different layers can learn different levels of representation. Earlier layers can learn simpler patterns, while later layers can combine those patterns into more complex features.

The term “deep” refers to the number of layers in the neural network. Adding more layers allows the model to learn a hierarchy of representations.

Building Blocks of Deep Learning

Deep learning models are built by combining several important components:

Together, these components allow a neural network to learn useful representations from data rather than requiring a person to manually specify all of the features.

Hierarchical Representations

One important idea in deep learning is that neural networks can learn hierarchical representations.

For example, when working with an image, earlier layers might learn simple patterns such as edges. Later layers can combine these patterns to recognize shapes, and even later layers can combine those shapes into more complex objects.

This allows the network to learn useful features automatically from the data.

The Rise of Deep Learning

Although neural networks had existed for decades, deep learning became much more successful when larger datasets, better algorithms, and more powerful hardware became available.

Benchmark datasets played an important role in measuring progress in machine learning.

Some important datasets include:

These datasets made it possible to compare different machine learning approaches and measure improvements in performance.

AlexNet

A major breakthrough came with AlexNet, which achieved a large improvement on the ImageNet image-classification benchmark.

AlexNet demonstrated that deep neural networks could perform extremely well when combined with large datasets and powerful hardware.

This helped lead to the rapid growth of modern deep learning during the 2010s.

The DL Hardware & Software Landscape

Hardware

CPU

GPU

Memory and Infrastructure Bottlenecks

AI compute demand now affects:

Self-Supervised Learning

Structured Data

Deep learning can also be applied to structured data such as graphs.

Massive Unsupervised Learning

From GPT-1 (2018) to GPT-4:

Compute-Optimal Scaling

Takeaway: “Bigger” alone is no longer the story. What and how much data matters as much as model size.

Open Directions