Lecture 14: Variational Autoencoders and Generative Adversarial Networks

An overview of VAEs, GANs, and a unified view of the two

1. Recap: Generative Models and Autoencoders

Last lecture introduced generative models, which model the joint distribution of data P(X,Y) in order to generate new data X. It also introduced the autoencoder, an unsupervised model that learns to reduce the dimensionality of data X into a more compact, latent space z, and then reconstruct X from the latent space.

Two Paths to P(Y|X)

Two ways of predicting a label $Y$ from an input $X$, given observations $X, Y$:

P(X)=\int_Y P(X|Y)P(Y)\,dY, \qquad P(Y|X)=\frac{P(X|Y)P(Y)}{P(X)}

Deep Generative Models

Deep generative models use neural networks to represent complex distributions. Multiple layers of hidden variables hope to capture the hidden structure $z$ that explains high-dimensional $X$ values.

Because we do not observe $z$, two important computations become difficult:

Each type of deep generative model makes a tradeoff.

Autoencoders

An autoencoder consists of an encoder, which maps an input $X$ to a latent space $z$, and a decoder, which reconstructs the input from that latent space.

Figure from the Lecture 14 slides.
\mathcal{L} (x,x')=\|x-x'\|_2^2=\sum_i(x_i-{x'}_i)^2

This bottleneck enables the autoencoder to capture useful structure. However, without knowing the distribution of this latent space $z$, we cannot conveniently use the autoencoder to generate new $X$: a randomly chosen $z$ may fall in a region the decoder has never learned to use.

A VAE addresses this problem by introducing a prior $p(z)$, allowing us to sample $z$ values.

2. Variational Autoencoders (VAEs)

The goal is to shape the latent space $z$ into a chosen prior distribution $p(z)$. Once the model is trained, this allows us to sample from this distribution and feed it through the decoder to generate new $x’$. This introduces a new problem: the encoder needs to learn to map inputs $x$ to our chosen latent space. To achieve this, rather than mapping each $x$ to a single point in $z$, the encoder will map inputs to conditional distributions $q(z\mid x)$ that align with our prior $p(z)$.

The Encoder: Variational Inference

We want to approximate the true posterior $p(z\mid x)$. Variational inference is a method that will allow the encoder to learn a proxy for this posterior.

The main idea is to choose a family of distributions parameterized by variational parameters $v$,

q(z|v)

then find $v$ that minimizes the discrepancy between q and the true posterior. Restricting the distributional family to known distributions makes this feasible.

Figure from the Lecture 14 slides.

In a VAE, an encoder network, with weights $\phi$, learns the parameters of $q_{\phi}(z\mid x)$. Our measure of how close q is to the posterior of interest is the KL divergence:

D_{\text{KL}}(q_{\phi}(z|x)\|p_{\theta}(z|x))

The Decoder

The ideal generation process:

  1. $z$ is sampled from the true prior $p_{\theta^*}(z)$
  2. $x$ is sampled from the true conditional $p_{\theta^*}(x\mid z^{(i)})$

To represent this model, we will choose the prior $p(z)$ to be a simple distribution, such as a Gaussian. Since the conditional $p(x\mid z)$ is more complex, we will represent it with a neural network.

To train the model, we’d ideally maximize the likelihood of training data:

p_{\theta}(x) = \int p_{\theta}(z)p_{\theta}(x|z)\,dz

Bringing it Together

The decoder network has parameters \theta, describing a distribution over reconstructed observations. It predicts a mean and diagonal covariance for x\mid z.

The encoder network has parameters \phi and describes a distribution over the latent space. It predicts a mean and diagonal covariance for z\mid x.

Figure from the Lecture 14 slides.

Deriving the Loss Function

Start with the log-likelihood of the observation $x$ under the model parameterized by $\theta$. We take the expectation over the posterior distribution produced by the encoder network (they are equivalent since $p_\theta(x^{(i)})$ does not depend on $z$).

\log p_\theta(x^{(i)}) =\mathbb{E}_{z \sim q_\phi(z|x^{(i)})}\left[ \log p_\theta(x^{(i)}) \right]

The next steps are an application of Bayes’ Rule, arithmetic, and the definition of KL divergence:

\begin{aligned} =&\mathbb{E}_{z}\left[ \log \frac{p_\theta(x^{(i)}|z)p_\theta(z)}{p_\theta(z|x^{(i)})} \right] \\ =&\mathbb{E}_{z}\left[ \log \frac{p_\theta(x^{(i)}|z)p_\theta(z)}{p_\theta(z|x^{(i)})} \frac{q_\phi(z|x^{(i)})}{q_\phi(z|x^{(i)})} \right] \\ =&\mathbb{E}_{z}\left[ \log p_\theta(x^{(i)}|z) \right] - \mathbb{E}_{z}\left[ \log \frac{q_\phi(z|x^{(i)})}{p_\theta(z)} \right] + \mathbb{E}_{z}\left[ \log \frac{q_\phi(z|x^{(i)})}{p_\theta(z|x^{(i)})} \right] \\ =&\mathbb{E}_{z}\left[\log p_\theta(x^{(i)}|z)\right] - D_{\text{KL}}(q_\phi(z|x^{(i)})\|p_\theta(z)) + D_{\text{KL}}(q_\phi(z|x^{(i)})\|p_\theta(z|x^{(i)})) \end{aligned}

This first term is the reconstruction loss. The second measures how close our learned function $q$, within the variational family, is to our chosen prior. The third term measures how far our learned function $q$ is from the true distribution $p_\theta(z\mid x^{(i)})$. We do not know $p_\theta(z\mid x^{(i)})$ so we choose to ignore it. We do know it’s nonnegative, so the first two terms form an evidence lower bound (ELBO):

\begin{aligned} \mathcal{L}_{\mathrm{ELBO}}(x^{(i)},\theta,\phi) &=\mathbb{E}_{z}[\log p_\theta(x^{(i)}|z)] -D_{\mathrm{KL}}(q_\phi(z|x^{(i)})\|p_\theta(z)) \\ &\leq\log p_\theta(x^{(i)}) \end{aligned}

Training maximizes the ELBO. The two terms serve different purposes:

Putting it Together

Maximizing the lower bound

\mathcal{L}(x^{(i)}, \theta, \phi) =\mathbb{E}_{z}[\log p_\theta(x^{(i)}\mid z)] -D_{\mathrm{KL}}(q_\phi(z\mid x^{(i)})\|p_\theta (z))

forward pass for a given minibatch:

  1. Start with input data $x$, pass it to encoder network
  2. Encoder network uses $q_\phi(z\mid x)$ to predict $\mu_{z\mid x}$ and $\Sigma_{z\mid x}$
  3. Sample $z$ from $z\mid x \sim \mathcal{N}(\mu_{z\mid x}, \Sigma_{z\mid x})$, pass it to decoder network
  4. Decoder network uses $p_\theta(x\mid z)$ to predict $\mu_{x\mid z}$ and $\Sigma_{x\mid z}$
  5. Sample $x\mid z$ from $x\mid z \sim \mathcal{N}(\mu_{x\mid z}, \Sigma_{x\mid z})$
Figure from the Lecture 14 slides.

Generation

After training, generate new $x$ by sampling $z$ from the prior and use only the decoder:

Figure from the Lecture 14 slides.

Generative Adversarial Networks (GANs)

GANs learn to generate data through an adversarial game between a generator and a discriminator. The goal is to learn the training set distribution and generate new data resembling the training data. Like the VAE, GANs generate an entire image all at once, in contrast to autoregressive models or RNNs.

Discriminator and Generator

The two networks have competing roles:

The generator takes random noise (sampled from a known prior distribution) as input and supplies the discriminator with generated data. The discriminator assesses the generated data and then supplies the generator its loss function.

Figure from the Lecture 14 slides.

Training

Discriminator objective:

\text{max}_D \mathcal{L}_D = \mathbb{E}_{x \sim p_\text{data}(x)}[\log D(x)] + \mathbb{E}_{x\sim G(z), z\sim p(z)}[\log (1-D(x))]

Generator objective:

\text{min}_G \mathcal{L}_G = \mathbb{E}_{x\sim G(z), z\sim p(z)}[\log (1-D(x))]

Together, the GAN objective is a minimax:

\text{min}_G \text{max}_D V(D, G) = \mathbb{E}_{x \sim p_\text{data}(x)}[\log D(x)] + \mathbb{E}_{ z\sim p_z(z)}[\log (1-D(G(z)))]

The discriminator maximizes this expression, assigning high probabilities to real examples and low probabilities to generated ones. The generator minimizes it, encouraging the discriminator to assign high probabilities to its outputs.

Training alternates between two updates:

  1. Update the discriminator: Sample real data and sample noise to generate fake data. Maximize the real/fake classification objective while holding the generator fixed.
  2. Update the generator: Sample noise vectors and minimize the generator objective while holding the discriminator’s parameters fixed. Gradients pass through the discriminator into the generator.

Discriminator stochastic gradient (update by ascending):

\nabla_{\theta_d}\frac{1}{m}\sum_{i=1}^{m} \left[\log D(x^{(i)})+\log(1-D(G(z^{(i)})))\right]

Generator stochastic gradient (update by descending):

\nabla_{\theta_g}\frac{1}{m}\sum_{i=1}^{m} \log(1-D(G(z^{(i)})))

where m is the mini-batch size.

At an ideal equilibrium, the generator distribution matches the data distribution, the optimal discriminator outputs 0.5, and neither can improve its objective by changing its strategy.

Training Problems

GAN training can be unstable because each network changes the learning problem faced by the other.

Non-saturating generator loss can help prevent vanishing gradients, ask your doctor if it’s right for you:

\nabla_{w_g}\frac{1}{n}\sum_{i=1}^{n} \log(D(G(z^{(i)})))

A Unified View (Bonus)

Both VAEs and GANs combine a process that produces observations with a process that infers something about them, but their objectives and network roles differ.

GANs: Rewritten

Consider an implicit distribution over $x\sim p_\theta (x\mid y)$ with a binary label y, where y=0 means generated and y=1 means real:

p_\theta(x\mid y)= \begin{cases} p_{g_\theta}(x), & y=0 \\ p_{\mathrm{data}}(x), & y=1 \end{cases}

The generated data is sampled via x=G_\theta(z), with $z\sim p(z\mid y=0)$. The discriminator represents a conditional distribution $q_\phi(y\mid x)$:

q_\phi^r(y\mid x) = q_\phi(1-y\mid x)

then the objectives become

\begin{aligned} \text{max}_\phi \mathcal{L}_\phi = \mathbb{E}_{p_\theta (x\mid y)p(y)}[\log q_\phi (y\mid x)] \\ \text{max}_\theta \mathcal{L}_\theta = \mathbb{E}_{p_\theta (x\mid y)p(y)}[\log q_\phi^r (y\mid x)] \end{aligned}

These objectives maximize the conditional log-likelihood of $y$ with the distribution $q_\phi (y\mid x)$ conditioned on data/generated data $x$ from $p_\theta (x\mid y)$. Here, $q_\phi (y\mid x)$ is the generative model and $p_\theta(x\mid y)$ is the inference model.

Compare these to the reconstruction term of a VAE’s ELBO. In a VAE, this term serves to maximize the conditional log-likelihood of $x$ with the generative distribution $p_\theta (x\mid z)$ conditioned on the latent code $z$ inferred by $q_\phi (z\mid x)$. From earlier, we know $p_\theta (x\mid z)$ is the generative model and $q_\phi (z\mid x)$ is the inference model.

VAEs vs GANs

Side-by-side

Figure from the Lecture 14 slides.

Note the asymmetry in KL divergence minimization: GANs minimize \text{KL}(P_\theta\|Q) where VAEs minimize \text{KL}(Q\|P_\theta).

Mode Covering vs Mode Choosing

Mode Covering vs. Mode Choosing

Models balance producing realistic examples with representing the diversity of the data. VAEs tend to cover modes: rather than commit to generating only the most realistic data, their distribution spreads out to cover many modes, including regions with small values of $p_{\text{data}}$. In the case of image generation, VAEs often produce varied but blurry images, although this blurriness is also influenced by their reconstruction loss. GANs, on the other hand, tend to miss modes, choosing to include some and not others in their generation. In the case of image generation, this often results in sharper but less diverse images.

One way to conceptualize this is through their training. A GAN’s generator gets to choose which images it sends to the discriminator; once it has learned which images best fool the discriminator, why branch out to represent rarer data? VAEs, however, are evaluated based on their ability to reconstruct every image in the training data, encouraging them to represent the entire distribution rather than concentrate on a few modes.

This behavior is captured by the asymmetry of the two models’ KL divergence:

Figure from the Lecture 14 slides.

What to Remember

Figure from the Lecture 14 slides.