Understanding Variational Autoencoders: Core Concepts and Training Explained

Variational AutoEncoder (VAE)

Two major generative models dominate current research: the Variational AutoEncoder (VAE) and the Generative Adversarial Network (GAN). Unlike GANs, which rely on adversarial training, VAE employs mathematically elegant variational inference to optimise a lower bound of the data likelihood, enabling stable model optimisation.

Key Questions

What is the basic idea of a VAE?

How is variational training performed?

Analysis and Answer

A VAE assumes that each observation x is generated from a latent variable z. The encoder network approximates the posterior distribution q_\phi(z|x), while the decoder network defines the likelihood p_\theta(x|z). Training maximises the evidence lower bound (ELBO):

The ELBO can be written as

and after applying Bayes’ rule and the KL‑divergence term, we obtain

The optimisation proceeds by sampling z from the approximate posterior, computing the reconstruction loss \mathbb{E}_{q(z|x)}[\log p(x|z)], and adding the KL‑divergence regulariser KL(q(z|x)\|p(z)). Gradients flow through the re‑parameterisation trick, allowing end‑to‑end training with stochastic gradient descent.

Reference

Kingma, D. P., & Welling, M. (2013). Auto‑encoding variational Bayes. *arXiv preprint arXiv:1312.6114*.

Next Issue Preview

The upcoming article will cover Generative Adversarial Networks (GANs), exploring their objective functions and creative applications.