Building a Flow Matching Model from Scratch: Theory Explained

Introduction

Diffusion models are everywhere, capable of generating impressive images, video, and music, but they are slow because they typically require hundreds of denoising steps to produce a single image. Flow matching offers a new approach that directly solves an ordinary differential equation (ODE) to generate data.

Why Not Use Diffusion Models Directly?

Diffusion models such as DDPM learn a score function (the gradient of the data distribution) by progressively adding Gaussian noise to an image until it becomes pure noise, then training the model to reverse this process. At generation time the model must start from pure noise and execute many small denoising steps, each guided by the learned score function, which makes sampling computationally expensive. Deterministic variants like DDIM remove stochasticity but still need dozens of neural‑network forward passes.

Flow Matching Principle

Flow matching reframes the generation problem as learning a smooth, time‑dependent transformation that moves samples from a simple prior distribution p₀ (e.g., standard Gaussian noise) to a complex target distribution p₁ (e.g., natural images). Instead of learning to denoise, the model learns a velocity field f(x, t) that describes the instantaneous speed of a point along a trajectory connecting the two distributions.

Problem description : we sample a point x₀ from p₀ and a point x₁ from p₁. The simplest trajectory is a linear interpolation

x(t) = (1 - t) * x₀ + t * x₁

The true velocity of this path is the derivative of x(t) with respect to t, which equals x₁ - x₀. The neural network is trained to predict this velocity, minimizing a supervised loss that compares the predicted velocity f(x, t) with the analytical velocity.

Thus the training objective is to make the predicted velocity as close as possible to the real velocity derived from the interpolation.

Sampling with Flow Matching

During generation we only have a noise sample x₀ ~ p₀. We discretize the time interval t ∈ [0, 1] into a sequence of steps and iteratively update the sample using the learned velocity field:

x_{t+Δt} = x_t + Δt * f(x_t, t)

At each step the current state x and time t are fed into f to obtain an estimate of the velocity, which is then used to advance the sample. When t = 1 the sample should lie in the data distribution, ideally looking like a realistic natural image.

The whole process can be seen as moving a particle along a learned flow field, progressively pushing it from noise toward the target distribution.