Why Energy‑Based Models Could Outperform Probabilistic LLMs, According to Yann LeCun

Yann LeCun repeatedly stresses that the prevailing probability‑based, token‑wise prediction paradigm of large language models (LLMs) is unlikely to achieve human‑level AI. His team favors an alternative framework: Energy‑Based Models (EBM), which use an energy function instead of a probability distribution.

In an EBM each possible data point x is assigned a scalar energy E(x). Low energy corresponds to high probability, mirroring the physical intuition that systems settle in low‑energy states. For example, given the query X = "Does a car have four wheels?", an LLM might output p("yes") = 0.9, p("no") = 0.1, whereas an EBM could assign E("yes", X) = -3.1, E("no", X) = 0.4; the answer with the lower energy wins.

Both models can be related through a normalization step: applying a softmax to the EBM’s raw energies yields a probability distribution, but the crucial difference is that the EBM does not enforce the sum‑to‑one constraint during training. This freedom removes the need to compute a partition function Z, which is notoriously intractable for high‑dimensional continuous data.

Constructing an EBM

The training objective is to learn a deep neural network that maps a combined input (X, Y) to a scalar energy. Correct (X, Y) pairs should receive low energy, while implausible pairs receive high energy. The loss has two parts: (1) push down the energy of observed, correct samples; (2) push up the energy of incorrect or sampled negatives.

From Energy to Density

To obtain a probability density function q(x) from the energy, one exponentiates the negative energy, exp(-E(x)), and normalizes over all states: q(x) = exp(-E(x)) / Z. The constant Z (partition function) is the sum or integral of exp(-E(x)) over the entire space and is generally impossible to compute exactly.

Training without Explicit Normalization

EBM training sidesteps Z by using contrastive methods. The gradient of the log‑likelihood splits into a positive phase (average gradient over data samples) and a negative phase (average gradient over samples drawn from the model distribution). The negative phase can be approximated with short Markov‑chain Monte Carlo (MCMC) runs, as in Contrastive Divergence (CD), which initializes the chain from real data and runs only a few steps.

Formally, the gradient is: ∇θ L = E_{data}[∇θ E(x)] - E_{model}[∇θ E(x)] where the first expectation lowers energy on data points and the second raises energy on model‑generated points, gradually aligning the model distribution with the data distribution.

Inference with a Trained EBM

Inference is an optimization problem: fix the observed variables (e.g., X) and search over possible Y values to find the configuration that minimizes the total energy. This yields a natural ranking: the lowest‑energy Y is the prediction. The approach extends to classification, ranking, and detection tasks, but it can be computationally expensive because it requires searching the output space.

Optimization Strategies

If Y is continuous and the energy surface is smooth, gradient‑based optimization can be applied directly.

If Y is discrete, the energy can be expressed as a factor graph and solved with min‑sum or dynamic programming (e.g., Viterbi) when the factorization permits.

When exact optimization is infeasible, approximate methods that replace the true energy with a surrogate are used.

Advantages of EBMs

EBMs belong to the exponential family, allowing the reuse of many statistical‑physics tools such as free energy and variational approximations. Because they do not require explicit normalization, they can model distributions that are intractable for conventional probabilistic methods, handle multi‑modal outputs, and avoid the “winner‑takes‑all” bias of softmax.

EBMs can be viewed as a Product of Experts: the total energy is the sum of several smaller expert energies, which corresponds to a factor graph representation. This modularity makes it easier to design complex models by combining simple sub‑energies.

Historical Context

The idea dates back to Hopfield’s 1982 recurrent neural network, which stored patterns as low‑energy attractors. Hopfield networks perform associative memory by iteratively lowering the system’s energy until a stable state is reached. Restricted Boltzmann Machines (RBMs) later introduced stochastic hidden units, enabling richer representations while retaining tractable training via contrastive methods.

Recent Directions

LeCun’s post‑Meta venture, AMI Labs, promotes EBMs as a path toward “world models” that evaluate entire output sequences holistically rather than token by token. Projects such as JEPA (Joint Embedding Predictive Architecture) and the Kona model incorporate EBM‑inspired energy terms to guide reasoning in latent spaces, achieving strong performance on tasks like Sudoku solving.

Overall, EBMs provide a flexible alternative to probability‑based models, offering a principled way to train and infer with unnormalized energies, leverage physics‑inspired concepts, and potentially overcome fundamental limitations of current LLM architectures.