Why AI Can’t Plan: LeCun’s Team Shows Time Is Curved in Latent Space

Background

Yann LeCun, a pioneer of deep learning, has emphasized a long‑term research direction: building "world models" that can understand and plan in the real world. Recent work from his team at Meta and NYU investigates a fundamental question for such models: what structure must the latent representation have to support planning?

The Curvature Problem

Although pretrained visual encoders capture rich semantic information, the trajectories they produce in latent space over time are typically highly curved. This curvature creates two fatal issues:

Distance failure: Euclidean distance no longer reflects the true difficulty (geodesic distance) of reaching a target state.

Planning instability: Gradient‑based planners get trapped in local minima on a warped landscape, causing agents to “spin in place” or lose logical continuity.

Temporal Straightening via Curvature Regularizer

Inspired by the neuroscience hypothesis of perceptual straightening, the authors introduce a geometric constraint called the Curvature Regularizer . The regularizer forces three consecutive latent embeddings \(z_{t-1}, z_t, z_{t+1}\) to have nearly identical displacement vectors, i.e., the angle between \(z_t - z_{t-1}\) and \(z_{t+1} - z_t\) should be close to zero. The loss can be expressed either as the squared difference of the vectors or as one minus the cosine of the angle between the unit vectors.

This term compels the encoder to map raw visual inputs into a smoother space where state transitions evolve linearly.

Training and Planning Procedure

During training, the model minimizes two objectives: (1) a prediction loss between the encoder’s latent output and a stop‑gradient target, and (2) the curvature loss that penalizes bent trajectories. At inference time, a learned predictor rolls out the latent dynamics and selects actions that minimize the cost between the predicted final state and the goal in the straightened space.

Experiments on Teleport‑PointMaze

The authors evaluate their approach on a challenging environment called Teleport‑PointMaze, where stepping on the right wall instantly teleports the agent to the left wall. This abrupt jump severely breaks traditional encoders such as DINOv2.

Heat‑maps of Euclidean distance to the goal show that, without straightening, the latent space is fragmented and fails to reflect the maze topology. After applying the curvature regularizer, the distance field becomes smooth and aligns with the true geodesic distance, allowing simple Euclidean distance to guide the agent through the teleport portals.

The straightened latent space yields higher success rates in open‑loop planning and more accurate distance estimates, confirming that reducing curvature directly improves planning performance.

Implications

The study suggests that a good latent space for planning should produce near‑linear temporal trajectories. This insight may influence future research in robot control, video‑based world models, and autonomous driving, where reliable planning from visual inputs is essential.