Breaking the UED Bottleneck: PACE Locates the Reinforcement‑Learning Zone of Proximal Development

When training reinforcement‑learning agents, levels that are too easy provide no new learning signal, while overly hard levels waste budget on ineffective exploration; the most valuable environments lie in the "zone of proximal development"—just beyond the agent's current capability.

Unsupervised Environment Design (UED) addresses this by dynamically generating, selecting, or replaying levels instead of using a fixed dataset, but a core challenge remains: how to reliably identify which levels truly advance learning.

Existing UED scores (regret, GAE, MaxMC, marginal benefit) either rely on indirect solvability gaps or require costly extra rollouts, leading to high variance.

PACE: Parameter Change for Unsupervised Environment Design

PACE proposes a direct metric: if a level causes a meaningful change in the policy parameters after a local update, it has contributed real learning progress. The method derives a first‑order Taylor expansion of the objective improvement, assuming the update follows the local gradient, yielding the approximate relation

which shows that the objective gain is proportional to the squared norm of the induced parameter change. Consequently, PACE defines the level score as

where the norm of the parameter change reflects the training value of the level.

The PACE workflow alternates two phases:

Level scoring : generate a candidate level, collect a temporary rollout with the current policy, perform a provisional parameter update, compute the score using the above formula, and insert the level into a buffer (replacing the lowest‑scoring entry if the buffer is full).

Policy training : sample levels from the buffer according to their scores (higher‑scoring levels are replayed more often) and use them for actual policy updates.

This loop continuously enriches the buffer with high‑impact levels and drives the curriculum to evolve with the agent’s ability.

Experimental Results

PACE was evaluated on MiniGrid and Craftax. In MiniGrid, agents were trained on a set of training mazes and tested zero‑shot on 12 unseen human‑designed levels. PACE achieved higher success rates and lower variance on complex levels (Labyrinth, Maze3) compared to baselines DR, PLR, PLR₂, and ACCEL. Using the rliable library, PACE attained an IQM of 0.964 versus the best baseline PLR at 0.808 , and reduced the Optimality Gap to 0.172 .

On Craftax, a JAX benchmark for open‑ended RL, PACE was trained with ~1B environment interactions (Craftax‑1B setting). Evaluated on 20 unseen levels, PACE achieved the highest average episodic reward, surpassing DR and PLR baselines.

Conclusion and Outlook

Accurately identifying levels that drive genuine learning progress is crucial for adaptive RL curricula. PACE leverages the simple, low‑variance signal of policy‑parameter change to evaluate environments, avoiding the bias and high‑variance of proxy metrics and extra rollouts, and offers a scalable path toward more stable and extensible self‑adapting training curricula.