Breaking the Traditional UED Bottleneck: Using RL to Precisely Locate the Zone of Proximal Development

Problem Motivation Training reinforcement‑learning agents faces a dilemma: overly easy levels yield no new learning, while overly hard levels cause wasted exploration. Effective training requires environments that sit in the agent’s “zone of proximal development,” just beyond current capability.

Unsupervised Environment Design (UED) addresses this by dynamically generating, selecting, or replaying levels instead of using a fixed dataset. Existing UED methods rely on indirect scores such as regret, GAE, or Monte‑Carlo returns, which do not directly measure actual learning progress.

PACE Overview PACE (Parameter Change Environment Design) proposes a direct learning‑progress signal: if a level causes a meaningful change in policy parameters, it is valuable. The method derives a score from the squared norm of the parameter change using a first‑order Taylor expansion of the objective function.

The resulting score is proportional to the squared norm of the induced parameter change, directly reflecting realized learning progress.

Algorithmic Process PACE operates in two alternating phases:

Level Scoring : Generate a candidate level, collect trajectories with the current policy, perform a temporary policy update to compute the parameter change, and calculate the score. The level is added to a buffer if space permits; otherwise it replaces the lowest‑scoring level when its score is higher.

Policy Training : Sample levels from the buffer according to their scores (higher‑scoring levels are replayed more often) and use them for actual policy updates.

This loop continuously refines the curriculum, keeping levels that most effectively drive policy improvement.

Experimental Evaluation – MiniGrid PACE was tested on MiniGrid, measuring zero‑shot transfer to 12 unseen human‑designed levels. Compared to baselines DR, PLR, PLR<sup>† , and ACCEL, PACE achieved higher success rates and lower variance on complex levels such as Labyrinth and Maze3. Using the rliable library, PACE attained an IQM of 0.964 versus the best baseline PLR’s 0.808 , and reduced the Optimality Gap to 0.172 , indicating more stable overall generalization.

Experimental Evaluation – Craftax On the open‑ended Craftax benchmark (≈1 B environment interactions), PACE achieved the highest average episodic reward across 20 unseen levels, surpassing DR, PLR, PLR<sup>† , and ACCEL under the same training budget.

Conclusion and Outlook By using policy parameter change as a simple, low‑variance, and computationally cheap signal, PACE directly ties environment evaluation to realized learning progress, mitigating the bias and high variance of proxy metrics and offering a scalable approach for adaptive curriculum generation in reinforcement learning.