Why Reinforcement Learning Unlocks Hierarchical Reasoning in LLMs: The HICRA Breakthrough

Recent observations in large‑model inference have highlighted three puzzling phenomena—sudden performance jumps (Aha moments), length scaling where longer reasoning chains improve accuracy, and non‑linear entropy dynamics during training—but no unified explanation existed.

Researchers from HKUST, Tsinghua, and Waterloo discovered that reinforcement learning (RL) drives a hierarchical evolution: models first master low‑level execution skills (e.g., arithmetic, token formatting) and later acquire high‑level planning abilities, creating a "high‑level planning – low‑level execution" structure.

Based on this insight, the team proposed HICRA (Hierarchical Reasoning via Concentrated Reward Allocation), a training paradigm that focuses gradient fire on the crucial planning tokens instead of distributing it uniformly across all tokens. The method defines a set of planning tokens τ and amplifies their advantage term by a factor α (set to 0.2 in the paper), leaving execution tokens largely untouched.

This targeted pressure yields two immediate effects: (1) strategic policies are reinforced and solidified faster, and (2) exploration is preserved because sub‑optimal strategies are not instantly discarded. Empirically, HICRA causes models to reach the Aha moment earlier and improves overall reasoning ability on both text and multimodal tasks.

Training dynamics further reveal that early in training the entropy of execution tokens drops quickly, while the semantic entropy of planning tokens rises steadily, mirroring validation accuracy improvements. Traditional token‑level entropy fails to capture this strategic diversity, whereas semantic entropy directly reflects the expansion of the model’s policy library.

Extensive experiments were conducted on a suite of text reasoning benchmarks (AIME, AMC, Minerva, Olympiad) and multimodal reasoning suites (MathVista, MathVerse, MathVision) using models such as Qwen, Llama, MiMO‑VL, and Qwen‑VL. Across all tasks, HICRA consistently outperformed the baseline GRPO, with especially large margins on complex problems.

Beyond the empirical gains, the authors propose a three‑step practical recipe to make RL truly teach models "how to think":

Build a strategy‑chunk library (SGs) that enumerates high‑level reasoning actions such as "case split", "proof by contradiction", or "substitute known values".

Monitor semantic entropy instead of token‑level entropy to gauge the diversity of the emerging policy space.

Apply HICRA’s directed amplification in the RL pipeline, using an adaptive scaling factor α and KL regularization to keep the distribution stable while accelerating reasoning capability.

When these steps are combined, the learning signal is no longer wasted on low‑value execution tokens; it is concentrated on the strategic bottleneck, turning the previously accidental Aha moment into an inevitable transition point.

In conclusion, HICRA demonstrates that reinforcement learning naturally induces a hierarchical evolution from execution to planning, breaking the long‑standing reasoning bottleneck for both textual and multimodal large language models and paving the way for predictable, large‑scale reasoning breakthroughs.