One Layer Is Enough: Single‑Layer RL Beats Full‑Parameter Training Across Models, Tasks, and Algorithms

Background and Motivation

Reinforcement‑learning with verified rewards (RLVR) has become a core component of large language model (LLM) post‑training, improving mathematical reasoning, code generation, and agent decision‑making. Existing RL post‑training methods (e.g., GRPO, Dr. GRPO, GiGPO) update all Transformer layers under the implicit assumption that each layer contributes equally to RL gains. This work questions that assumption.

Layer Contribution Metric

To quantify each layer’s contribution, the authors define a simple metric called C(k) = (S_k - S_{base}) / (S_{full} - S_{base}), where S_{base} is the pre‑training score, S_{full} is the score after full‑parameter RL, and S_k is the score after training only layer k. C(k)=1.0 means single‑layer training matches full‑parameter RL; C(k)>1.0 indicates it exceeds full‑parameter RL. The gradient is still back‑propagated through the whole network, but parameter updates are confined to the target layer.

Experimental Setup

The authors conduct systematic layer‑wise experiments on seven models covering two families (Qwen3 1.7B/4B/8B and Qwen2.5 1.5B/3B/DeepSeek‑Distilled‑7B), three RL algorithms (GRPO, Dr. GRPO, GiGPO), and three task domains (mathematical reasoning, code generation, agent decision‑making). Evaluation uses up to twelve benchmarks per task, with a unified hyper‑parameter protocol (identical learning rate, batch size, KL coefficient, clip range, and training epochs) to ensure fair comparison.

Key Findings

Finding 1 – RL gains are highly concentrated in middle layers. Across all seven models, layers in the 40‑60 % depth range consistently achieve the highest C(k) values, while layers near the input or output contribute little. The ranking is stable across model sizes, RL algorithms, datasets, and tasks (Spearman ρ = 0.76–0.59, p < 0.001).

For example, on Qwen3‑1.7B‑Base, training only Layer 10 yields a score of 51.8 % (Math + Code + Reasoning), giving C=1.14, whereas the worst Layer 24 reaches only 46.1 % ( C=0.28), a >4× difference.

Finding 2 – Single‑layer training can surpass full‑parameter RL. In every model, the best single layer achieves C(k) ≥ 1.0. On Qwen3‑1.7B, the best layer improves the benchmark average from 44.1 % (pre‑training) to 51.8 % (single‑layer), exceeding the full‑parameter GRPO result of 50.8 %.

Practical Strategies Guided by Layer Contribution

Based on the observations, three simple strategies are proposed:

Layer‑adaptive learning rate: Increase the learning rate for high‑contribution layers (1e‑5) while keeping others at the default (5e‑6). This yields consistent gains (e.g., Qwen3‑1.7B: 53.70 % vs 50.82 % baseline).

Selective layer training: Freeze all layers except the top‑k high‑contribution layers. Training only the top‑10 layers of Qwen3‑8B reaches 69.11 % (vs 66.43 % full‑parameter), a +2.68 % absolute improvement.

Zero‑analysis heuristic: Without any contribution analysis, train the middle five layers (e.g., layers 11‑15 for a 28‑layer model). This heuristic already outperforms full‑parameter RL on all model scales.

Analysis of Parameter Changes

The authors measure L2 norm changes ‖Δθ_k‖₂ for each layer under full‑parameter RL and under single‑layer RL. Two observations emerge:

Full‑parameter RL produces uniformly small weight changes across layers (0.5‑0.8), yet contribution is highly uneven.

Single‑layer RL causes similar magnitude changes (0.8‑1.0) for both high‑ and low‑contribution layers, but only high‑contribution layers yield large performance gains.

This decoupling indicates that layer contribution reflects the intrinsic effectiveness of a layer’s parameter sub‑space for capturing RL improvements, not merely the amount of weight change.

Conclusion

The study reveals a previously unrecognized structural property of RL post‑training: (1) RL gains are highly concentrated in a small set of middle layers; (2) this layer‑contribution pattern is consistent across model families, RL algorithms, datasets, and task domains; and (3) the insight directly enables training optimizations that consistently outperform standard full‑parameter RL.