One Layer Is Enough: Single‑Layer RL Beats Full‑Parameter Training Across Models, Tasks, and Algorithms

Background and Motivation

Reinforcement learning with verified rewards (RLVR) has become a core component of post‑training large language models, improving mathematical reasoning, code generation, and decision‑making. Existing RL post‑training methods update all Transformer layers, implicitly assuming each layer contributes equally to RL gains. This work questions that assumption.

Layer Contribution Metric

The authors define a simple metric, Layer Contribution C(k), for a model with L layers: freeze all parameters except layer k, train only that layer with RL, and compare the performance gain to the full‑parameter RL baseline. C(k)=1.0 means single‑layer training matches full‑parameter RL; C(k)>1.0 means it surpasses it.

Experimental Setup

The study evaluates 7 models spanning two families (Qwen3 1.7B/4B/8B, Qwen2.5 1.5B/3B/DeepSeek‑Distilled‑7B) with three RL algorithms (GRPO, Dr. GRPO, GiGPO) across three task domains (math reasoning, code generation, agent decision‑making). For Qwen3‑1.7B‑Base, 12 benchmarks covering math, code, reasoning, and language are used. All experiments share identical hyper‑parameters (learning rate, batch size, KL coefficient, clip range, epochs) and use a learning‑rate‑ablation to confirm fairness.

Finding 1: Middle Layers Dominate RL Gains

Across all models, layers in the 40‑60% depth range consistently achieve the highest contribution scores, while early and late layers contribute far less. This pattern holds regardless of model size, RL algorithm, dataset, or task, indicating an intrinsic structural property of pretrained LLMs.

Finding 2: Single‑Layer Training Can Surpass Full‑Parameter RL

For every model, the best single layer attains C(k) ≥ 1.0, meaning it matches or exceeds the full‑parameter RL baseline. For example, on Qwen3‑1.7B‑Base, training only Layer 10 yields a score of 51.8% versus 50.8% for full‑parameter RL (C = 1.14), while the weakest Layer 24 reaches only 46.1% (C = 0.28), a >4× gap.

Practical Strategies Based on Layer Contribution

Strategy 1: Layer‑Adaptive Learning Rate – Increase the learning rate for high‑contribution layers (1e‑5) while keeping others at the default (5e‑6). This yields consistent gains (e.g., Qwen3‑1.7B: 53.70% vs 50.82% baseline).

Strategy 2: Selective Layer Training – Train only the top‑k layers by contribution and freeze the rest. Results: Qwen3‑8B top‑10 layers achieve 69.11% vs 66.43% baseline (+2.68%). Training the lowest‑contribution layers harms performance.

Strategy 3: Zero‑Analysis Heuristic – Without any contribution data, simply train the middle five layers (e.g., Layers 11‑15 for a 28‑layer model). This heuristic matches or exceeds contribution‑guided selection across model sizes.

Deeper Analysis: Parameter‑Space Quality vs. Magnitude

Two observations challenge the notion that weight‑change magnitude explains contribution:

Full‑parameter RL induces uniformly small L2 weight changes (0.5‑0.8) across layers, yet middle layers contribute disproportionately.

Single‑layer training causes similar L2 weight changes (0.8‑1.0) for both high‑ and low‑contribution layers, but only the former yield large performance gains.

Thus, layer contribution reflects the effectiveness of a layer’s parameter sub‑space for capturing RL improvements, not merely the amount of weight change.

Conclusion

The study uncovers a previously unrecognized structural property of RL post‑training: (1) RL gains are highly concentrated in a small set of middle layers; (2) training a single such layer can match or exceed full‑parameter RL; (3) simple, contribution‑guided strategies consistently outperform standard full‑parameter RL. These insights provide new directions for understanding and improving RL fine‑tuning of large language models.