Embedding Error Correction into the Policy Space: How Search‑R2 Redefines Search‑Enhanced Reasoning

Motivation and Problem Statement

Recent advances in large language models (LLMs) have relied heavily on scaling parameters and data, but when deployed for real‑world tasks such as research assistance, web search, or complex decision support, this scaling approach reaches its limits. In open‑ended environments, models must perform multi‑turn search and reasoning, and failures often stem not from insufficient reasoning ability but from the inability to handle errors that arise and propagate during the reasoning process.

Search results inevitably contain noise; an early mis‑retrieval can steer subsequent reasoning into an erroneous semantic space, leading to seemingly plausible yet incorrect answers. Existing training methods optimize only for the final answer correctness, giving identical feedback to both lucky successful trajectories and unreliable ones, which over time weakens the model’s constraint on intermediate errors and search quality.

Search‑R2: Actor‑Refiner Collaboration

The joint team from Tencent Hunyuan, MBZUAI, and Hong Kong Chinese University proposes Search‑R2: Enhancing Search‑Integrated Reasoning via Actor‑Refiner Collaboration . The core idea is to embed error correction directly into the policy space, allowing the model to recognize, locate, and fix errors during training rather than assuming a flawless reasoning chain.

The method consists of three tightly coupled modules:

Reasoning Generation (Actor) : Generates a full trajectory of search and reasoning steps, allowed to explore and even err.

Trajectory Judgment (Refiner) : Evaluates the entire trajectory, not merely the final answer, checking for semantic drift, entity misalignment, or evidence mismatch.

Error Localization : When a trajectory is deemed faulty, the system pinpoints the first substantive deviation—typically a specific search or reasoning operation that introduced noise.

Once the error position is identified, the prefix up to that point is retained, the erroneous suffix is discarded, and generation resumes from the error location. This “trimming” operation enables the reward signal to be back‑propagated precisely to where the mistake first occurred, encouraging the model to avoid the most damaging search errors.

To prevent the model from merely fixing the final answer while ignoring root causes, a process‑level reward is introduced. This reward measures the information density of retrieved evidence, rewarding high‑quality searches only when the final answer is correct, thereby making search quality a necessary but not sufficient condition for optimization.

Experimental Evaluation

The authors evaluate on two families of tasks: ordinary factual QA (requiring one or two retrievals) and multi‑hop QA (requiring repeated “search‑reason‑search” cycles). Datasets include HotpotQA, 2WikiMultiHopQA, and Bamboogle.

Results show stable improvements across all tasks, with markedly larger gains on multi‑hop QA. On Bamboogle, the relative accuracy increase exceeds 20 %. The advantage is attributed to effective suppression of error propagation rather than enhanced parameter memorization.

In a comparison against a rejection‑sampling baseline that doubles the sampling budget per question, the baseline still underperforms Search‑R2 even when operating with a larger budget, demonstrating that the improvement is not merely due to more sampling attempts.

Ablation Studies

Systematic ablations reveal that introducing only the mid‑trajectory error‑correction mechanism (without process rewards) already yields significant performance gains, confirming that locating and fixing key errors directly addresses the core bottleneck of search‑enhanced reasoning.

Adding the process reward that distinguishes high‑quality from low‑quality search results provides further improvement, indicating that explicit modeling of search quality offers a more stable optimization direction.

The full joint optimization of generation, judgment, and error‑localization modules achieves the best results on every benchmark, showing that error correction is not a static rule but a behavior learned and internalized during training.

Analysis and Theoretical Insight

The authors formalize the trimming capability as a necessary condition for overall performance improvement. By precisely back‑propagating rewards to the first error location, the model learns which search mistakes are most destructive and should be avoided.

The joint training shares parameters across all three modules under a single RL objective, treating the decision to trigger correction and the choice of correction point as policy actions. Consequently, even without explicit correction at inference time, the initial generated trajectories become higher‑quality.

Conclusion

Embedding error correction into the policy space provides a learning paradigm that aligns closely with real failure modes of search‑type agents. Instead of relying on more attempts, the approach emphasizes precise handling of failure paths, yielding robust improvements especially on challenging multi‑hop reasoning tasks.