DR-Rubric: Deep Research as Rubric for Open-Ended Reinforcement Learning

Xiaohongshu and Fudan University introduce DR-Rubric (Deep Research as Rubric), a framework that treats rubric construction itself as a research problem for open-ended reinforcement learning. The work has been accepted at NeurIPS 2026, with paper and code publicly available.

Background: Rubrics Need Research

Rubrics decompose a holistic quality judgment into fine-grained, interpretable dimensions, providing denser training signals for tasks lacking unique ground-truth answers. Existing approaches either rely on costly expert-written rubrics or prompt models to generate rubrics directly, which often yields only generic criteria (e.g., clarity, completeness) and misses task-specific knowledge constraints.

DR-Rubric's core insight: constructing a rubric is itself a research problem . Judging correctness, coverage, and insight requires discovering, collecting, and synthesizing relevant knowledge — a process well-suited for Deep Research.

Core Method: Evidence Collection Then Constraint Synthesis

The framework comprises three sequential components: automatic rubric generation, rubric-based reinforcement learning, and bootstrapped rubric generation. Using the example "How much sugar should one consume daily?", the model first retrieves relevant guidelines, clarifies different sugar-intake concepts and applicable scopes, then forms evaluation standards for accuracy, concept distinction, and practical guidance.

Step 1: Evidence Elicitation

Given a task question p, external toolset T, and interaction budget k, the generation model performs multi-round retrieval and verification to build an evidence trail containing domain facts, structural constraints, and common failure modes. The process iterates "query construction → evidence acquisition → query refinement," with later queries building on prior findings. The goal is to uncover what knowledge is needed to judge the question, not to produce the final answer. This differs from standard RAG: retrieval serves evaluation-need discovery, and multi-round exploration helps recover missing key constraints.

Step 2: Constraint Synthesis

The model distills the evidence into atomic, independently checkable constraints forming the rubric R(p). Constraints can be positive (e.g., "Answer must cover at least two efficient attention methods") or negative (e.g., "Answer must not claim standard attention complexity is O(n log n)"). Both forms coexist in the same constraint set without categorical distinction or differential weighting. n(max) limits the number of constraints to avoid redundancy and noise. Crucially, each constraint traces back to specific information in the evidence trail, grounding the scoring standard in external facts.

Step 3: Rubric to GRPO Reward

A judge model independently checks each atomic constraint. A satisfied constraint scores 1, otherwise 0; the final reward is the arithmetic mean across all constraints. This yields partial rewards for partially satisfied constraints, providing more informative feedback than a task-level binary score. Training uses GRPO: for each question, G=4 responses are sampled; the relative advantage is computed as (reward − group mean) / group standard deviation. The policy update combines a clipped probability-ratio objective (ε=0.28) with a KL penalty (β=0.001) to constrain deviation from the reference model.

Bootstrapping: Model Generates Its Own Rubrics

To reduce reliance on external frontier models (GPT-5, Gemini) for rubric generation, the trained policy model takes over the research and synthesis roles for subsequent training rounds. This closed loop still requires external retrieval tools and a judge model. Experiments show bootstrapping does not guarantee monotonic improvement.

Experimental Results: 1,000 Samples, Reward Quality Matters

Experiments cover six benchmarks in two categories: agent tasks (ResearchQA, DeepResearchBench, LocalSearchBench) and expert reasoning (GPQA, MMLU-Pro, MMLU). Training starts with 1,000 ReAct-format trajectories for SFT on Qwen3-8B, yielding a shared Qwen3-8B-SFT initialization, followed by GRPO with rubric rewards. GPT-5 and Gemini variants each use 1,000 RL samples; bootstrap rounds BS-2 and BS-3 cumulatively use 2,000 and 3,000 samples respectively (not distinct questions). Hyperparameters: learning rate 5e-6, 5 epochs, 16 H800 GPUs. Retrieval uses a shared static document snapshot for reproducibility; appendix validates with live web search.

Key results (Table 2):

GPT-5 rubric: ResearchQA 73.7 (+3.8 over best baseline), DeepResearchBench 43.0 (+3.2).

Gemini rubric: LocalSearchBench 40.0 (+2.9), GPQA 57.3 (+1.3).

Bootstrap BS-3: MMLU-Pro 78.0 (+3.7), MMLU 85.3 (+1.5).

Figure 2 compares training data volumes: baselines report 16K–90K samples, while DR-Rubric achieves strong performance with far fewer samples, suggesting better reward signal design is a promising direction. The three rubric sources show complementary strengths: GPT-5 favors agent-task coverage breadth; Gemini balances both task types; bootstrap excels on general reasoning benchmarks. Scaling experiments with 14B and 30B-A3B models confirm advantages on ResearchQA and LocalSearchBench.

Source Ablation: Best Rubric Depends on Task

Fixing Qwen3-8B-SFT, GRPO, and atomic rubric reward, only the rubric source varies. A key finding: benchmark-provided rubrics are not the upper bound for training effectiveness . Even when training rubrics align with test evaluation protocols, GPT-5, Gemini, and bootstrap variants surpass them on multiple benchmarks. This indicates rubrics matching test metrics may not be optimal for learning; coverage, granularity, and structure all affect feedback quality.

Bootstrap Dynamics: Performance Dips Then Rebalances

Bootstrap average scores follow a non-monotonic trajectory: BS-1 60.8 → BS-2 60.2 → BS-3 61.2 (Figure 3). BS-1 rubrics resemble benchmark style with broad constraints, yielding strong reasoning supervision (GPQA 57.0, MMLU 84.0) but weaker agent performance. BS-2 constraints become more specific with more thresholds, while reasoning scores drop. BS-3 constraint length contracts, vocabulary diversity rebounds, and MMLU-Pro/MMLU reach 78.0/85.3. This trajectory resembles a search in reward-structure space balancing coverage, specificity, and reasoning transfer — not a steady improvement across all tasks.

A fresh-start control (re-initializing from Qwen3-8B-SFT each round using only the current-round rubric) replicates the BS-2 dip and matches BS-3 cumulative results, confirming later-generated rubrics work on the shared initialization. This does not imply the cost of prior training disappears.

Rubric Structural Changes

Table 3 and Figure 3 show structural statistics: GPT-5 rubrics have many dimensions (avg 6.50) and short constraints (avg 4.8 words, 24.39 constraints). Bootstrap rubrics do not converge to a single style; dimensions decrease, constraints lengthen, and verifiable proportion spikes at BS-2 then stabilizes. Word-level Jaccard similarity between bootstrap rounds is only 0.13–0.15; between GPT-5 and bootstrap ~0.03, indicating substantially different content, not minor rewrites. Structural shifts correlate with performance changes, offering clues for interpreting capability profiles.

Case Study: Rubric Style Affects Search and Synthesis

On ResearchQA, multi-dimensional rubrics encourage models to query prevalence, clinical features, and treatment outcomes simultaneously rather than focusing on the most salient aspect. BS-3 trajectories show domain-filtered queries (.gov, .org, .edu), reflecting attention to evidence hierarchy. Even with identical search queries, final scores can differ (0.500 vs 0.875); short specific constraints map to discrete fact extraction, while long generic constraints may bias toward narrative and miss concrete items. These cases link training feedback to behavioral differences but do not establish causal attribution for any single rubric property.

Bootstrap Limits: Reward Rises While Model Collapses

Self-generated rubrics do not enable unlimited self-improvement. A stress test on Qwen3-30B-A3B pushed bootstrap to BS-5 (Figure 4). The paper introduces Reward Polarization — the proportion of extreme scores (0 or ≥0.99). At BS-3, polarization reaches 0.195, exceeding a 0.15 stopping threshold. Continuing to BS-4, agent-task average drops from 51.3 to 22.7; BS-5 degrades further. Meanwhile internal reward rises from 0.24 to 0.66, creating a divergence between reward and true task quality.

Stopping rule: halt bootstrap when polarization > 0.15. In this experiment, polarization spikes one iteration before task performance collapses. GRPO amplifies degradation: when all four rollouts per question receive similar scores, within-group relative advantages lose discriminative power, weakening the learning signal.

Comparing model sizes: 14B degradation accompanies reward decrease; 30B degradation accompanies reward increase and output anomalies — different mechanisms but both exhibit extreme-score clustering. Practical advice: stop excessive bootstrap early or reintroduce external high-quality rubrics for recalibration; this heuristic needs broader validation and is not a universal rule.

Appendix Highlights

Generation prompt division: Phase 1 casts the model as an evaluation-framework research analyst, outputting query decomposition, synthesized fact base, key quality dimensions, and edge cases. Phase 2 explicitly requires designing an evaluation framework (not answering the original question) with thresholds aligned to evidence-report facts.

Bootstrap correction example: A non-valvular atrial fibrillation stroke-prevention case shows an early rubric containing inappropriate treatment advice; later rounds correct that entry and add dosage and perioperative information. This illustrates rubric content can change during bootstrap, but does not imply all generated standards self-correct.

Research interaction budget: Phase 1 allows up to 10 interactions with early stopping; evidence trails average 12.4 pieces, typically stabilizing in 6–8 steps. Implementation notes report GPT-5 and Gemini rubrics average 11.4 and 10.8 constraints respectively, with negative-constraint proportions ~27% and ~22% (different statistical scope from Table 3).

Summary and Outlook

DR-Rubric transforms rubric construction from a static pre-prepared template into an active research process that acquires external evidence. For open-ended tasks, this makes reward sources more concrete: which facts must be covered, which concepts clarified, which errors avoided — each fed back to the model as a checkable item. Experiments demonstrate high-quality rubrics' value under small data budgets and the feasibility of models taking over rubric generation. More critically, bootstrap changes not only model parameters but also the yardstick for the next training round. Maintaining yardstick reliability, deciding when to stop iteration, and recalibrating with external evidence remain open questions for this line of work.