Defining Standard Answers for Agent‑Era LLMs: A Rubrics Survey

When large language models (LLMs) move beyond simple Q&A to deep research, medical consulting, multimodal generation, and long‑horizon agent tasks, quality assessment shifts from single correctness to fine‑grained, explainable standards. Rubrics decompose a "good answer" into explicit, checkable items, providing a clear interface between human expectations and model behavior.

Why Rubrics Become Important

Early LLM tasks had clear, verifiable goals—accuracy for QA, test‑case passing for code, or final‑result checking for math—so scalar metrics sufficed. In open, high‑risk applications (research reports, medical or legal analysis, tool‑using agents), output quality depends on factuality, completeness, safety, reasoning process, evidence support, expression quality, and practical usability. Existing methods (reward models, RLVR, LLM‑as‑a‑Judge) each have limitations such as opacity, narrow applicability, or sensitivity to prompt phrasing.

What Rubrics Are

In education, a rubric is a scoring guide describing criteria and performance levels. For LLMs, a rubric set is a collection of natural‑language criteria, each with a description and optional weight. A judge model scores each item and aggregates the scores (average, weighted sum, or implicit fusion) into an overall evaluation.

Rubrics differ from related concepts: LLM‑as‑a‑Judge decides "who judges"; Rubrics define "what standards". Reward models output a single score, while Rubrics make the scoring basis explicit. RLVR relies on fully verifiable answers; Rubrics suit multidimensional, partially unverifiable tasks.

How Rubrics Are Constructed

The survey classifies construction methods into four paradigms, illustrated in Figure 2:

Direct generation : Given a task prompt, candidate answer, or reference evidence, an LLM directly generates a set of criteria. Low cost but may miss key dimensions and lacks validation of discriminative power.

Contrast generation : The model receives a high‑quality and a low‑quality answer, extracts their differences, and formulates criteria that explain the preference, yielding more discriminative rubrics.

Iterative optimization : A generate‑validate‑split‑filter loop refines rubrics; standards that fail to consistently distinguish preferences are split, duplicated items are removed, and the set becomes more atomic and reliable.

Online/co‑evolution : In reinforcement‑learning or agent settings, fixed rubrics can be quickly adapted. Researchers update rubrics based on policy rollouts, incorporating newly observed failure modes so that evaluation stays synchronized with model capabilities.

Rubrics for Policy Model Training

During training, rubrics translate complex quality requirements into optimizable supervision signals. Instead of a single preference label, rubrics indicate which dimensions perform well and which need improvement, which is especially useful for open‑ended generation and multi‑step agents.

The typical rubric‑based RL pipeline (Figure 3) is:

Provide input and model‑generated answer.

A judge model scores the answer on each rubric item.

Aggregate the multidimensional scores into a reward (e.g., via weighted sum) for PPO, GRPO, or similar algorithms.

The reward can be applied to final answers or to intermediate reasoning or tool‑use trajectories, which is crucial for deep‑research or tool‑using agents.

Simple scalar aggregation can be brittle; some dimensions act as vetoes (e.g., safety violations in medical QA make the whole answer unacceptable). Recent work explores learnable weights, veto mechanisms, saturation, curriculum training, and advantage‑estimation designs to address these issues.

Beyond post‑hoc scoring, some approaches feed rubrics into the generation process itself: the model first generates or reads rubrics, then plans its answer, or uses unmet rubrics as feedback for iterative rewriting.

Rubrics for Reward Model Training

Rubrics also enhance reward models by making their judgments interpretable and providing finer‑grained training signals. The survey groups related work into three categories (Figure 4):

Interpretability : Reward models expose the rubric criteria they used, allowing researchers to verify that the model follows the intended dimensions.

Fine‑grained supervision : Rubric‑level reference signals guide the model’s intermediate reasoning, or the model generates rubrics before judging, and the quality of those rubrics is itself evaluated.

Data quality improvement : Rubrics help filter out superficial cues (length, format) in preference data, focusing training on core quality dimensions.

Rubrics for Evaluation

Rubrics serve as explicit contracts for open‑ended task evaluation. The survey reviews benchmark usage for both general and domain‑specific tasks.

In general tasks, rubrics assess reasoning steps (math), information coverage and evidence (deep research), tool selection and execution order (agent benchmarks), and overall alignment. In specialized domains, rubrics address factual correctness and safety in medical QA, fact‑application and auditability in legal/financial tasks, and entity placement, temporal consistency, and visual hallucination detection in multimodal tasks (Figures 5 and 6).

Conclusion

The survey argues that Rubrics provide a unified, explicit quality interface for LLM training and evaluation. As LLMs advance toward open‑ended, high‑risk, and agentic applications, clear quality definitions become essential; Rubrics turn vague intuition about a "good answer" into a concrete, discussable, inspectable, and optimizable set of rules.

The Rules of the Game: A Survey of Rubrics for Large Language Models
Paper link 1: https://8421bcd.github.io/_pages/Rubrics_Survey.pdf
Paper link 2: http://playbigdata.ruc.edu.cn/dou/publication/Rubrics_Survey.pdf
GitHub repo: https://github.com/RUC-NLPIR/Rubrics_Survey