Anthropic’s Generator‑Critic Approach for Reliable Test‑Case Evaluation

When an Agent writes a test case and then immediately checks its own work, the review often misses obvious defects because the generation and evaluation share the same context and reasoning chain, creating an illusion of self‑validation.

This self‑review problem stems from the Agent playing two roles within a single memory space; the Critic sees the output through the same explanatory framework that produced it, so it cannot objectively spot omissions, boundary errors, or hidden assumptions.

Anthropic addresses the issue with an Outcomes evaluation mechanism that runs the scoring model in a completely separate context. The Critic only receives the final output and a rubric, never the intermediate reasoning, ensuring an unbiased assessment.

The Generator‑Critic workflow can be visualised as:

┌──────────┐
│ 任务输入 │
└────┬─────┘
     │
     ▼
┌─────────────────────┐
│    Generator        │
│ 执行任务，生成输出 │
│ （独立上下文）      │
└──────────┬──────────┘
          │ 输出结果
          ▼
┌─────────────────────┐   ┌──────────────┐
│      Critic          │◄──│   Rubric    │
│ 独立评分            │   │ 评分标准   │
│ （不看推理过程）    │   └──────────────┘
└──────┬──────┬───────┘
       │      │
       达标   不达标
       │      │
       │      ▼
       │ ┌──────────────────────┐
       │ │ 附上问题说明，交回重试 │
       │ └──────────┬───────────┘
       │            │
       │            └──────► 回到 Generator
       ▼
┌──────────────┐
│   最终输出   │
└──────────────┘

The crucial element is the rubric. A vague rubric (“is the quality good?”) provides no actionable basis for the Critic. Effective rubrics are concrete, checkable, and include criteria such as coverage of positive, negative, and boundary scenarios; explicit expected results for each case; and avoidance of duplicate logic.

Does the test cover forward, reverse, and edge‑case scenarios?

Is each case’s expected outcome clearly defined?

Are redundant test steps eliminated?

When the rubric is precise, the Critic can return deterministic feedback (e.g., missing boundary value, ambiguous expectation, unreproducible step). Conversely, a poorly written rubric limits the overall quality regardless of how sophisticated the Critic is.

Retry mechanisms must be bounded. Re‑running the Generator after a failed rubric check only amplifies the original mistake if the rubric itself is flawed. Unlimited retries can cause infinite loops when the underlying task definition is the problem. A practical approach sets a maximum retry count, after which the issue is escalated to human review, and distinguishes between rubric strictness, task ambiguity, or insufficient input.

In summary, a robust Agent pipeline separates execution and evaluation into distinct contexts, relies on a concrete, testable rubric, and enforces explicit retry limits. Designing with these three questions—independent reviewer, judgeable rubric, and clear retry ceiling—turns the Generator‑Critic pattern from a prompt trick into a controllable engineering structure.