How Anthropic Scales Code Review with AI: 90% Automated, 40% Fewer Rollbacks

A Change's Lifecycle: Automated Gates, Human Attention at Risk Points

The article opens with a striking metric: from June to September, merged PRs in the Claude Code repository more than doubled while rollback rate dropped over 40%. The team did not eliminate review; they shifted most "reading" to AI and reserved human attention for where risk actually lives.

A change passes through four stages:

Pre-PR: Claude runs tests, build, and linter, then a fresh-context sub-agent re-reviews the diff.

On PR: Required CI, Claude code review, independent security review, plus a risk-graded human approval gate.

At Merge: Merge queue re-runs all checks on latest main to prevent two individually-passing PRs from breaking trunk together.

Post-Merge: Feature flags roll out to internal staff first, then a latest channel, then stable; instant rollback remains an option.

Seven Layers, Not Gates: Each Catches Different Failures

The system's essence is "layers" not "gates" — each layer catches some issues, misses others, and almost nothing penetrates all layers; the remainder is handled by rollback. Three design details stand out:

Review standards live in files. CLAUDE.md tells Claude the project stack, conventions, and goals; it is loaded on every review. REVIEW.md, written by code owners, describes what failure looks like in their area and demands reviewers cite file and line numbers as evidence rather than guessing by name.

AI review conclusions must pass a vote. Multiple agents hunt bugs in parallel; a second set of agents votes on each finding. Only findings with sufficient votes are emitted. Pure style or cleanup opinions are discarded — these filtering strategies were A/B tested on the team's own PRs. Result: PRs receiving substantive review comments rose from 16% to 54% after AI review introduction.

Human approval is risk-graded, and the grade is determined by the change, not the author. High-risk areas (billing, auto-update, CI workflows) always require senior sign-off regardless of who wrote the code.

Incidents Are the Evaluation Set

How does the reviewer improve? The team collects historical PRs that caused incidents, along with their fixes and rollbacks, asks code-familiar engineers to confirm each bug, and builds an evaluation set. The reviewer runs against this set daily. Prompt changes, tool swaps, model upgrades — all are judged by whether the score goes up.

Noise is measured pragmatically: each comment is classified by the author's actual reaction — fixed, false positive, intentional, or don't care. The written priority is "catch real bugs first, then reduce review rounds." Noise costs more in the agent-as-author era: a human dismisses a nitpick with a wave; an agent obediently submits another commit, triggering another review round.

Reviews Must Converge

An extreme case: a CLI PR saw 50+ rounds of back-and-forth between agent author and agent reviewer overnight. git blame revealed ~59% of the repeatedly flagged code existed since round one — the reviewer was not finding new bugs each round, just resurfacing old ones.

The team added convergence mechanics: cap at three rounds; thereafter only re-review new diffs; fix bugs by category in bulk; do not re-emit already-published findings. The effect was immediate and dramatic (internal PR review-round distribution shifted sharply left).

Five Places Humans Stay in the Loop

Pulling a senior engineer out of deep work for a trivial error costs far more than an automated review. Human decision points are collapsed to five:

Whether to do the thing at all: direction, taste, scope — machines cannot supply these.

High-risk code: billing, auto-update, CI workflows.

Architecture: changes that become patterns others will copy.

Reading evidence: test output, commands run, screenshots, review findings — if you cannot say what a check would miss, do not rubber-stamp.

Owning the outcome: every shipped change has a specific person accountable, regardless of who typed the code.

No One Carries the Whole Codebase Anymore

Review's second traditional role is building shared codebase understanding. With machines doing most reading, Anthropic's answer is on-demand generation: when you need to understand a module, ask Claude to write an explainer for that slice — minutes later you have a document reflecting the code as it stands today. Some go further, having Claude build a tiny visualization tool tuned to their learning style.

Reading a copy-change PR line by line teaches almost nothing; a targeted explainer for the module you are about to modify teaches a lot.

Source:

https://claude.dev/blog/solving-code-review-at-anthropic/