Skip to content
How Claude Code Verifies AI-Written Code: 39 Checks, Verify Skills, and Baseline Rules

How Claude Code Verifies AI-Written Code: 39 Checks, Verify Skills, and Baseline Rules

The Claude Code team shares their AI code verification process: selecting relevant checks from a 39-item reliability list based on code diffs, running verify skills to execute the application, using baseline branch review rules to prevent relaxed standards, binding results to specific commits, and escalating to humans after three failed fix attempts.

Architect
Architect
Architect

Introduction

For coding agents, we want them to not only write code but also run the program and iterate on failures. If we still have to manually review code and run tests after the agent finishes, the promised productivity gain disappears. On November 16, the Claude Code team published their verification approach (written by Addy Osmani). They report that roughly 90% of their code is written and reviewed by Claude. At that scale, manual re-checking would become a bottleneck.

A GitHub issue from April 9 illustrates the risk: a developer asked Claude to test business functionality, and it uncovered foreign-key migration, rounding assertion, and transaction handling problems. Claude marked those tests with @unittest.skip(...) and reported completion, violating the project rule against skipping failing tests. Only after the developer pressed further did it continue fixing. This shows that the agent's definition of "done" can differ from ours.

TL;DR

Turn recurring manual checks into runnable agent checks. The team selects 3–10 relevant checks from a 39-item reliability list based on the diff, then runs a verify skill to exercise the application. Your project can start with one frequently problematic feature.

Give "pass" a concrete meaning. Which rule set was used, which samples were tested, and which commit the results correspond to are all traceable. Rules modified in the current PR are not used to judge that same PR.

Enable humans to pick up where the agent left off. If a check fails after three repair attempts, the team hands it to a human. The failing sample and attempted fixes are preserved, reducing re-investigation; missed issues feed back into the checklist.

39 Reliability Checks

Suppose we add a workspace field to session records. New records write and read the field successfully, but yesterday's records lack it. On upgrade, reading transcript["workspace"] throws an error. This bug only appears when loading old records.

The Claude Code checklist includes a check: " --resume can load old sessions." The official guidance: when session storage or message types change, verify that new fields don't break loading of old records. The full 39 checks are not listed; only the old-session check is expanded. Each check documents: what changes trigger it, what to test, and which incident or PR it originated from. The code owners maintain this list. Claude reads the diff and picks the relevant 3–10 checks.

Figure 1: 39 checks, not all run every time (when session format changes, old records are checked. Redrawn from Claude Code team original.)
Figure 1: 39 checks, not all run every time (when session format changes, old records are checked. Redrawn from Claude Code team original.)

Key insight: pair each past pitfall with the condition that should trigger its check. A standalone post-mortem may be forgotten during coding; writing "if session format changed, test old records" lets Claude act automatically when it sees that diff.

In your project, start by recording one recent rework. For example, if restoring old records failed, keep a reproducible sample and note which future changes should trigger its test. The checklist becomes useful immediately and grows over time.

Running the Application

The team encodes how to start the app, interact with it, and interpret results into a verify skill stored under .claude/skills/ near the relevant code. Different parts need different verification: web changes use a Chrome extension, mobile uses an emulator, backend runs the service locally. Boris Cherny shared the same pattern: let Claude drive the UI and fix based on what it sees. For the old-session example, prepare a saved record, run resume, send a message, and confirm interaction continues.

They run low-cost checks first: before opening a PR, Claude runs tests, builds, and linters locally to catch issues early and reduce shared CI load. The verify skill itself must be reliable; flaky environments make it hard for Claude to know what to fix.

When assigning a task to an agent, also provide the startup command, sample location, and expected result . It can only improve if it knows what went wrong. This verification setup is reusable for future changes to the same feature.

Addy describes end-to-end, property-based, and differential testing. Using the old-session example:

End-to-end test: Run --resume with a pre-upgrade record, verify historical messages appear, send a new message, confirm continued interaction.

Property test: Use Hypothesis or fast-check to generate message lists of varying lengths and records missing optional fields; check that message content and order remain unchanged after read/write.

Differential test: Feed the same old record to both old and new implementations; compare restored messages and workspaces, ignoring irrelevant differences like timestamps, and verify intentional behavior changes against new requirements.

However, if generators only produce new-format records, thousands of runs won't hit missing- workspace cases. When comparing old and new implementations, if the old one had bugs, the new results should differ. Therefore, beyond test count, you must know what samples were used and what property was verified.

Same Code, Two Results

Even with a runnable app and samples, the code might be unfixed while the check criteria were relaxed. The Claude Code team prevents this by reading review rules from the target or default branch; rules modified in the current PR are excluded from this round's review.

A synthetic experiment demonstrates the effect. A temporary Git repo contains a simplified resume() function that returns a workspace name. It must satisfy:

assert resume({"workspace": "demo"}) == "demo"
assert resume({"messages": ["hello"]}) == "default"

The first assertion tests the new format (explicit workspace), the second tests the old format (missing field, returns default). Both are synthetic records testing only this function.

The baseline branch requires both checks. In a candidate branch, two changes are made: the code directly reads workspace without handling missing fields, and the rule changes the old-record check from "required" to "optional". The checker and samples live outside the repo.

With the same code and commit 0b4524a9, switching the rule source changes the outcome.

Figure 2: Code unchanged, why did the result change? (Synthetic experiment: code, checker, and samples unchanged; only rule source swapped. Old record only checked in second run.)
Figure 2: Code unchanged, why did the result change? (Synthetic experiment: code, checker, and samples unchanged; only rule source swapped. Old record only checked in second run.)

The first run shows PASS but outputs checks=1/2: only one of two checks executed. The old record readability was not checked at all.

In practice, the agent may not intentionally bypass checks; it might assume "this field always exists now" and write both code and tests that omit the old format. A second reviewer agent using the same relaxed rules would also miss it. Hence, the team prefers tuning the existing reviewer to continue from prior findings. For us, when opening a new review session, we must pass the original requirements and old samples to catch such omissions.

Restoring the baseline rule and changing the read to transcript.get("workspace", "default") passes both records: preserves the field when present, supplies default when absent. Simon Willison's Red/Green TDD emphasizes this order: first see the test catch the bug, then see the fix pass. If it's green from the start, we can't be sure the test actually exercised the fix.

Claude Code's best practices also recommend having Claude output the run command and its output. Seeing checks=1/2 tells us a check was skipped; a simple "all tests passed" would hide that.

Changing Rules Safely

Rules do become outdated. If the product drops support for an old session format, tests must change, but user data migration and version cutoffs must be planned together.

In Claude's cloud Code Review, new rules take effect only after REVIEW.md changes are merged. Modifications only in the current PR still use the target/default branch version for this round.

Thus, the agent can propose dropping support and write the code, but whether to change compatibility requirements is a separate human decision. Review uses already-merged rules, so difficulty of the current fix doesn't lower the bar.

The team's automated briefings use a similar split: human requirements live in read-only preferences, agent progress goes into state. New ideas during execution are recorded for later human approval.

Branch protection also matters: protected default branch main can be a rule source; other target branches need a compliant ruleset. If neither is protected, only the default branch root file is read; if protection status is unknown, no rules are read. After adding REVIEW.md, verify it was actually used this round.

Local and cloud differ: cloud reads CLAUDE.md and REVIEW.md; local /code-review reads only CLAUDE.md. A local pass doesn't guarantee REVIEW.md rules were checked.

In a team project, separate REVIEW.md from merge conditions. The former tells the model what to scrutinize; mergeability still depends on test results, approvals, and permissions. Even with unchanged rules, deleted samples or swapped checkers can cause missed checks. The earlier experiment kept checker and samples outside the repo to isolate the rule-source variable.

What about flaky tests? Addy notes the team forbids Claude from skipping, adding sleep, or re-running until pass. If root cause is unknown, the test is deleted and the reason recorded. The article doesn't detail who approves deletion or how coverage is restored. A remaining concern: if a test is deleted, old sessions must still open — what then verifies that?

Which Commit Passed?

The team binds review results to specific commits. "Pass" refers to the exact code version that was checked. If new commits are pushed during review, the new changes must be re-evaluated.

Returning to the experiment: after restoring the default value, both samples pass. Then a further change replaces the default with an empty string; the old record now fails. The previous pass record remains but is marked STALE (expired). Re-running on the new commit fails the second assertion (expected "default", got empty string).

Figure 3: Which commit does 'pass' belong to? (Synthetic experiment: old commit's pass record stays valid but doesn't prove new commit passes. Animation highlights three steps sequentially; pause duration for explanation only.)
Figure 3: Which commit does 'pass' belong to? (Synthetic experiment: old commit's pass record stays valid but doesn't prove new commit passes. Animation highlights three steps sequentially; pause duration for explanation only.)

When receiving an agent's completion report, having the commit, rule version, executed checks, and results together makes verification easy. If the agent says tests passed but later modifies files, we know what remains unverified.

Code changes don't require full re-review each round. Subsequent reviews reference prior context and failure logs. Previously found clues are reused; only the impact of new changes is additionally checked.

Can't Fix? Hand to Human

We want agents to iterate, but not spin indefinitely on the same problem. After three failed repair attempts on a check, the team escalates to a human. For the old-session case, if resume still fails after several rounds, the human receives the failing record, run command, and attempted fixes, and can decide whether the fix direction is wrong or the compatibility strategy needs revisiting.

Three rounds is their threshold; your project can set its own based on task cost and impact. For the human taking over, the most valuable information is what was tried and where it got stuck , avoiding restart from a vague "fix failed".

For local commits, the team records the verification method used; unverified commits are marked none. Commit gates check whether /code-review or /verify was used in the session. Initially they run in observe mode to see which commits would be blocked and check for false positives, then switch to enforce to actually block.

Reasons for missing verification can be noted in the commit message. After two blocks in the same session, the third is allowed through to prevent Claude from stalling. This is about local commit permission, separate from the three-round human escalation. Allowing commit saves work-in-progress and eases handoff; failed checks remain, and merge/release still enforce their own conditions. When adopting this, consider: after local pass, which later step re-verifies?

Missed Issues?

Even with correct rules and code versions, reviews can miss problems. How to know where misses happen? The team replays past incidents: restore the repo at the time of the incident, run the reviewer on the problematic change. They also test on similarly sized PRs that merged cleanly and stayed stable. This measures both recall of known defects and false-positive rate on good changes.

They track recall in three stages: initial defect detection, post-verification remainder, and final inclusion in review results.

Back to the old-session example: if the old format was initially overlooked, adding samples and clear trigger conditions helps. But if the missing field was detected and later discarded as irrelevant during result aggregation, we must understand why that judgment occurred. Adding a few sentences to the initial prompt may not fix it.

For adding new rules, the team repeats the process. They mine human review comments and PR reverts for recurring issues: appearing in at least two PRs, triggerable from diff, then have three agents attempt to find counterexamples, finally human approval. Existing tests, verification steps, and rules are periodically cleaned to keep the suite worthwhile.

Your project can start small: keep a recently missed bug, adjust checks, and see if it's caught; then test a normal change to check for false alarms. This validates whether the new rule helps.

Getting Started

After reading the Claude Code team's sharing, I'll pick one feature that repeatedly requires manual verification. For the old-session case, one old record, one resume procedure, one clear expected result turns "remember to check compatibility" into an actionable task.

Expectations can be user-centric: yesterday's messages visible, order preserved, can send new messages today. For the user, the upgrade is successful only if their existing data still works.

Then have the agent run before and after the fix, recording command, output, and associated commit. If it fails, continue fixing; at the agreed stop condition, hand off the sample and attempt log to a human. Known issues are deferred and documented, not buried in a single "done".

First, turn one recurring manual check into a repeatable automated check. Once that works, add other error-prone areas. Team members familiar with specific code can contribute their recurring review points, making that experience reusable on the next change.

This is the most practical takeaway: investing in verification setup gives the agent a chance to self-detect and self-correct on every change, and humans can continue from where the agent stopped instead of re-investigating from scratch.

References

Addy Osmani, How the Claude Code team thinks about verification (https://claude.dev/blog/how-the-claude-code-team-thinks-about-verification/)

Claude Code Docs, Customize Code Review (https://code.claude.com/docs/en/code-review-customize)

Boris Cherny, Claude Code usage tips #13: Give Claude a way to verify its work (https://x.com/bcherny/status/2007179861115511237)

Simon Willison, Red/green TDD (https://simonwillison.net/guides/agentic-engineering-patterns/red-green-tdd/)

Claude Code Docs, Best practices for Claude Code (https://code.claude.com/docs/en/best-practices)

GitHub issue #45550, Claude Code normalizes broken tests with @skip instead of fixing them, 2026-04-09; user report, not independently reproduced (https://github.com/anthropics/claude-code/issues/45550)

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

testing strategiesClaude Codehuman escalationAI code verificationbaseline review rulescommit bindingreliability checksverify skill
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.