Debugging AI-Generated Code: The 'Facts First' Method That Catches Agent Hallucinations

Introduction

Cindy Sridharan, a noted distributed-systems engineer, recently published a 63-minute Medium article recounting how she used Claude to debug three production bugs in code she did not fully understand. This summary distills her core methodology: treat debugging as a constraint-satisfaction problem, feed the agent hard facts first, and let hypotheses survive only if they explain every observed fact.

Three Bugs, Three Failure Modes

Bug 1: Latency Spike — One-Line Fix

A 9,000-line feature written by an agent in ~1.5 days caused P99 latency to spike on both client and server within 10 minutes of canary rollout. Claude’s first analysis: TLS certificates reloaded on every RPC. The author couldn’t refute it because she only shallowly understood the 9,000 lines. She then asked Claude to log into a production box, capture packets, monitor the process, and compare against a control where the feature flag was off. The certificate loaded once — Claude was wrong. The real signal: DNS resolution returned ~5 addresses in the control but ~400 in the failing path. The new mTLS code path used the wrong client ( DatabaseClient instead of ShardedDatabaseClient), so requests weren’t routed to the correct shards. Fix: one line change. Why tests missed it: staging had 2 shards; production had 5 shards × 80 replicas.

Bug 2: Memory Leak — Agent Wrong Three Times

Containers were OOM-killed. User provided repro steps and heaptrack output. Claude first blamed infinite retries, then a disabled optimization — both rejected because the author knew the state machine she wrote. On the fourth try, she had Claude trace the “no valid credentials” path line by line. Root cause: a default-enabled process-level cache caused a supposedly single-use state machine to be re-entered recursively, stacking frames until memory exhausted.

Bug 3: Mystery Hang — Needle in Kernel Data

A desktop daemon intermittently hit 60-second timeouts with no logs and no local repro. The author knew little about macOS internals. She had Claude dig through a 30 MB sysdiagnose dump: packet capture showed successful handshake then only heartbeats; lsmp revealed the auth daemon accepted the XPC request but never replied — the one-time reply ticket still existed after 67 hours.

Core Philosophy: Code Is an Unreliable Narrator

The author’s counter-intuitive conclusion: letting the agent read code first is an anti-pattern . Code shows what might happen, not what did happen. The state space is huge; the agent cherry-picks snippets and easily constructs a plausible but wrong narrative. Reading code should be the last step of the investigation.

Three Key Terms

Variable — the failing case.

Control — a case differing only in the suspected factor (e.g., feature flag off).

Signal — any data point that could explain the bug (logs, latency, call stacks, counters).

Two-Layer Framework

Fact Layer — logs, packet captures, profiles, configs, kernel data. Collected by tools; agent observes but cannot mutate.

Derivation Layer — hypotheses, inferences, narratives. Where the agent excels at storytelling and also fails most.

Rule: every fact in the Fact Layer is a constraint . Only hypotheses satisfying all constraints survive.

Three-Step Debugging Loop

Step 1: Use Controls to Find Signals

Only data present in the Variable but absent in the Control is high-fidelity signal; common data is noise. If the “healthy” path differs too much, pick a “louder failure” as control.

Step 2: Find Strong Signals via Hypothesis × Fact Matrix

A signal explained by many hypotheses is weak; a signal explained by only one or two is strong. Have the agent build a matrix:

Hypothesis                | P99 high | DNS ~400 addrs
--------------------------|----------|---------------
Server self-degradation   | ✓        | ✗
Cert frequent reload      | ✓        | ✗
<strong>Client routing regression</strong> | <strong>✓</strong>        | <strong>✓</strong>
Server capacity shortage  | ✓        | ✗
Event-loop stall          | ✓        | ✗
CPU throttling            | ✓        | ✗
Upstream slowdown         | ✓        | ✗

P99 is a weak signal (every hypothesis checks it). DNS 5 vs 400 is strong — only the routing hypothesis explains it. This explains why AI initially latched onto “cert reload”. Rule of thumb: SLO/SLI metrics are often symptoms, i.e., weak signals.

Step 3: Verify Against All Facts

The candidate hypothesis must pass every fact in the Fact Layer, including noise. One contradiction → discard.

When No Hypothesis Fits: Five Tactics

Re-check fact collection — wrong machine, time window, tool version?

Split coarse hypotheses — “this release caused it” → specific changes.

Merge fine hypotheses — per-host causes may share a single AZ issue.

Combine complementary hypotheses — each explains a subset, both required (memory leak = process cache + single-use state machine).

Only then invent a new hypothesis — strictly grounded in existing facts.

Progressive Signal Discovery

If signals are weak, dive one layer deeper to collect lower-level data. Stop as soon as a strong signal appears. Goal: strongest signal, not deepest dive. Humans often quit after 1–2 layers; agents don’t tire — this is their advantage.

Two Major Pitfalls

Pitfall 1: Agent Reproduction Must Be Discounted

In the hang case, a colleague reproduced a clock-skew token-rejection retry loop that looked identical to the user report. But packet-capture tails differed, and the kernel IPC table lacked the un-replied ticket. Reproduction lives in the Derivation Layer; it’s only credible when it emits the same strong signals as the real incident.

Pitfall 2: Adversarial Review Is an Eliminator, Not a Verifier

Adversarial review cannot guarantee a complete Fact Layer, and using the same model/context often double-counts a single perspective. Better: ask the reviewer to find counter-examples, perform elimination, and cite evidence for every claim. The author found ~1/3 of challenged claims had no citation and were dropped.

Reshaping the Software Factory for Agent-Native Debugging

The author’s metric: human intervention count is the honest measure of agent-native maturity. Key mindset shifts:

Past: Understand code → Future: Design verification loops (pick controls, define constraints, veto unverified hypotheses)

Past: Read code → Future: Collect facts (ensure agent gets data from the right layer at the right time)

Past: Ad-hoc collection after incidents → Future: Default collection (strong signals at the shallowest layer, e.g., log DNS resolution counts)

Past: Human as meat proxy (copy-paste) → Future: Human as policy maker — only handle ambiguous, accountable decisions

Past: “You don’t know how to use it” (skill issue) → Future: Infrastructure problem — make rigorous debugging the default, not prompt-dependent

Humans want less data (dashboards, aggregates); agents need raw data (aggregation destroys signal value). Therefore: optimize the Fact Layer for agents, optimize decision points for humans .

Two enduring principles:

“Full stack in one box” remains an anti-pattern. Local environments cannot replicate production Fact Layers. Give agents read-only, time-limited, auditable access to production signals.

Debugging framework can double as a release gate. If staging checked “client DNS resolution count matches last release”, the latency bug would have been caught at small scale.

Conclusion

Models grow stronger, but engineers don’t automatically keep pace. Agents amplify the engineer’s weaknesses: people who can’t debug now ship “looks-like-a-fix” non-fixes faster. The answer: replace line-by-line code understanding with higher-level system understanding plus a rigorous verification process.

Reference: Cindy Sridharan, How to effectively debug when you no longer understand the code — https://copyconstruct.medium.com/how-to-effectively-debug-when-you-no-longer-understand-the-code-bd3dcb2c5f7b