5 AI Patterns to Pinpoint Flaky Production Bugs

The article opens with a scenario of an intermittent P99 latency spike in an order service at 2 AM, where logs show no ERROR entries and local reproduction fails. The author argues that the core challenge of flaky bugs is the absence of a reproducible crime scene — no stack trace, no clear reproduction path, and even the affected machine is uncertain.

Boundary: What AI Can and Cannot Do

AI can only do retrieval and enumeration — shrinking 50 k log lines to 20 candidates, cutting 8 hypotheses to 2. These tasks are large-scale, rule-based, and carry no accountability, which matches model strengths. AI cannot make responsibility judgments: deciding which line to change and assessing ripple effects remains a human duty. The author cites Stripe’s 2018 Developer Coefficient survey: developers spend 17.3 hours/week (42 % of a 41.1‑hour week) on maintenance, but AI can only replace the “finding” portion.

Pattern 1: Pin the Timestamp First, Then Log Retrospection

First, define a time window centered on the alert: 5 minutes before, 2 minutes after. Expand before because causes often precede symptoms (e.g., connection‑pool queueing starts 30 s earlier); expand after to see whether recovery was spontaneous or triggered by restart/scale‑out. Then narrow by anomaly density — this is called a “time box.”

Second, do not search only for ERROR. Flaky failures often appear as HTTP 200 + business error code, WARN‑level slow queries, connection‑pool waits, or thread‑pool queue buildup. Searching only ERROR yields “no logs” — a false negative caused by the query filter.

The article provides ready‑to‑use queries for three common log stores:

Loki (logcli): pulls non‑INFO logs in the time box with a 5000‑line limit.

Elasticsearch / Kibana (DSL): aggregates by upstream dependency for requests exceeding 1000 ms.

ClickHouse / Doris (SQL): groups by upstream, computing count, p99 latency, and business‑error count, ordered by error count and p99.

Feed the aggregated results plus ~20 raw samples to AI with a strict prompt that forces it to:

Sort anomaly samples chronologically and identify the 1–2 densest sub‑windows (to the second).

List recurring field combinations (upstream, error code, thread name, machine IP, etc.).

Explicitly state which fields are absent in the data and therefore cannot be used for judgment.

Every conclusion must cite a sample line or aggregation row; anything not in the data must be answered “data insufficient” — no speculation.

Pattern 2: Make AI Write Comparative Queries, Not Search Queries

Search queries require knowing what to look for; flaky bugs are defined by not knowing. Instead, pull failing requests and same‑time‑window successful requests for the same interface, then do a field‑level diff. Sample the success set at 10–50× the failure count, same interface, same window, random sampling to avoid selection bias.

The prompt borrows a rule from obra/superpowers’ systematic-debugging: “List every difference, however small. Don’t assume ‘that can’t matter’.” The model’s value is that it won’t skip tedious differences that humans dismiss (e.g., machine IP, user ID). The prompt requires:

Per‑field comparison outputting three categories: values only in failures, fields with distribution shifts, fields identical (excludable).

Label each difference as potential cause or effect (e.g., high latency is usually an effect).

List all differences, no matter how small.

Retain only the two most verifiable hypotheses; move the rest to “not verifying now” with reasons.

Pattern 3: Minimum Reproduction Is Crafting Concurrency, Not Writing Steps

Flaky bugs often need specific thread interleaving; single‑threaded runs miss them. Adding sleep to make the bug disappear is measurement, not a fix — it reveals the race window size. The correct direction: amplify the failure rate from 1/10 000 to 1/10 via concurrent load testing with scheduling jitter and a fixed random seed for replay.

The article provides a Go skeleton ( repro.go) that accepts -n (total requests), -c (concurrency), -seed (fixed seed). It uses a semaphore for concurrency control, random 0–8 ms jitter between calls, and a ring‑buffer logger (flight‑recorder mode) that keeps recent context in memory and only flushes on error, avoiding I/O perturbation of the critical path. A prompt to generate such a script is also included.

Pattern 4: Single‑Hypothesis Validation with Forced Two‑Column Evidence

Confirmation bias leads to cherry‑picking supporting evidence. The countermeasure: force AI to output Supporting Evidence and Opposing Evidence in two columns. The model is creative at finding support; requiring opposition surfaces contradictions.

Two disciplines accompany this:

Change one variable at a time. If a change doesn’t work, revert and try a new hypothesis; never stack changes.

Hypothesis cards. Each card contains: Hypothesis, Supporting Evidence (with source), Opposing Evidence (must write “No opposing evidence found” if none), Verification Method (executable in staging within 30 min), Current Conclusion (only “To Verify / Falsified / Confirmed” — “maybe” is forbidden).

The author introduces the Three‑Failure Rule : after three failed fix attempts, stop and question the architecture instead of guessing parameters. The first failure means the hypothesis was wrong; the second means the hypothesis space was wrong; the third means the problem lives at a higher abstraction layer (e.g., a synchronous call that should be async). This rule is codified in superpowers’ systematic-debugging and backed by the maxim: “95 % of ‘no root cause’ cases are incomplete investigation.”

Pattern 5: Connect Company Logs and DB Monitoring to AI via MCP

The first four patterns assume low‑cost data feeding. Manual copy‑paste doesn’t scale. Integration is layered:

Off‑the‑shelf MCP. Grafana MCP (3,413 stars as of 2026‑09‑05) covers 19 data sources, 40+ tools, with tool categories to limit context window. Sentry MCP ( mcp.sentry.dev) runs as a remote streamable‑HTTP + OAuth service; every event carries release attribution, enabling “which release first introduced this issue?” queries.

Read‑only DB MCP. Many flaky timeouts trace to DB slow queries or lock waits. Grafana often shows only aggregated curves. Give AI a read‑only account to query pg_stat_statements (PostgreSQL) or performance_schema (MySQL), ordering by max latency not average — because a 1‑in‑10 000 5‑second spike is invisible in the mean.

Custom log MCP. For self‑hosted ClickHouse/Doris/ELK, wrap with FastMCP. A public case (not the author’s) exposes 11 tools ( find_errors, trace_request, search_keyword, log_statistics, …) with auto table discovery via service‑name mapping and 24‑hour cache.

MCP vs. Skills for team sharing: Skills store credentials locally per user, no connection pooling, no audit, permission granularity depends on personal accounts. MCP holds credentials server‑side, uses connection pools, provides audit logs, and can enforce read‑only accounts. Conclusion: Skills are for individuals; MCP is for teams.

Six‑item pre‑flight checklist before going live:

Read‑only account ( SELECT only, no DDL/DML).

Dedicated service account (not a personal account).

Credential rotation with defined cycle and revocation process.

Result row limit (e.g., max 200 rows per query).

Query timeout (hard kill to prevent SELECT * dragging down the log store).

Sensitive field sanitization (phone, ID, token, cookie) before data enters the model — most often skipped, highest risk.

Two real pitfalls: Grafana MCP on Grafana <9.0 silently returns empty results for key tools (not an error, just empty), leading to false “no logs” conclusions. Security: log content becomes model input; user‑controlled strings (User‑Agent, request params, error messages) enable indirect prompt injection (GrafanaGhost, April 2024). Principle: let it read, never let it act.

When AI Debugging Definitely Fails

True races / memory visibility. AI cannot see CPU instruction reordering, cache coherence, or scheduler interleaving. It can help read code and write reproduction scripts, but “why a memory barrier is needed” must be verified against specs.

Cross‑host clock drift. Log‑timestamp correlation assumes trustworthy clocks. NTP misconfiguration causing tens of milliseconds drift can reverse causality. Check ntpq -p and per‑host offsets first.

Third‑party black‑box jitter. If the downstream is a cloud vendor or external provider, you lack internal metrics. AI can only confirm “the other side is slow”; options reduce to retry, degrade, or switch providers.

Legacy systems without trace_id. Requests spanning 5 services with no unified trace_id make Patterns 1 and 2 impossible — you can’t even stitch “the same request.” Fix: implement trace_id first, don’t ask AI to guess request ownership from timestamps and user IDs.

Data point: Sherlocks’ analysis of 73 AI Agent incidents (Jan–May 2026) shows no‑observability MTTR = 4.2 hours vs. schema‑validated tooling MTTR = 54 minutes. Note: this is AI Agent system statistics, not traditional backend bugs, but the direction holds — evidence absence causes super‑linear MTTR growth.

FAQ

Small team, no log platform? Use grep + awk to slice logs by time window, feed the file to AI. Never skip the time window or ERROR‑only search. Long term, prioritize trace_id over a full platform.

Will MCP leak data? Server‑side credentials (MCP) are far safer than local Skills — unified account, audit logs, read‑only enforcement. Real risks: missing sanitization and granting write permissions.

Why does AI still guess after I paste logs? Three likely reasons: (1) missing “answer ‘data insufficient’” constraint; (2) pasted only a single error, no same‑window success samples for diff; (3) asked “why did it fail?” which invites speculation. Rephrase to “what fields differ between these two request groups?”

Give up on a long‑standing flaky bug? “Can’t find root cause” is a property of the investigation, not the bug. Check the four steps: time window aligned? Only searched ERROR? All differences listed? Hypotheses falsified? If all done and still no result, then consider third‑party black box or missing trace_id.

Closing

One‑sentence takeaway: Let AI do retrieval and enumeration, keep “which line to change” for yourself, and after three failed fixes go question the architecture.

Minimum viable next step: pick a recent unsolved flaky bug, run only Pattern 1 and Pattern 2 — define the time window, pull success/failure samples, do a field diff. No infrastructure changes required, runnable today, and often narrows the scope from “entire system” to “one module.”