Why SRE Agents Misdiagnose Despite Complete Evidence: Align Time Windows & Versions First

An SRE team hands a post-deployment latency alert to an Agent. The Agent retrieves metrics, anomalous logs, and change records, each with citations, yet the on-call engineer cannot reproduce the conclusion. The problem often lies between the citations: each record is real, but do they describe the same incident?

Three True Records, One Wrong Conclusion

Expanding the Agent's citations reveals three different "time" definitions. The table below shows the same day, same timezone.

Evidence | Original Time Meaning | Entity or Scope | What It Proves
E1 Connection pool timeout | Occurred 09:57, observed/collected at 10:02 | old-7, v41 | Old instance timed out before deployment
E2 Latency alert | Evaluated at 10:03, looking back 5 minutes | Order API aggregated traffic | Latency statistic abnormal in this window
E3 Deployment record | Started receiving traffic at 10:00 | New batch v42 | New version began serving

If the query tool only returns "10:02, connection pool timeout", the event time is lost. If the metric query only gives a P95, the aggregation hides instance and version. By the root-cause candidate step, the model sees only "connection pool timeout and latency rise after deployment." The error occurs before root-cause formation. Citation verification confirms E1, E2, E3 exist, but cannot supply the missing link: "slow requests ran on v42 and were slow acquiring connections."

Put Evidence Back Where It Occurred

OpenTelemetry's log data model distinguishes two timestamps: Timestamp (source-recorded event time) and ObservedTimestamp (collection system observation time). The latter is not the backend indexing time. If tools collapse them into one timestamp, the Agent loses the basis for detecting late-arriving logs.

E1 need not be deleted; it should be placed on the old instance timeline at 09:57. If source time is missing, explicitly record "event time unknown". If clock skew between machines makes ordering indeterminate, preserve the uncertainty — do not use collection time to fabricate a definite causal link.

Then align entities. OpenTelemetry's service.name, service.version, and service.instance.id express logical service, component version, and instance. Merging only by service name blends old and new instances during a rolling deployment.

On the diagnosis side, map instances to the image digest, config revision, and deployment batch at the time of failure, retaining the mapping's validity period. The key is "at the time of failure": stamping current inventory onto historical logs with v42 writes a version error into the evidence. Code version unchanged but config changed must also be separable for comparison.

Metrics add another time layer. Prometheus range selectors sample before the evaluation instant; the example's 10:03 five-minute window covers 09:58 onward to 10:03, straddling the 10:00 deployment. A spike at this point cannot be read as "slowdown started at 10:03". Instant selectors may return the most recent sample within the lookback range, not necessarily collected exactly at evaluation time.

Therefore, query records must fix the expression, evaluation time or start/end times, step, and aggregation granularity. Prometheus's query_range interface takes start, end, step as parameters; postmortems should retain these actual inputs, not just "last five minutes". If the version dimension was lost in aggregation, fall back to a data source that retains it, or admit comparison is currently impossible.

A Usable Reference Format

A reference usable for judgment can look like this. It is a diagnostic-side record sketch, not a product interface:

source=logs; evidence_id=E1
event_time=2026-10-10T09:57:00+08:00
observed_time=2026-10-10T10:02:00+08:00
scope=prod-a/order-api/old-7; version=v41
query_basis=event time; result_ref=saved raw result E1
usable_for=pre-deployment old instance anomaly; not_for=v42 failure window connection wait

The tooling adapter must also retain whether results were truncated, sampled, or timed out, and which instances the results cover. Otherwise "no connection wait found" could mean either truly none or that such records were not collected. Empty results only have discriminative meaning together with coverage scope.

Let Candidate Explanations Face Counter-Evidence

Aligning time and entity only makes evidence comparable; it does not prove cause. The next step is not to keep searching "v42 connection pool issues", but to write the current explanation as a falsifiable prediction.

In this example, at least the following explanations must compete, rather than treating "database slow" and "connection pool full" as synonymous candidates.

Candidate Explanation | Expected Phenomenon | Priority Counter-Evidence Query | Result Interpretation
v42 own connection management regression | Same-class requests on v42 wait more for connections | Compare v41/v42 connection wait and request latency in same window | Both versions slow and wait not increased weakens this specific explanation
Shared dependency slowed | Requests of both versions show increased latency in same dependency call segment | Query same-window call traces and dependency-side processing latency | Only client side slow still requires network or queue investigation
Request composition changed | Overall P95 up, certain slow routes proportion increased | Recalculate latency and traffic composition by route and request category | Group stable while overall worse, prioritize traffic change investigation

New-vs-old version comparison must be restricted to comparable traffic. Beyond same cluster and route, request size and load affect results; if new version only receives large requests, two aggregate curves cannot judge code regression. If deployment does affect shared resources, old version slowing simultaneously does not rule out deployment impact.

For instance, supplementary query E4: in prod-a, same route, same request category, request size and load comparable. Compare pre-deployment (09:55,10:00] vs post-deployment (10:00,10:03], connection acquisition wait P95: v41 4→4 ms, v42 after deployment 4 ms; request P95: v41 120→880 ms, v42 after deployment 900 ms. v42 has no pre-deployment baseline; this is version-grouped window statistics, not the initial five-minute aggregated alert.

Label "only v42 own connection management regression" as candidate H1: E4 shows old version also slowed, both versions' connection wait unchanged, H1 moves from to-be-proven to weakened, citing E4; next step query shared call path; P95 may still hide severe wait for a few requests, so connection issue not excluded. Even if dependency call segment latency increases later, must still distinguish network, queue, and retry, cannot directly conclude dependency service failure.

If counter-evidence query does not return complete results, a useful response can be: "Existing E1 belongs to pre-deployment v41 instance, does not support v42 connection pool root cause. Connection wait data for both versions in failure window incomplete; retain candidate as to-be-proven. Next step supplement same-route connection wait and call traces; if still missing, hand over to on-call SRE."

Model only proposes candidates; diagnostic side runs deterministic rules. Example rule for E4: SRE verifies request size and load, sets comparable=true; adapter returns coverage=complete and both versions' wait P95. Program sees v41 before/after both 4ms, v42 4ms, outputs H1=weakened, cites E4; any field missing → to-be-proven. Equal values used for demonstration; actual tolerance predefined by team, not improvised by model.

"Support" does not equal root cause confirmed: only when failure requests are confirmed to spend time on connection acquisition, the corresponding connection occupancy cause is found, and competing explanations like shared dependency are examined, is it handed to SRE for root cause confirmation.

Scheduler should save responses and step logs, decide in-budget supplementary queries; account by query ID, candidate status and completion markers submitted together. E.g., E4 saved but H1 not updated → rerun that step to verify, no re-query, no extra budget; already completed → skip. Response not saved → result unknown, supplementary query separate budget; insufficient permission, budget exhausted, or critical observability missing → hand to on-call SRE.

Diagnostic tools remain read-only; rollback or production config changes handled by incident commander per existing authorization and runbooks. Mitigation before root cause confirmed and after root cause proven are two different decisions.

Use Offline Replay for Acceptance, Not Another Demo

Before onboarding real alerts, teams can locate improvement points by symptom. Adding tools is not the only answer.

Symptom | Primary Cause to Check | Discrimination Method & Next Step
Citations all real but sequence invalid | Event time and observed time mixed | Expand both times, rebuild timeline, redo association
Same query yields different result on replay | Relative window drift, data backfill, or query config change | Cross-check actual parameters and original result snapshot
New vs old version comparison contradictory | Aggregation lost dimension or traffic not equivalent | Supplement grouped data; if cannot complete, retain unknown
Always supports initial guess | Queries only seek supporting evidence | Add competing hypotheses and explicit counter-evidence queries
Empty query result declares exclusion | Insufficient coverage or tool failure swallowed | Check sampling, truncation, error status, then decide supplement or handover

Replay package must save alert, entity mappings at that time, query inputs, tool returns, evidence citations, candidate state changes, and record model, prompt, and tool versions. Platform team maintains this evidence contract; storing only the final answer cannot reproduce the materials it used.

Fixed-response replay: SRE annotates test cases with allowed evidence relations, forbidden conclusions, and stop conditions; platform test program validates structured output, preventing model from self-scoring. E.g., E4 allowed to weaken H1 but not to confirm dependency failure; accepting a different to-be-proven candidate okay, treating E1 as v42 failure window evidence fails. Semantic disputes go to SRE review, then decide whether to update test case. Re-querying backend is a separate check, must also examine late-arriving data, retention, and config changes.

Replay cutoff follows each query. E.g., alert triggers 10:03, query executes 10:04; records queryable by 10:03:30 may enter that response, records queryable only at 10:05 cannot be backfilled. Collection availability ≠ backend queryability; fixed replay prefers the saved tool response from that run.

Example: test R1 removes failure-window version mapping alone, supplementary query fails and budget exhausted. Expected: H1 stays to-be-proven, returns missing fields and failed queries, hands to on-call SRE; if output "v42 connection pool exhausted" or "v42 excluded", both judged fail. Acceptance targets evidence selection, support relations, and stop conditions, not verbatim text match; model variance requires multi-run replay recording differences, cannot substitute one correct answer for validation.

This approach adds maintenance cost for result snapshots, historical mappings, and coverage markers. Architects must deliver usable evidence, counter-evidence results, and stop reasons together so on-call personnel can continue investigation. For systems with traceable time and entity, push these constraints forward into tool returns and diagnostic control; for incidents with large observability gaps and multiple interacting causes, let the system produce auditable investigation materials, leaving final attribution to humans.