Skip to content
Code Understood ≠ Behavior Verified: Agent-Driven Runtime Validation for Legacy Systems

Code Understood ≠ Behavior Verified: Agent-Driven Runtime Validation for Legacy Systems

After mapping code relationships in a legacy import system, the author demonstrates how to verify actual runtime behavior through isolated experiments using a three-row sample, testing partial success, duplicate submissions, lost responses, timeouts, and process interruption after write—revealing a duplicate-write bug that only appears when progress isn't persisted before a crash.

Data Bricklaying Diary
Data Bricklaying Diary
Data Bricklaying Diary

Separate "What Is" from "What Should Be"

The previous article mapped call relationships: submission interface, message consumer, customer data write, and task status query. But the diagram doesn't answer: will messages arrive? What happens if the process crashes after write? Will a user resubmit create duplicate records? Code shows retries, exception handling, and success/failure states in the task table, yet gaps between logic segments often hide real problems.

The author uses a hypothetical import system for illustration. First, three business rules are confirmed: valid rows succeed, invalid rows fail; same user, data scope, request ID, and file on resubmit returns the original task; same task re-execution must not create duplicate customer records. If the original request ID is reused with a different file, reject as conflict—don't silently swap the original task's input. Deduplication is split into two layers: prevent duplicate task creation and prevent duplicate customer writes. A corrected file re-upload is a separate question—whether to create a new task and how to handle previously succeeded customers must be confirmed separately, not treated as a retry.

Confirm Which System You're Testing Before Running

Missing valid configurations from the previous step must be supplied: know which version the build corresponds to, whether the consumer is enabled, and which databases, queues, and file stores the interface and consumer connect to. "Started the app locally" isn't enough—the interface and consumer in the same app may read different data sources or message configs; the interface may be reachable while the consumer listener isn't even enabled. Such results can't validate business logic.

Test data, queue destinations, and file directories are isolated. Notifications and callbacks are redirected to inspectable test receivers, and replaced dependencies are documented. Full production replication isn't required: checking file fields needs only a lightweight env; verifying message redelivery, DB commits, and consumer acks requires the real components. Replacing everything with success-returning mocks bypasses exactly the issues under investigation. The agent assists with startup commands, samples, and collection scripts; humans verify target addresses and data scope first, and tool permissions are scoped to the test environment. Missing run conditions mean retaining static conclusions and pending verification items—no gambling on real customer environments.

Use a Tiny File to Trace Results End-to-End

The first sample needs only three rows: customer A001 and A002 valid, A003 missing a required field. The file gets a fixed name and a saved checksum to detect content changes. Each run receives an experiment ID (e.g., E01) to correlate materials, kept separate from the system's request ID and task number; retries must not swap the business request ID just to change the experiment ID.

Start with the happy path. Confirm initial state of the three customer records, submit the file, save the task number from the response, then trace through consumer, row-level results, and final query. Don't just screenshot "import succeeded"; cross-check row-level results, actual customer records, and task query together. Use the system's own status names, not a new "success" definition.

Scenarios are organized in a table with expected basis and observations to record:

One illegal row in three: partial success allowed → observe row results, customer records, final task status.

Original request resubmitted: same request ID and file returns original task → observe both task IDs and whether new processing occurred.

Task accepted but response lost: recover original task per convention → observe acceptance fact, retry response, task and data counts.

Dependency timeout: verify against existing timeout and compensation agreements → observe timeout location, writes already made, subsequent state.

Process interrupted after write, then recovered: same task must not create duplicate customers → observe data before/after submit, recovery path, re-write result.

Rows without explicit rules are still recorded with conclusion "expected pending confirmation"—don't fill in what the agent thinks is reasonable. Async tasks need an agreed observation window; beyond it, record "not finished within observation window" with current state and queue status. Absence of locatable request/consumer records or data changes means insufficient evidence, not proof of non-execution.

"Accepted but response lost" must control timing: confirm task created first, then drop response at test client or proxy, then retry with original request ID and file. Network failure before request delivery tests a different scenario. For dependency timeouts, verify whether the downstream already completed the write—don't assume local error means no impact.

The Most Critical Gap: Written but Not Yet Booked

Using the same three-row sample, the author targets a moment normal tests miss: A002's customer record committed, but processing progress not yet persisted and message not yet acknowledged, then process crashes.

Before the experiment, the agent marks three points: write commit, progress persistence, message ack. If they actually commit together, the hypothesized gap doesn't exist—a real interrupt point must be found. Otherwise you're validating a fabricated problem.

In an isolated test build, a debug pause or temporary test instrumentation controls the interrupt point. Instrumentation (code added temporarily to observe or pause) must itself be recorded and inspected. Throwing an exception before the call can't simulate "data committed, subsequent steps not done."

Sequence: restore initial data for this scenario, start processing; pause at target point, verify customer write committed; kill process, then resume consumer or redeliver task per system's existing mechanism. Record recovery entry and message ack state—don't assume restart always triggers redelivery.

Suppose the experiment yields this result (E05):

Scenario: E05, A002 interrupt after submit, before progress persisted
Linked: request R01, task T01, sample sample-01
Code derivation: resume per persisted progress; duplicate-check effectiveness on re-write TBD
Before interrupt: A001, A002 each have one committed record; A003 not written
Recovery: original message unacknowledged, redelivery observed after recovery
After recovery: A001 one record, A002 two records, A003 not written
Task query: two success, one fail
Rule basis: same task re-execution must not create duplicate customer
Current judgment: final data violates rule, task stats don't reflect duplicate write
To investigate: why was A002 written again on recovery, which check failed

Real runs attach code version, valid config, sample checksum, events, and query results so colleagues can reproduce.

The agent then traces the recovery path: did missing progress persistence cause the consumer to re-run A002? Was there a duplicate check before write? What key did the check use—request, task, or customer ID? Is the DB constraint actually enforced in the running instance?

These explanations are preserved first. An extra customer record warrants further investigation but doesn't yet dictate which code to fix, let alone swapping the message queue.

Trace Discrepancy Sources Before Fixing Code

Finding duplicate writes tempts an immediate agent fix. The author resists, narrowing the cause range first—otherwise a patch may hide the current sample while burying the original clues.

If the consumer didn't receive the message, first check the config gaps from the previous article: do actual send and subscribe destinations match? Is the consumer started? Config mismatch or test env not started is a different problem from business code defect.

If task shows complete but data doesn't match, verify statistics source, transaction commit, and query location. Results from cache or read replicas may have visibility latency. Record query time and final verification result—don't conflate transient latency with permanent loss.

Another case: actual behavior matches old docs but business stakeholders no longer accept it. This requires a rule change with compatibility impact analysis—not solved by changing test expectations to match current output.

Next questions to the agent become specific:

Investigate duplicate write per E05 materials, causes only, no code changes. List evidence for and against each explanation, missing observations, and the next minimal verification. Keep multiple explanations when indistinguishable.

Save this round's evidence, then clean test data. Change one relevant condition at a time, rerun scenario. If multiple experiments share state, either restore known initial conditions or explicitly mark them as a continuous sequence; stale queue messages and scheduled compensations must be accounted for. Otherwise today's reproduction vanishes tomorrow simply because experiment conditions drifted.

Deliver Materials the Next Person Can Actually Use

After this round, the relationship map and current-state diagram are updated—but only for portions actually verified. If the test env confirms message delivery, note the env and scenario; don't erase the original "on-site config unknown" marker.

Results are categorized with supporting materials:

Verified behaviors (with evidence)

Defects pending investigation

Business decisions pending

If "how to recover task after response loss" has no agreed convention, it goes to business pending items—document who decides and what rule is missing, don't auto-invent an interface to complete the report.

This round's output feeds later architecture discussions: where state-handoff risks exist, which boundaries need adjustment, which are just outdated docs. Uncovered concurrency, load, and real deployment differences remain documented—cannot declare the system reliable based on a few samples.

The author values most that the next person can state: this conclusion holds under which version and conditions; if conditions change, what must be rechecked. Only then does the current-state diagram become decision-support material.

Next: examine architectural flaws—agent flagged a cross-module access; on what basis is it deemed a boundary violation rather than a legitimate business dependency?

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

message queuesroot cause analysisruntime validationlegacy systemsagent-assisted verificationduplicate write detectionedge case testingisolated testing
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.