How Traffic Replay Safeguards Production Systems: Strategies and Best Practices

1. Background

In 2023 the testing team achieved zero production issues thanks to meticulous work, risk control, strict test case review, automation, integration, regression verification, regular pressure testing, high‑fidelity promotion testing, and especially extensive traffic‑recording‑replay practices.

The article examines traffic replay from an R&D perspective, inviting feedback from testing experts.

2. Traffic Replay

Traffic replay records real online traffic, replays it in a pre‑release environment, and compares sub‑calls and responses to locate code problems. It offers low creation cost, zero intrusion to business code, realistic call chains, multi‑scenario coverage, traceable data, diff comparison, precise issue localization, and early problem detection. Risks include potential online impact from careless operations, downstream traffic spikes, or dirty data caused by write actions.

3. Traffic Recording

Sources are real online traffic, which can be live, historical (offline), or artificially generated. The key concern is whether the recorded traffic sufficiently covers the business scenarios affected by code changes.

1. Traffic Rules

Filtering: R2 supports visual configuration of field‑level filters or custom scripts for complex rules.

Deduplication: Duplicate traffic can be reduced while preserving interface coverage; custom scripts may be used to identify duplicates based on selected parameters.

Scenario Coverage: R2 provides a coverage metric to reveal missing or unknown scenarios. For example, a special evening‑only business requires recording traffic after 20:00 to ensure coverage.

2. Promotion & Holiday Special Scenarios

Some flows only appear during big promotions or Chinese New Year (e.g., pre‑sale scenarios). Capturing these requires careful filtering, mock handling of downstream services, or extracting logs, while ensuring performance impact is minimal.

4. Replay

Replay can be performed as offline DIFF or real‑time DIFF. Real‑time DIFF suits time‑sensitive scenarios such as billing or activities, though it may still fail for extremely latency‑critical cases. Injecting the recorded timestamp (e.g., via System.currentTimeMillis()) can help align replay time.

1. Read‑Only Interface Replay

Replaying read‑only interfaces without mocking provides a final regression check, but beware that an interface may later gain write behavior.

Case: A capacity‑control interface was read‑only until a write logic was added in 2023, causing unexpected capacity consumption during replay.

Solution: Close the new write logic via a DUCC switch and manually verify write paths before each replay.

2. Write Interface Replay

Write interfaces have side effects that can pollute production data. Strategies include building a UAT environment with shadow databases/Redis, using a force‑bot flag to route writes to shadow resources, and focusing verification on key data and logs rather than simple diff.

3. Read‑Write Interface Replay

Core logic is read‑dominant with some internal writes. Approaches include mocking return values or disabling write logic via DUCC switches.

4. Should Replay Be Mocked?

Mocking is not directly tied to read/write classification. Mock replay is a white‑box testing method; non‑mock replay is black‑box.

5. Replay Coverage Statistics

R2 supports offline replay code‑coverage collection, generating reports to verify whether traffic recording is comprehensive and to identify testing blind spots.

6. Diff Result Comparison

Define flexible comparison strategies—ignore irrelevant fields, focus on key outputs, and adapt to scenario requirements. Some diffs indicate configuration changes, others reveal bugs that must be fixed and retested. Occasionally, differing diffs are intentional due to logic updates; in such cases, traffic replay enriches test cases rather than merely comparing results.

I am a developer new to testing; the article may contain inaccuracies, and I welcome expert feedback for improvement.