Skip to content
Building a Traffic Recording & Replay Platform with Spring Boot: Shadow Traffic, Structured Diff & Regression Validation for System Rewrites

Building a Traffic Recording & Replay Platform with Spring Boot: Shadow Traffic, Structured Diff & Regression Validation for System Rewrites

This article details a production-grade traffic recording and replay platform built with Spring Boot, covering five-module architecture, deterministic desensitization, shadow database isolation, structured JSON diff with ignore rules, and CI/CD gate integration to validate system rewrites against real traffic.

Xiaolin Talks Programming
Xiaolin Talks Programming
Xiaolin Talks Programming

Background: Regression Tests Can't Keep Up with Code Changes

During an order-system rewrite involving 100k lines of code, the team had only 300 regression cases. Unit tests were mostly hello-world level; integration test data was hand-crafted and missed real-world edge cases (empty fields, oversized text, concurrent orders, historical dirty data, downstream timeouts). Manual test cases based on requirement docs covered happy paths but failed on long-tail production scenarios. The core mismatch: reconstruction demands "external behavior unchanged" but acceptance often only verified "new features work". Subtle differences — a missing field, precision change from 2 to 4 decimals, list reordering — went undetected until user complaints.

Traffic recording/replay solves this: record real requests+responses on the old system as baseline; replay same requests against the new system in a shadow environment; run structured diff on responses; feed differences into CI as a gate. Production traffic may be noisy but is real, comprehensive, repeatable, and far more reliable than hand-written cases.

Overall Architecture: Five-Stage Pipeline

The platform comprises five decoupled modules communicating via messaging and storage, each independently deployable and scalable:

┌──────────────┐   ┌──────────────┐   ┌──────────────┐
│ ① Recorder   │──▶│ ② Storage    │──▶│ ③ Replay     │
│ (SDK/Agent)  │   │ DB+OSS+ES    │   │ Scheduler/   │
└──────────────┘   └──────────────┘   │ RateLimit/Mock│
      ▲                                 └──────┬───────┘
      │ Old System (Prod)                       ▼
      │                              ┌──────────────────┐
      │                              │ New System       │
      │                              │ (Shadow Env)     │
      │                              └────────┬─────────┘
      │                                       │ New Response
      │                              ┌────────▼─────────┐
      │                              │ ④ Diff Analysis  │
      │                              └────────┬─────────┘
      │                                       ▼
      │                              ┌──────────────────┐
      └──────────────────────────────│ ⑤ Report & CI    │
                                     │ Gate             │
                                     └──────────────────┘

Recorder captures HTTP/RPC/MQ request/response and full call chains using Filters, Interceptors, bytecode enhancement, and async buffering. Storage uses MySQL for index metadata, MinIO/OSS for full payloads, ES for search, and manages TTL archiving. Replay engine handles task scheduling, concurrency control, time/random seeding, dependency mocking, and shadow DB routing. Diff engine performs structured comparison, ignore rules, and severity classification. Report module provides visualization, diff drill-down, CI gating, and failed-case persistence.

Typical acceptance flow:

Record (Prod, 7 days, sampled) → Cleanse & Desensitize → Generate Traffic Suite
→ Deploy New Version to Shadow → Restore Baseline Data Snapshot
→ Replay at 1x → Per-item Diff → Produce Report → Gate Decision

Traffic Recording: Capture Request, Response, and Full Call Context

3.1 Entry Collection: Writing a Filter That Doesn't Break

Spring Boot's natural interception point is Filter because it accesses the raw body before consumption. HandlerInterceptor runs after the request stream is consumed — too late. The core challenge: body can be read only once .

@Component
@Order(Ordered.HIGHEST_PRECEDENCE + 10)
public class TrafficRecordFilter extends OncePerRequestFilter {
    private final TrafficRecorder recorder;
    private final TrafficProperties props;

    @Override
    protected void doFilterInternal(HttpServletRequest request, HttpServletResponse response,
                                    FilterChain chain) throws ServletException, IOException {
        // Entry sampling decision. DROP passes through untouched, no wrapping, no body touch.
        TrafficContext ctx = TrafficContext.start(request, props);
        if (ctx.getDecision() == SampleDecision.DROP) {
            chain.doFilter(request, response);
            return;
        }

        // NORMAL and ERROR_BOTH wrap: ERROR_ONLY persists only on error
        CachedBodyRequestWrapper  req  = new CachedBodyRequestWrapper(request, props.getMaxBodyBytes());
        CachedBodyResponseWrapper resp = new CachedBodyResponseWrapper(response, props.getMaxBodyBytes());
        long start = System.currentTimeMillis();
        try {
            chain.doFilter(req, resp);
        } finally {
            resp.copyBodyToResponse();          // CRITICAL: missing this leaves caller with empty body
            ctx.finish(System.currentTimeMillis() - start,
                       resp.getStatus(), resp.getCachedBody());
            if (ctx.shouldPersist()) {          // Normal sample hit, or ERROR_ONLY with abnormal status/biz code
                recorder.submit(ctx.snapshot());
            }
        }
    }
}

Critical details: copyBodyToResponse() cannot be omitted — the #1 failure point of custom wrappers.

Skip file upload/download by Content-Type and URI whitelist; otherwise memory explodes.

Cap request/response bodies (e.g., 512KB); oversized payloads store only digest + hash.

Error recording must be enforced in finally by checking HTTP status and business code in response body; entry-point slogans don't capture error bodies.

3.2 Async Write: Don't Slow Down Business

Recording is a sidecar; it must never impact the main path. Implementation uses a bounded in-memory queue + dedicated thread for batch flush:

public class TrafficRecorder {
    private final BlockingQueue<TrafficRecord> queue = new ArrayBlockingQueue<>(10000);
    private final ScheduledExecutorService flushExecutor =
            Executors.newSingleThreadScheduledExecutor(r -> new Thread(r, "traffic-flush"));

    public void submit(TrafficRecord record) {
        if (!queue.offer(record)) {
            // Queue full → drop sample and metric; never block business
            Metrics.counter("traffic.record.dropped").increment();
        }
    }

    @PostConstruct
    public void start() {
        flushExecutor.scheduleWithFixedDelay(() -> {
            List<TrafficRecord> batch = new ArrayList<>(200);
            queue.drainTo(batch, 200);
            if (batch.isEmpty()) return;
            try {
                recordRepository.batchInsert(batch);  // or push to Kafka
            } catch (Exception e) {
                // scheduleWithFixedDelay tasks die on exception; must catch
                log.error("flush traffic failed, size={}", batch.size(), e);
                Metrics.counter("traffic.flush.error").increment();
            }
        }, 1, 1, TimeUnit.SECONDS);
    }

    @PreDestroy
    public void shutdown() { flushExecutor.shutdown(); }
}

For high-volume production (5k+ QPS), recorder should only assemble and push to Kafka; persistence moves downstream, keeping recorder near-zero dependency.

3.3 Upstream/Downstream Call Chain Correlation

Recording only HTTP entry is insufficient; many bugs surface downstream. Each entry request gets a trafficId propagated to downstream services:

public class TracePropagationInterceptor implements ClientHttpRequestInterceptor {
    @Override
    public ClientHttpResponse intercept(HttpRequest req, byte[] body,
                                        ClientHttpRequestExecution ex) throws IOException {
        TrafficContext ctx = TrafficContext.current();
        if (ctx != null) {
            req.getHeaders().add("X-Traffic-Id", ctx.getTrafficId());
            req.getHeaders().add("X-Trace-Id",   ctx.getTraceId());
        }
        return ex.execute(req, body);
    }
}
ClientHttpRequestInterceptor

is for RestTemplate; Feign uses RequestInterceptor, Dubbo uses Filter, MQ producers inject headers. Downstream services with the same SDK read X-Traffic-Id and merge into a single call tree (entry + downstream calls). During replay you can replay only entry with full downstream mock, or replay downstream as well, depending on validation goal.

3.4 Async and Multi-thread Scenarios

ThreadLocal

loses context when crossing thread pools — the primary cause of missing recording data. Three-pronged approach:

Spring thread pool decoration via TaskDecorator to capture/restore context.

Manual new Thread and third-party pools use TransmittableThreadLocal.

MQ consumers restore trafficId from message headers.

@Bean
public TaskDecorator trafficContextDecorator() {
    return runnable -> {
        TrafficContext captured = TrafficContext.current();
        return () -> {
            TrafficContext previous = TrafficContext.current();
            TrafficContext.set(captured);
            try {
                runnable.run();
            } finally {
                if (previous == null) TrafficContext.clear();
                else TrafficContext.set(previous);
            }
        };
    };
}

Adapt separately for @Async, CompletableFuture, WebFlux, MQ consumers, scheduled tasks. Scheduled tasks (no entry request) are treated as "triggered traffic": record trigger time + params, replay via same cron.

Data Desensitization & Storage: Secure Yet Comparable

4.1 Desensitization: Determinism Over Strength

Production data contains PII (phone, ID card, address, password, token). Desensitization is mandatory and must happen at recording side, not storage. Key design: desensitization must be a deterministic function — same input → same output. Otherwise the same user across two recordings gets different masked values, causing false diffs.

public final class Desensitizers {
    private static final String SALT = System.getenv("TRAFFIC_SALT");
    static {
        if (SALT == null || SALT.isBlank()) {
            throw new IllegalStateException("TRAFFIC_SALT not configured, refusing to start");
        }
    }

    public static String mobile(String raw) {
        return raw.replaceAll("(\d{3})\d{4}(\d{4})", "$1****$2");
    }
    public static String idCard(String raw) {
        return raw.replaceAll("(\d{6})\d{8}(\d{4})", "$1********$2");
    }
    // For fields needing correlation (e.g., userId in URL and body), use salted hash
    public static String stableHash(String raw) {
        return "H" + DigestUtils.sha256Hex(raw + SALT).substring(0, 16);
    }
    public static String password(String raw) { return "******"; }
}

Rules applied in priority order:

DTO field annotation @Sensitive(type = MOBILE) handled at serialization.

JSON Path rules: $.data.user.mobile, $.data.list[*].idCard.

Regex fallback for unrecognized text (phone, ID, bank card, email patterns).

Gotchas: Do not desensitize amounts, order IDs, timestamps — they are core diff fields. Desensitize only PII. If phone numbers sometimes include +86 and sometimes not, mask results differ; normalize format first.

4.2 Sampling: Not All Traffic Is Worth Recording

Full recording is costly and noisy. Sampling by entry URI and tier:

Core transaction paths (order, pay, refund): 100%.

General queries: 5–20% random.

High-frequency heartbeats/polls: 0.1% or exclude.

Peak promotions: degrade to 1% to protect main path.

Error requests use a separate ERROR_ONLY channel: URIs still wrap response, but persist only when status ≥ 400 or biz code indicates failure. Don't claim 100% error recording at entry if you haven't cached the response body — you'll end up with only status codes. Sampling tag propagated via header for whole-chain consistency; error补录 tagged separately for later filtering.

4.3 Storage Selection & Lifecycle

Clear separation: MySQL for index/metadata ( traffic_id, uri, status, latency, time, tags); OSS/MinIO for full payloads (gzip → 5–20KB each); ES for search/analysis by uri, traceId, error code.

CREATE TABLE t_traffic_record (
    id           BIGINT PRIMARY KEY AUTO_INCREMENT,
    traffic_id   VARCHAR(64) NOT NULL,
    trace_id     VARCHAR(64),
    app_name     VARCHAR(64),
    uri          VARCHAR(512),
    method       VARCHAR(8),
    status       INT,
    cost_ms      INT,
    sample_tag   VARCHAR(32),  -- CORE / RANDOM / ERROR
    payload_key  VARCHAR(256), -- OSS object path
    record_time  DATETIME(3),
    env          VARCHAR(16),
    KEY idx_trace (trace_id),
    KEY idx_uri_time (uri, record_time)
) ENGINE=InnoDB;

Lifecycle: Hot (7 days, MySQL+ES for instant replay), Warm (30 days, index only, payloads in OSS), Cold (90 days, archive to infrequent-access storage), Delete after 180 days. Cleanup via sharded scheduled jobs + daily partitioned tables (e.g., t_traffic_record_20241101) to avoid large-table DELETE.

Capacity is trivial: 1000 samples × 20KB ≈ 20MB per regression run. Bottleneck is retrieval and replay scheduling, not storage.

Replay Engine: Precisely Re-enact the Past

Replay is the hardest part because it must fight the system's inherent non-determinism.

5.1 Scheduling & Concurrency Control

Replay tasks split by Suite, then by shard across multiple workers. Each worker uses a fixed thread pool; a distributed token bucket protects the target environment:

@Scheduled(fixedDelay = 5000)
public void dispatch() {
    List<ReplayTask> tasks = taskRepo.fetchPending(200);
    for (ReplayTask task : tasks) {
        if (!rateLimiter.tryAcquire()) {  // Global QPS cap, e.g., 200/s
            break;
        }
        replayExecutor.submit(() -> doReplay(task));
    }
}

private void doReplay(ReplayTask task) {
    ReplayContext.init(task);                     // Mark thread as replay mode
    try {
        TimeProvider.fix(task.getRecordTimestamp()); // Freeze business time
        RandomProvider.fix(task.getRandomSeed());    // Replay random seed
        HttpResponse resp = httpClient.execute(task.toHttpRequest());
        resultRepo.save(task.getId(), resp);
    } catch (Exception e) {
        resultRepo.saveError(task.getId(), e);
    } finally {
        ReplayContext.clear();
        TimeProvider.clear();
        RandomProvider.clear();
    }
}

Concurrency must be layered: same user / same order traffic must run serially to avoid interference; different users can run in parallel. Simple hash sharding isn't enough — same business key must land in same worker's serial queue or use distributed lock. Chosen approach: hash by business key to queue, single-threaded consumption per queue — simple but debuggable.

5.2 Timestamp & Random Noise

Top source of false diffs: LocalDateTime.now(), System.currentTimeMillis(), UUID.randomUUID(), Math.random() produce different values each run.

Option 1: Business code refactor (controlled but invasive) — introduce TimeProvider / RandomProvider with ThreadLocal overrides.

public final class TimeProvider {
    private static final ThreadLocal<Long> FIXED = new ThreadLocal<>();
    public static long currentTimeMillis() {
        Long fixed = FIXED.get();
        return fixed != null ? fixed : System.currentTimeMillis();
    }
    public static void fix(long ts) { FIXED.set(ts); }
    public static void clear() { FIXED.remove(); }
}

Cost: requires changing business code (even one line), hard to push in large systems.

Option 2: Java Agent bytecode enhancement (zero business intrusion, maintenance cost) — use ByteBuddy or jvm-sandbox-repeater to advise calls to System.currentTimeMillis, LocalDateTime.now, UUID.randomUUID in business packages, reading recorded values from ThreadLocal. Only enhance business packages; don't touch framework timeouts/logging clocks or HTTP clients break. Time freeze applies to business time only; real clock still used for timeout judgments.

Option 3: Diff-layer ignore rules (fallback) — time fields ultimately handled by ignore rules (see §6). Practice: combine — core fields via Agent for precise compare; non-core time fields via ignore rules.

5.3 External Dependency Mock: VCR Mode

Downstream state has changed; responses will differ. Solution: record downstream calls during recording, return recorded responses during replay — VCR/Betamax pattern:

public class MockingInterceptor implements ClientHttpRequestInterceptor {
    @Override
    public ClientHttpResponse intercept(HttpRequest req, byte[] body,
                                        ClientHttpRequestExecution ex) throws IOException {
        if (!ReplayContext.isReplaying()) {
            return ex.execute(req, body);
        }
        // Match key: method + path + request body fingerprint + call sequence for same key
        MockKey key = MockKey.of(req.getMethodValue(), req.getURI().getPath(), body,
                ReplayContext.nextSeq(req.getMethodValue() + req.getURI().getPath()));
        MockRecord mock = mockStore.get(ReplayContext.trafficId(), key);
        if (mock == null) {
            Metrics.counter("replay.mock.miss").increment();
            // Don't return fake 200 to fool diff; missing mock = replay failure
            throw new MockMissException("mock miss: " + key);
        }
        return MockResponses.of(mock);
    }
}

Key principle: replay environment must be physically isolated or forced through mock. Once had an incident where replay traffic hit a real payment gateway's test merchant — no money moved but downstream monitoring was flooded. MQ, RPC, Dubbo/Feign, Redis, third-party HTTP all need independent mock channels. Mock match key should include call sequence number; otherwise a retry gets the first response, making diff unreliable.

5.4 Shadow Database & Shadow Tables

Old and new system data state must match, otherwise comparison is meaningless. Approach: isolated shadow DB + data snapshot.

public class ShadowDataSource extends AbstractRoutingDataSource {
    @Override
    protected Object determineCurrentLookupKey() {
        return ReplayContext.isReplaying() ? "shadow" : "primary";
    }
}

Further refinement: shadow tables allow old and new systems to share one DB. MyBatis interceptor rewrites table names for replay traffic:

// Illustrative only; don't use this regex in production
String rewritten = sql.replaceAll(
    "(?i)\b(from|join|into|update)\s+(t_\w+)",
    "$1 $2_shadow");

Production: use JSqlParser to parse SQL, rewrite by table whitelist; regex fails on complex subqueries/joins. Before each replay, restore shadow DB from baseline snapshot ( mysqldump restore or Docker volume mount) to guarantee identical starting point. This is the prerequisite for trustworthy diffs — data drift produces masses of false failures.

Diff Comparison: From String Inequality to Semantic Difference

6.1 Structured Comparison, Not Raw String Compare

Direct JSON string comparison is worst: field order, whitespace, empty arrays vs null all cause false positives. Must do tree-structured comparison, outputting at field-path level:

public class JsonDiffEngine {
    public DiffResult diff(JsonNode baseline, JsonNode actual, List<IgnoreRule> rules) {
        DiffResult result = new DiffResult();
        walk("$", baseline, actual, rules, result);
        return result;
    }

    private void walk(String path, JsonNode a, JsonNode b,
                      List<IgnoreRule> rules, DiffResult out) {
        if (rules.stream().anyMatch(r -> r.matches(path))) return; // Ignore rules first
        if (a == null && b == null) return;
        if (a == null || b == null || a.getNodeType() != b.getNodeType()) {
            out.add(DiffItem.of(path, DiffLevel.BLOCKING, a, b));
            return;
        }
        if (a.isObject()) {
            Set<String> keys = new LinkedHashSet<>();
            a.fieldNames().forEachRemaining(keys::add);
            b.fieldNames().forEachRemaining(keys::add);
            keys.forEach(k -> walk(path + "." + k, a.get(k), b.get(k), rules, out));
        } else if (a.isArray()) {
            if (a.size() != b.size()) {
                out.add(DiffItem.of(path, DiffLevel.WARN, a.size(), b.size()));
            }
            for (int i = 0; i < Math.min(a.size(), b.size()); i++) {
                walk(path + "[" + i + "]", a.get(i), b.get(i), rules, out);
            }
        } else if (!normalizedEquals(a, b)) {
            out.add(DiffItem.of(path, DiffLevel.BLOCKING, a, b));
        }
    }
}

Distinguish missing field vs explicit null — use Jackson's has(k) not get(k). Array length mismatch: beyond WARN, decide whether extra elements need individual comparison; don't just compare min(size). Pre-compile rule patterns; don't re-parse regex at every recursion level.

6.2 Ignore Rules & Normalization

Rules configured in YAML per Suite, inheritable:

ignoreRules:
- path: "$.timestamp"
- path: "$.data.traceId"
- path: "$.data.list[*].createTime"
- path: "$.data.list[*].id"          # Auto-increment PK always changes
- regex: "^\$.data.list\[\*\]\.(updateTime|version)$"
- type: NUMERIC_TOLERANCE            # Amount tolerance 0.01 for float precision
  path: "$.data.amount"
  tolerance: 0.01
- type: ARRAY_ORDER_IGNORE           # Unstable list order
  path: "$.data.list"
  sortBy: "orderId"

Normalization equally critical. Numeric: 1 equals 1.0; amounts as BigDecimal. Time: unify format before compare — 2024-11-01 10:00:00 vs timestamp can't be naively compared. Null handling per config: null, "", [] can be treated as equivalent. Enum values/strings mapped via dictionary. Unstable list order → sort by business key then compare.

6.3 Difference Severity Levels

Not all diffs are defects. Three levels: BLOCKING: Core business field value/structure change, status code change → blocks release. WARN: New fields, optional field missing, list length change → logged for manual review. IGNORE: Matched ignore rule or noise field → hidden.

Biggest fear: WARN volume drowns BLOCKING. Report UI must aggregate by level and path so engineers can spot patterns among hundreds of diffs.

6.4 Visualization & Drill-down

Report renders diffs as three columns: path, baseline value, actual value. Click to jump to raw payload side-by-side view. Critical: when diffs number in hundreds, engineers need instant judgment — "same class of issue" vs "real breakage". Poor report UX kills platform adoption.

Load Testing & Capacity Assessment: Shadow Load Testing with Same-Source Traffic

Recorded traffic is a natural load test script — more realistic than JMeter: real parameter distributions, real call sequences, real downstream dependencies.

1x replay = regression validation, outputs diff report.

2x–20x replay = capacity assessment, observe RT, QPS, error rate, resource metrics.

Before speed-up, must decouple request dependencies (e.g., same user's order then pay) or speed-up causes chaos. Solution: bucket by userId, serial within bucket, parallel across buckets.

Must support emergency stop. Auto-pause when error rate exceeds threshold:

// Circuit breaker: error rate > threshold → auto pause replay
if (windowMetrics.getErrorRate() > 0.05) {
    replayOrchestrator.pause("error rate > 5%");
    alarm.send("Replay paused: target environment error rate abnormal");
}

Target environment also gets Sentinel rules (QPS limit, slow-call ratio degradation) as second defense.

Capacity comparison goes beyond QPS. In the order-system rewrite: old system stable at 800 QPS (P95), new system reached 1150; P99 dropped from 320ms to 210ms; CPU 78% → 55%; slow SQL 42 → 6; heap peak 4.2GB → 3.1GB. One page of such data in the acceptance report outweighs ten architecture diagrams.

CI/CD Integration: Regression as Mandatory Release Gate

Pipeline stages:

build → unit test → deploy to shadow env → restore baseline snapshot
→ replay suite → diff & gate → publish report → (fail → block)

Gate rules defined by business, not guesswork:

gate:
  blockingDiffCount: 0          # BLOCKING diffs must be 0
  diffRate: 0.001               # Diff cases / total cases < 0.1%, mainly catches WARN
  replaySuccessRate: 0.995      # Replay execution success rate >= 99.5%
  p99Regression: 0.20           # P99 RT degradation <= 20%

GitLab CI example:

stages: [build, deploy-shadow, replay, gate]

regression-replay:
  stage: replay
  script:
    - |
      TASK_ID=$(curl -s -X POST "$REPLAY_API/v1/suites/run" \
        -H "Content-Type: application/json" \
        -d "{\"suiteId\":\"order-core\",\"baseline\":\"prod-20241101\",
             \"target\":\"${SHADOW_HOST}\",\"speed\":1.0}" | jq -r .taskId)
      echo "TASK_ID=$TASK_ID" > replay.env
      curl -s "$REPLAY_API/v1/tasks/$TASK_ID/wait?timeout=1800"
  artifacts:
    paths: [replay.env]
    when: always

diff-gate:
  stage: gate
  needs: [regression-replay]
  script:
    - source replay.env
    - RESULT=$(curl -s "$REPLAY_API/v1/tasks/$TASK_ID/gate")
    - echo "$RESULT" | jq .
    - echo "$RESULT" | jq -e '.passed == true' || exit 1
  artifacts:
    paths: [report.html]
wait

endpoint should poll; don't let curl hang 30 minutes or CI runner times out.

Failed-case persistence is the highest long-term value. Every diff failure auto-enters DB with regression tag, forming a regression case pool:

CREATE TABLE t_regression_case (
    id          BIGINT PRIMARY KEY AUTO_INCREMENT,
    traffic_id  VARCHAR(64) NOT NULL,
    suite_id    VARCHAR(64),
    diff_path   VARCHAR(512),
    first_fail  DATETIME,
    last_fail   DATETIME,
    fail_count  INT DEFAULT 1,
    status      VARCHAR(16),  -- OPEN / RESOLVED / IGNORED
    owner       VARCHAR(64),
    UNIQUE KEY uk_traffic (traffic_id)
);

Next release replays this pool first — hundreds of cases in minutes as smoke test. Over a year, this pool becomes the system's real behavioral contract. Fixed cases marked resolved; re-failure triggers alert to prevent regression. Final practice: daily commits run 300 curated cases (3 min); pre-release runs full 5000 cases (~40 min).

Closing Principles & Boundaries

In a 600k-line order-system rewrite, quantified gains: regression cases from manual 300 → 5000+ real traffic; 37 regression bugs caught pre-release (5 were money-calculation precision); single regression from 3 person-days → 40 minutes, parallelizable; post-release P0 incidents from 2/quarter → 0.

Non-negotiable principles:

Recorder never steals business cycles — sidecar collection, bounded queue, drop samples on saturation, business first.

Desensitization must be deterministic — salted hash or masking, same data → same output across recordings.

Replay must be isolated — shadow DB, dependency mock, forced network cut, zero real side-effects.

Diff must be structured — field-path level + ignore rules + severity, don't let noise drown signal.

Gate must be automated — if CI can't block, platform is a toy.

Applicability boundaries: Strong stateful scenarios (e.g., flash-sale inventory) need custom snapshot/restore strategies. Full-chain mock coverage determines accuracy — one missed downstream call = mass false failures. Time/random governance is long-term debt; introduce TimeProvider abstraction from Day 1. Don't chase 100% zero-diff; accept <1% noise, focus on BLOCKING. Recording data is production data — access control and desensitization audit are compliance, not just technical.

Traffic recording/replay isn't new, but its value peaks during old↔new system transition. It replaces the shaky bridge between "what we changed" and "what users experience" with a road paved by real traffic. If your team is rewriting, adopt early — waiting for production complaints multiplies cost.

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

Traffic ReplaySpring BootTraffic RecordingRegression TestingCI/CD IntegrationShadow TrafficStructured DiffSystem Rewrite
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.