Traffic Recording and Playback: vivo's Moonlight Treasure Box Platform Practice

This article introduces vivo's traffic recording and playback solution for regression testing, presented at the 2021 vivo Developer Conference. As vivo's internet business scaled with numerous high-concurrency user-facing applications, traditional testing methods proved insufficient for ensuring system correctness during iterations.

The platform, named "Moonlight Treasure Box" (月光宝盒), is built on the open-source Jvm-Sandbox-Repeater project with custom modifications. It consists of server-side modules (task management, data management, coverage analysis, configuration, monitoring) and a Java Agent with a four-layer architecture implementing bytecode instrumentation.

The recording process captures entry calls and sub-calls, binding them with a unique ID. Key challenges addressed include: Full GC issues (solved through concurrency limits and monitoring), call chain tracking across custom thread pools (solved via collaboration with tracing systems), data security through configurable desensitization, and traffic deduplication based on parameters and execution stack.

The playback process re-invokes recorded traffic against updated systems, mocking external calls. Challenges solved include: time difference handling (bytecode manipulation to synchronize timestamps), system noise reduction (configurable noise fields at global/application/interface levels), environment unification (recording and applying online configurations), sub-call matching strategies (different approaches for cache, HTTP, Dubbo), and problem investigation tools.

The platform supports dozens of plugins covering common middleware and provides additional features including visual call chain analysis, regression code coverage statistics, and scheduled recording/playback tasks. In under a year, it integrated nearly 200 business systems, completed over 10,000 recording/playback sessions, and identified dozens of production issues, improving team efficiency by over 80%.