Ant Service Mesh Exploration and Reflections (Part 2): Addressing R&D Efficiency Challenges

In the past year Ant has built extensive capabilities on Service Mesh, especially MOSN, which decouples business from infrastructure but also brings new challenges for R&D efficiency.

R&D efficiency challenges include massive code changes per MOSN release and a centralized release model that makes testing and gray‑release difficult, threatening both quality and speed.

Quality assurance focus covers multi‑module co‑development quality, version stability across clusters, and performance of testing pipelines.

Testing strategy relies on cloud‑native multi‑module data modeling and a “Test Mesh” capability that records decrypted TLS traffic, reports business‑feature and memory‑configuration data, and samples data with configurable rates while providing circuit‑breaker protection based on CPU/MEM thresholds.

Data pipeline records traffic to disk, syncs it to offline storage, cleans it via regex and deduplication, and builds business‑scenario baselines for later replay.

Scenario replay uses the recorded traffic and modeled scenarios in a MOCK mode to verify MOSN behavior offline, with a replay system handling task orchestration and result visualization.

Version stability is reinforced by a CI pipeline that aims for every PR to be immediately publishable, and a pre‑release “dry‑run” application that validates MOSN upgrades before affecting production workloads.

Low‑impact high‑frequency release ("warm upgrade") closes traffic without notifying the application, inherits dynamic configuration via a shared volume, and reconnects services automatically, enabling frequent nightly builds and rapid feedback.

Health metrics are exposed both statically (component‑level health interfaces) and dynamically (full‑stack traffic view), allowing automated pause or rollback of releases when unhealthy states are detected.

Problem diagnosis monitors CPU, RSS, and goroutine counts, triggers Go profiling (CPU, heap, goroutine) on anomalies, and stores profiles for offline analysis, which has already uncovered hidden bugs now packaged as the open‑source holmes library.

Overall, the past year’s Service Mesh rollout has dramatically accelerated infrastructure evolution, but the primary remaining challenge is improving MOSN’s own R&D efficiency; ongoing work on quality assurance, risk mitigation, and diagnostic tooling aims to unlock the full value of Service Mesh.