Tackling Ultra‑Large‑Scale Service Mesh Deployment: Lessons from Alibaba

Background

Service Mesh is an infrastructure layer that handles service‑to‑service communication in cloud‑native applications. Alibaba’s massive micro‑service ecosystem built on Dubbo RPC, MetaQ and Java services faces scaling and observability challenges, especially because Dubbo’s interface‑level discovery creates an explosion of service‑endpoint metadata.

Challenges in ultra‑large‑scale deployment

Smooth evolution of a new technology without disruptive rewrites.

Balancing rapid technical iteration with stable business goals.

Managing accumulated technical debt during evolution.

Handling the massive endpoint data generated by interface‑level discovery.

Scaling deployment, rollout and upgrade of sidecar proxies.

Evolution stages of the Service Mesh implementation

Start stage : Pilot runs as a separate container in the same pod as the sidecar, enabling quick adoption of open‑source Istio and Envoy with minimal resource impact.

Three‑in‑one stage : Pilot is extracted to an independent cluster while still using xDS (LDS/CDS/RDS/EDS). The EDS pushes a huge number of endpoint IPs to sidecars, causing high CPU consumption at large scale.

Scale‑out solution : Sidecars query the service registry directly instead of receiving EDS data, reusing Envoy’s data structures and leveraging Alibaba’s incremental push capability. This dramatically reduces control‑plane traffic and CPU load.

Business‑technical co‑evolution

Short‑term benefits: middleware capabilities are offloaded to sidecars, eliminating painful SDK upgrades and making middleware upgrades invisible to business services.

Long‑term benefits: full decoupling of business logic from infrastructure, standardized micro‑service governance, multi‑language support, and contribution to global cloud‑native standards.

Transparent traffic interception and hot upgrade

Traffic is intercepted via iptables, allowing operators to enable or disable mesh insertion per application or machine through a console, providing a safe fallback to direct SDK communication.

Hot‑upgrade workflow (consumer side):

Old Envoy passes listening file descriptors to the new Envoy process.

New Envoy takes over new connections.

Old Envoy calls the RPC SDK graceful‑shutdown interface.

A 15‑second timer ensures in‑flight requests complete before the old process exits.

The same steps apply on the provider side; no special handling is required for dual‑role services.

Removing Groovy script debt

Groovy scripts tightly coupled the Dubbo framework and applications, creating governance risk. They were replaced with Istio VirtualService and DestinationRule extensions that route by application name, method and parameters, thereby paying off this technical debt.

Performance optimizations

CPU: the scale‑out solution consumes roughly one‑third the CPU of the three‑in‑one approach under large‑scale pressure.

Memory: Envoy’s memory usage dropped from >3 GB to ~500 MB for comparable workloads.

Open‑source contributions: 9 PRs to Istio and 14 PRs to Envoy (including a 50 % memory‑footprint reduction and support for Dubby/RocketMQ protocols). An attempted EGDS feature was not accepted but provided valuable lessons.

Operational infrastructure

Sidecar deployment, gray‑release and upgrades are managed via OpenKruise’s SidecarSet, providing a unified mechanism for Service Mesh sidecars.

Monitoring and alerting rely on Prometheus and ARMS. The OneOps control plane consists of a global console and a region‑aware OneOps Core operator (built on Kubernetes) that manages sidecars and ingress gateways across multiple data centers.

Application‑level service discovery

Dubbo’s interface‑level discovery generates n × m endpoint records (n interfaces, m instances), inflating control‑plane load. The scale‑out architecture moves endpoint discovery to sidecars that query the registry directly, using Alibaba’s incremental push capability to reduce the volume of data pushed to the data plane.

Systematic solution to ultra‑large‑scale problems

Continuous optimization of Service Mesh (CPU, memory, latency) through both software and hardware techniques.

Application‑level service discovery instead of interface‑level to cut metadata push volume by orders of magnitude.

Hierarchical, unit‑closed service registration for localized metadata governance.

Conclusion

Alibaba’s systematic approach—addressing technical debt, redesigning control‑plane data flow, and building robust operational tooling—demonstrates that Service Mesh can be deployed at Alibaba‑scale with acceptable CPU and memory overhead, offering a practical reference for the broader industry.