Upgrading 3000+ Services: Our Journey from Dubbo 2 to Dubbo 3 on Kubernetes

In 2022, Suzhou Zhonglun Network completed a full‑stack migration from Dubbo 2 to Dubbo 3, adopting application‑level service discovery, Kubernetes‑native deployment, and advanced service‑governance capabilities. Technical lead Lai Binbin shared the selection criteria, upgrade process, and benefits.

Company Background and Technical Stack

Zhonglun Network provides an integrated retail‑operation platform, including POS, e‑commerce, big‑data analytics, mobile payments, and more. The original microservice foundation was built on Dubbo 2.5.3 with Zookeeper, later migrated to Alibaba Cloud using ACK (Kubernetes), MSE (Zookeeper), Dubbo, and PolarDB, enabling dynamic scaling.

The rapid growth to 3,000+ partners in over 300 cities increased the number of services and deployment nodes, exposing challenges in service governance, address registration, and smooth Kubernetes‑based rollouts.

Dubbo 3 Upgrade Summary

The upgrade focused on three pain points: service‑governance, cloud‑native friendliness, and instance‑level upgrade control.

1. Service Governance

Use the Dubbo admin console to manage configurations.

Integrate Dubbo APIs directly into the system.

Dubbo 3’s extensible SPI mechanism allows custom implementations. Zhonglun built custom filters for full‑link monitoring and tag‑based routing to isolate test environments. Example usage: inv.setAttachment(TAG_KEY, routeTag); By tagging producer services (e.g., tag=fat‑v2) they could test new versions without affecting production, and later apply gray‑release strategies.

2. Cloud‑Native Friendliness

Dubbo 3 aligns with cloud‑native principles, supporting Kubernetes scheduling and synchronizing service lifecycle with container lifecycle. The team customized startup, liveness, and readiness probes.

The rollout followed a three‑stage, lossless upgrade process:

Upgrade a small subset (≈20%) of instances and shift limited traffic.

After manual verification, upgrade a larger portion (≈90%) and pause for confirmation.

Upgrade the remaining instances, verify, and complete the release.

During upgrades, any abnormal metrics (CPU, memory, 500 errors) trigger an immediate rollback.

Because Dubbo services register themselves before Kubernetes marks the pod ready, premature exposure can occur; the team mitigated this with Dubbo Qos commands: dubbo qos offline to gracefully stop instances and dubbo qos online to bring them back once ready.

3. Instance‑Level Upgrade & Metadata Center

Dubbo 3 introduces application‑level service discovery, reducing the number of service entries the registry must push. With a metadata center, interface configuration size shrinks, lowering memory consumption and registration‑center load, resulting in higher stability.

Conclusion and Outlook

The migration required careful risk assessment, but it delivered significant benefits: higher resource utilization, expanded core functions, and improved stability. All business domains have now completed the switch to Dubbo 3.

The team also contributed back to the Apache Dubbo community, encouraging others to join the contributor group (DingTalk group 31982034) and follow the weekly technical sharing sessions.