CTrip’s CDubbo Journey: Scaling 10k Services with Registration, Monitoring, and Service Mesh

Background

CTrip started micro‑service exploration on a .Net stack, later migrated to Java and replaced a self‑built framework with the high‑performance CDubbo framework, which combines CTrip’s governance layer (C) with Alibaba’s open‑source Dubbo SDK.

Past: Self‑Built Service Framework

The early architecture used an ESB bus, causing single‑point failures and whole‑network outages. Switching to a registry‑based SOA model distributed calls and mitigated these risks, but the custom framework still suffered from scalability and maintenance issues.

Current: CDubbo Service Framework

CDubbo has been deployed to nearly ten thousand service instances since its first release in April 2018. Its core capabilities are:

1. Registration & Discovery

Service registration integrates with CTrip’s registry and supports health‑check extensions. Services send a heartbeat every 5 seconds; missing N heartbeats triggers automatic client notifications. Clients subscribe using a push‑pull model to achieve eventual consistency. Dubbo’s extension mechanism enables custom routing based on method name, request parameters, or locality (prefer same‑datacenter nodes).

2. Monitoring – CAT

CAT provides distributed tracing, detailed reports, and scenario‑based problem analysis. It shows total request volume, per‑machine QPS, latency percentiles (e.g., 99th), and pinpoints slow stages (client, server, serialization, execution). Exception stacks are captured for rapid debugging.

3. Monitoring – Metrics

CTrip’s Dashboard displays global request counts, error rates, thread‑pool statistics, and allows filtering by datacenter, protocol, or serialization format. Custom alert rules enable early intervention when anomalies arise.

4. Dynamic Configuration

Hot‑reloading configuration allows rapid adjustments without redeploying services. For example, during a datacenter outage a global check can be disabled to break circular dependencies. Method‑level timeout overrides (default 1 s, slow methods 5 s) are configurable via a visual UI.

5. SOA Protocol Compatibility

CDubbo accepts existing SOA requests via a Tomcat port, reusing the current serializer to convert request objects while preserving Dubbo’s internal filter chain. This enables dual‑protocol operation without code changes.

6. Test Platform

Because the binary protocol lacks generic tools, CTrip built a custom test client (coreStone on GitHub) that leverages Dubbo 2.7.3’s metadata center and generic invocation. It supports direct connections, local testing, and protobuf serialization; the protobuf test suite has been contributed upstream.

7. Upgrade to Dubbo 2.7.3

Upgrade details are documented at https://www.infoq.cn/article/kOxdaV3y9fMZ0Bzs0jb2. After migration, 99 % of CTrip services run on Dubbo 2.7.3 with zero incidents; incompatibilities were caught at compile time.

8. Threadless Execution (Dubbo 2.7.5)

Threadless mode removes the DubboClientHandler thread, allowing Netty I/O threads to hand responses directly to business threads. This reduces thread count by 60‑70 % in high‑QPS, multi‑service scenarios.

9. CDubbo Service System

The system simultaneously supports Dubbo (TCP) and SOA (HTTP 1.1) protocols, handling internal and external gateway traffic while unifying configuration and preserving SOA serialization formats.

10. Performance

Dubbo’s protocol reduces average latency from ~1 ms (SOA) to ~0.3 ms. Under 3‑4× traffic spikes, client error rates remain zero, and long‑connection multiplexing provides strong resilience.

11. Extensibility

Dubbo’s extensible architecture allows teams to add custom routers, load‑balancers, or replace the transport layer, addressing the 80/20 rule where a small portion of framework work consumes most effort.

12. Ecosystem

Leveraging open‑source Dubbo admin, Dubbo‑go, and other components reduces development and learning costs, allowing teams to adopt familiar tools across companies.

13. Dubbo Protocol Issues & Dubbo 3.0 Roadmap

Current Dubbo 2.x protocols lack gateway friendliness and lightweight mobile SDKs. Dubbo 3.0 aims to introduce a next‑generation protocol, application‑level service discovery, and cloud‑native infrastructure support. CTrip contributes to its development.

Future: Service Mesh

Standardizing on a Service Mesh (e.g., Istio control plane with Envoy or Sofa‑Mosn data plane) can lower R&D cost, decouple processes, enable multi‑language support, and improve cloud deployment flexibility.