Scaling NetEase Kaola’s Backend: Dubbo‑Driven Architecture, Governance & Monitoring

Kaola Architecture Evolution

When Kaola launched in early 2015, all functions such as product detail pages and shopping cart were tightly coupled in a single online project, which was efficient for a small team but later caused low development efficiency and high deployment cost as the business grew.

To address these issues, Kaola began splitting the monolith into distributed services, creating separate centers for products, users, and orders, and gradually refined service granularity.

Service‑Oriented Practices

Dubbo was chosen as the core micro‑service framework to handle service definition, registration, discovery, routing, and governance. Kaola added circuit breaking (via Hystrix), rate limiting (custom NFC framework built on Dubbo filters), and integrated these controls into a monitoring platform.

Circuit Breaking

Remote calls introduce latency and failures; circuit breakers prevent cascading failures by short‑circuiting calls when a provider becomes unstable, with automatic recovery once health is restored.

Rate Limiting

Kaola’s NFC framework enforces concurrency and QPS limits on Dubbo services and URLs, reporting metrics to a real‑time monitoring dashboard.

Monitoring Enhancements

Monitoring now includes four dimensions: logs, metrics, tracing, and health checks. Service metrics (CPU, latency, QPS) are collected, while a custom trace platform captures Dubbo call chains for latency and dependency analysis. Health checks verify readiness of dependencies before registration.

API Gateway

Kaola built a custom API gateway that forwards external traffic to Dubbo services, providing routing, flow control, and logging. The gateway uses Dubbo’s generic invocation, which was optimized to outperform standard calls.

Multi‑Language Support

To enable Node.js clients, Kaola exposed Dubbo services via a JSON‑RPC protocol alongside the native Dubbo protocol, handling protocol selection and serialization compatibility.

Registration Center Bottleneck and Solution

With over 4,000 Dubbo instances and more than a million ZooKeeper nodes, Kaola faced write‑performance limits and “thundering herd” effects during service publishing. The team designed a new registration center that stores only essential IP/port data in ZooKeeper, off‑loads metadata to Redis, and plans to adopt Nacos for future stability.

Future Plans

Kaola is constructing a second data center for active‑active deployment, enhancing Dubbo routing to prefer same‑site instances, and exploring containerization, DevOps pipelines, and Service Mesh to further decouple services and support multi‑language ecosystems.