Inside Eastern Securities' Multi-Language gRPC-Nebula Platform for Microservice Governance

Background

To support rapid business innovation, Eastern Securities migrated from a monolithic architecture to a microservice‑centric architecture. The legacy securities systems (e.g., Jinshida, XinYi, Hangsheng, Dingdian, Tonghuashun) expose heterogeneous interfaces such as SPX, T2, REST, WebService, and TCP, making unified service registration, discovery, traffic control, and global monitoring difficult.

Service Governance Platform (Star‑Chen)

The platform consists of six core modules:

Registration Center – a highly available ZooKeeper‑based store that holds service metadata and configuration.

Service Consumers – retrieve registration data, invoke services, and report client‑side metrics.

Service Providers – publish service definitions and receive configuration updates.

Information Collectors – collect call traces, response times, exceptions, and other runtime metrics.

Data Processing Engine – performs real‑time performance statistics, dependency analysis, threshold alerts, clustering, and state tracking; results are stored in a performance‑management database.

Governance Portal – visualizes instances, interfaces, dependencies, health status and provides black‑white list, traffic shaping, weight configuration, and failover controls.

gRPC‑Nebula Framework Architecture

gRPC‑Nebula builds on the open‑source gRPC stack and adds comprehensive service‑governance capabilities. gRPC is chosen for its multi‑language support (C, C++, Java, Python, PHP, Node.js, C#, Objective‑C, Go, Ruby, Dart), active community, Protobuf 3.0‑based language‑agnostic serialization, and HTTP/2 transport with streaming, SSL, and custom authentication. gRPC‑Nebula integrates ZooKeeper as a registration center and implements service discovery, load balancing, black‑white list, dynamic grouping, cluster fault tolerance, and traffic control.

Key Mechanisms

Service Registration & Discovery – Providers register endpoints to ZooKeeper; consumers obtain address lists and receive push notifications on changes. ZooKeeper guarantees high availability, strong consistency via the ZAB protocol, and real‑time health checks.

Routing & Load Balancing – Supports both connection‑level and request‑level balancing with four algorithms: random, round‑robin, weight‑based, and consistent hashing.

Black‑White List – Dynamically restricts or permits specific IPs or instances.

Weight Configuration – Assigns traffic weights to instances for fine‑grained load distribution.

Dynamic Grouping – Instances carry a group attribute (e.g., data‑center) that can be configured via files or the management portal, enabling preferential routing and business isolation.

Cluster Fault Tolerance – Provides Failfast (immediate error on failure) and Failover (automatic rerouting to healthy instances) strategies with configurable failure‑count and failure‑rate thresholds.

Traffic Control – Limits request rate and concurrent connections per IP to prevent avalanche effects during traffic spikes.

Access Protection – Allows operators to mark instances as “unavailable” for graceful upgrades or fault isolation; the registration center pushes the status to all consumers.

Multi‑Registry Support – Services can be registered to multiple ZooKeeper clusters to accommodate segmented network environments.

Master‑Slave Service – Primary instances handle traffic; backups are used only when all primaries are unavailable.

Internal/External Service Visibility – Distinguishes services intended for intra‑project calls from those exposed to external systems.

Native gRPC Enhancements

Connection Backoff – Modified the exponential backoff algorithm to allow a configurable maximum reconnection delay, avoiding excessively long pauses after server recovery. See

https://github.com/grpc/grpc/blob/master/doc/connection-backoff.md

.

Keepalive Heartbeat – Exposed configurable keepalive intervals for both client and server to match firewall timeout policies (default 2 h in gRPC). See https://github.com/grpc/grpc/blob/master/doc/keepalive.md.

Performance Evaluation

Benchmark tests compared Dubbo, native gRPC, and gRPC‑Nebula. The latency and throughput of gRPC‑Nebula were within 1 % of the other frameworks, satisfying high‑performance requirements for securities trading workloads.

Deployment and Operational Results

Since early 2019 the framework has released 14 Java versions and 8 C++ versions across four platform generations. By September 2020 the platform managed 46 applications, 99 projects, 369 services, and 3 125 methods, handling tens of millions of requests per day. Integrated systems include Eastern Winner APP, Private Banking APP, Tonghuashun Institutional, Eastern Rui, Eastern Brain, intelligent advisory, and core trading platforms. Major vendors (Microsoft, Hangsheng, Jinshida, XinYi, Dingdian, Tonghuashun) have also adopted the governance framework.

Open‑Source Release and Industry Impact

The gRPC‑Nebula source code was open‑sourced on GitHub at https://github.com/grpc-nebula in June 2019. An independent community committee now maintains the project, which received the 2019 OSCAR Innovation Award and the 4th China Excellent Cloud Computing Open‑Source Case Award. The framework is promoted within the Shenzhen Stock Exchange technology alliance to encourage industry‑wide standardization of service governance.