Distributed Message Governance and Microservice High‑Availability Practices

The article presents a comprehensive guide on governing traffic and ensuring high availability for the Hello mobility platform, which now includes two‑wheel (bikes, e‑bikes) and four‑wheel services (car‑hailing, ride‑hailing). Rapid traffic growth leads to production incidents, making flow control, monitoring, and fault‑tolerance critical.

What is governance? It aims to improve the operating environment by identifying shortcomings through past experience, user feedback, and industry comparison, then applying monitoring, alerts, and remediation measures.

The document is organized into three major parts:

Building a distributed message‑governance platform

RocketMQ practical pitfalls and solutions

Designing a microservice high‑availability platform

Message‑governance design guidelines focus on defining key vs. secondary metrics, abstracting middleware complexity (RocketMQ/Kafka) behind a unified SDK, and providing integrated resource control, search, monitoring, alerting, inspection, disaster‑recovery, and visual operations.

Key considerations include:

Simple, unified APIs

Safety checks for client usage

Health indicators for clusters

Visualization of common operations

Mitigation measures for identified risks

Client governance monitors usage patterns and covers scenarios such as traffic spikes, large messages, outdated client versions, consumption removal/recovery, latency detection, and troubleshooting efficiency. Required monitoring data: send/consume speed, latency, message size, node info, trace IDs, and version.

Typical governance actions:

Regular inspections to flag risky applications (e.g., latency >800 ms, message size >10 KB)

Smooth sending (traffic pre‑heating)

Consumption throttling

Consumption removal and recovery

Topic/consumer‑group governance tracks resource usage, lag, speed, node health, and partition imbalance, with measures such as real‑time alerts, scaling threads/partitions, and self‑service query tools.

Cluster health governance monitors core metrics: node count, heartbeat latency, write TPS, consume TPS, and TPS variation. Measures include periodic inspections, disaster‑recovery strategies (cross‑AZ deployment, failover), tuning of system/cluster parameters, and classification of clusters by business criticality.

RocketMQ case studies :

CPU spikes on CentOS 6 nodes were eliminated by upgrading to CentOS 7 (kernel 3.10).

Lost delayed messages were restored by deleting delayOffset.json and consumequeue/SCHEDULE_TOPIC_XXXX files and restarting brokers.

The article also emphasizes the value of reading source code for problem solving, design insight, and knowledge sharing.

Microservice high‑availability platform classifies applications into four levels (S1‑S4) based on business and user impact, and adopts grouped deployment (Stable vs. Standalone) to isolate core services. It implements circuit‑breaking, rate‑limiting, and pre‑heating mechanisms, illustrated with diagrams of traffic smoothing, queuing, and combined pre‑heat + queue scenarios.

In summary, the guide identifies key metrics versus secondary ones, distinguishes core from non‑core services, and advocates a combined source‑code‑plus‑practice approach for robust system governance.