How Vipshop’s Three‑Tier Monitoring System Keeps Services Running Smoothly

Business systems are typically composed of many distributed components such as web services, RPC services, caches, message queues, and databases. When a front‑end request reaches the back‑end, it traverses multiple components, leaving scattered logs that make troubleshooting difficult.

Performance monitoring systems address this challenge by collecting, aggregating, and analyzing log information to pinpoint issues quickly, enabling developers to fix problems promptly.

Vipshop’s three‑tier monitoring

1) System layer : monitors server metrics (CPU, memory, disk, traffic, TCP connections) and database metrics (QPS, replication lag, processes, slow queries).

2) Application layer – Mercury platform: an in‑house application performance monitoring solution that injects probes into applications to monitor code, databases, and caches in real time.

3) Business layer : extracts key business indicators (PV, UV, product displays, login/registration, conversion rate, cart, order count, payment, shipment, warehouse data) either by embedding instrumentation on pages or by pulling data from business databases, then stores processed results in an operations‑maintained database for dashboard display and customizable alerts.

Mercury’s main functions

Locate slow calls (slow Web/RESTful services, OSP services, slow SQL).

Locate errors (4XX, 5XX, etc.).

Locate exceptions (exception dependency and topology).

Trace call chains, showing end‑to‑end calls with context, exception logs, and per‑call latency.

Application alerts based on predefined rules, reporting to Vipshop’s central alert platform.

Core architecture

Two data paths are used:

Raw logs (Trace/Exception) are sent through Kafka, then Flume, and finally persisted to HBase.

Log information is sent through Kafka directly to Spark Streaming, where it is analyzed and transformed into performance metrics written to OpenTSDB.

The most critical aspect of the pipeline is ensuring that data consumption does not lose or backlog messages. When alert rules configured by operations staff are satisfied, the alert module triggers actions, reporting faults to the central alert platform in real time.