How to Build a Reliable Multi‑Channel Payment Monitoring System with Redis and Prometheus

Background

Third‑party payment channels are often unstable, and failures can occur without immediate detection. Traditional monitoring relies on a large number of alerts or user feedback, which is too slow for a core payment system that must provide stable services.

Challenges

Multi‑channel and multi‑entity monitoring capability

Rapid fault discovery and root‑cause localization

Minimize false positives and false negatives

Automatic channel failover

Technology Selection

1. Circuit Breaker

Hystrix was evaluated but rejected because its degradation logic works at the interface level and cannot handle channel‑ or merchant‑specific degradation, and it cannot customize traffic probing during failover.

2. Time‑Series Storage

After discarding off‑the‑shelf solutions, the team chose to build a custom monitoring system based on Redis, with Prometheus used for visualization.

Prometheus offers simplicity and low operational cost but sacrifices absolute data accuracy, which is acceptable for many metrics but not for high‑sensitivity channel switching scenarios. For example, a short spike occurring between two 15‑second samples may be missed, and latency percentiles (P95, P99) are only estimates.

3. Cost and Learning Curve

Prometheus requires a learning curve for developers, while Redis is already familiar to Java backend engineers, making it a lower‑cost choice for both development and maintenance.

Architecture Design

During payment or receipt, the system first selects a list of available channels based on routing rules, then calls the gateway to place an order or make a payment. The gateway interacts with the third‑party channel, and the result is reported back to the monitoring system via MQ.

The monitoring service listens to these messages, stores data in Redis, and a calculation module periodically pulls the data, aggregates failure rates per channel, and triggers alerts according to configured rules.

Redis data is periodically backed up to MySQL for post‑mortem analysis.

Offline jobs clean up stale Redis entries to control storage size.

Metrics are also exported to Prometheus and visualized with Grafana.

Implementation Details

Data Structure

Although a dedicated time‑series database was not used, the Redis schema follows time‑series principles:

Tags : a set records monitoring dimensions (e.g., merchant ID).

Timestamp : a zset stores seconds‑level timestamps, enabling range queries and ordering.

Fields : a hash stores actual metric values such as success count, failure count, and specific error reasons.

Example Redis storage layout:

1. set
key: routeAlarm:alarmitems
value: 微信-打款-100000111
       微信-打款-100000112
       ...

2. zset
key: routeAlarm:alarmitem:timeStore:微信-打款-100000111
score: 1657164225 value: 1657164225
score: 1657164226 value: 1657164226
...

3. hash
key: routeAlarm:alarmitem:fieldStore:微信-打款-100000111:1657164225
  success: 10
  fail: 5
  balance_not_enough: 3
  thrid_error: 2
  ...

Core Algorithm

To avoid missing short spikes and ensure no false negatives, the system uses:

Partial counting method

Sliding window aggregation

Each second a point records success and failure counts. The failure rate for a channel is computed over a configurable window (e.g., 1 minute) with a sampling frequency (e.g., every 10 seconds). The window size and frequency directly affect detection sensitivity.

Handling Low‑Traffic Channels

When a channel receives very few requests, a single failure can inflate the failure rate. The system expands the time window proportionally (e.g., from 1 minute to 2 minutes, up to 10 ×) until enough samples are collected; if the rate remains above the alert threshold, an alarm is still raised.

Results

Fast Fault Localization

Deduplication of Alerts

Automatic Recovery Detection

Future Plans

Continuously improve the monitoring algorithm to achieve >99 % alert accuracy.

Integrate with the monitoring system to enable automatic channel offline when a fault is detected.

Implement automatic channel recovery detection and online switching.