How to Implement Real‑Time API Traffic Counting at Scale

Scenario

Real‑time counting of API requests under high concurrency is required for security monitoring (e.g., detecting an IP that calls an API 1,000 times within a minute), credential‑stuffing detection (sudden login spikes with low success rates), and data‑leak tracing (who, when, and how many times a sensitive export API was called).

1. Direct database storage

Each incoming API request is written directly to a relational database. This approach is straightforward and provides immediate visibility, but the database quickly becomes a bottleneck when request volume grows, making it unsuitable for high‑concurrency workloads.

2. Flink + Kafka + Redis + Grafana pipeline

Typical flow:

API request passes through a filter that asynchronously publishes a log entry to a Kafka topic.

Flink consumes the Kafka stream and opens a fixed‑time sliding window (commonly 1 minute) with a slide interval of a few seconds (e.g., 5 s). Within each slide Flink aggregates the number of calls per API path.

The aggregated result is written to Redis using a sorted‑set (ZSET). The API path is stored as the member and the request count as the score.

Grafana connects to Redis, queries the ZSET for the top‑N APIs (e.g., top‑10), and refreshes the dashboard in real time.

Key configuration points:

Window size and slide interval determine latency vs. accuracy trade‑off.

Redis ZSET key naming should reflect the time bucket (e.g., api_counts:20230417:01) to allow historical queries.

Grafana refresh rate should match the slide interval to avoid unnecessary polling.

Advantages: fully asynchronous, does not block the business thread, Flink provides accurate windowed computation, and Redis enables fast top‑N retrieval.

3. ELK stack solution

Workflow:

Logstash collects server access logs (e.g., Nginx access logs) and parses the API path and timestamp.

Parsed records are indexed into Elasticsearch with a time‑based index.

Elasticsearch aggregation (date_histogram + terms) computes per‑API request counts for a recent interval such as the last minute.

Kibana visualizes the aggregation results, providing a ready‑made dashboard.

Pros:

No additional development; leverages existing ELK infrastructure.

Long‑term log storage enables retrospective analysis.

Cons:

Indexing introduces a short delay (seconds) before counts become visible.

Maintaining an ELK cluster incurs resource overhead.

Conclusion

Low‑traffic, simple scenarios: direct database counting is easy, fast, and stable.

High‑concurrency, low‑latency requirements: the Flink‑Kafka‑Redis‑Grafana pipeline provides asynchronous processing and real‑time top‑N results.

When a mature ELK stack already exists: using ELK avoids extra development, though it sacrifices a bit of immediacy and adds cluster maintenance overhead.