Designing a Millisecond‑Level Real‑Time Data Processing System

Introduction

Backend engineers often face the requirement to process incoming data "immediately" with latency measured in milliseconds. Whether it is fraud detection at order time, IoT sensor alerts, or click‑through‑based recommendation adjustments, the core problem is reducing end‑to‑end latency.

Core Latency Trade‑offs

The fundamental tension is "fast, accurate, stable". Speed and accuracy conflict because precise results need more data, while stability mechanisms such as back‑pressure, retries, and checkpoints add latency. Teams typically set a latency target (e.g., P99 ≤ 200 ms) and allocate a time budget to each pipeline stage.

Example: an e‑commerce fraud system with a 150 ms budget splits it into data ingestion 10 ms, deserialization 5 ms, rule evaluation 50 ms, state lookup 30 ms, result write‑back 20 ms, and network transmission 35 ms.

Overall Architecture

Data flows left‑to‑right, each layer performs its own responsibility, and layers are decoupled via asynchronous messaging.

Data Ingestion Layer: Speed Without Loss

Protocol choice : For sub‑50 ms latency, replace HTTP/JSON with gRPC + Protobuf or a native Kafka client. Tests show 1 KB messages take ~8 ms over HTTP/JSON versus ~2 ms with gRPC.

Kafka 4.0 : Removes ZooKeeper, adopts KRaft mode, reducing metadata latency from seconds to milliseconds. Upgrading from 3.x is strongly recommended.

Partition strategy : Default hash partition can overload hot keys. A custom partitioner that double‑hashes hot keys solved a case where the top 1 % of users generated 30 % of traffic and caused partition saturation.

Batch vs. latency : linger.ms and batch.size control the trade‑off. Setting linger.ms=5 (or 0) yields low latency for workloads under ten million events per day; larger volumes may require larger values.

Stream Computing Engine Selection and Tuning

Apache Flink remains the de‑facto standard in 2026, with Flink 2.x introducing architectural changes. Flink’s record‑by‑record processing model suits low‑latency needs better than Spark Structured Streaming’s micro‑batch approach, which bottoms out at hundreds of milliseconds.

Apache Kafka Streams is viable for lightweight scenarios, but for complex windowing and state management Flink is more mature. RisingWave, a streaming database, offers SQL‑based stream processing and is suitable for real‑time OLAP workloads.

Key tuning knobs:

Operator chaining: avoid disabling chaining ( disableChaining()) unless necessary.

Network buffers: taskmanager.network.memory.buffers-per-channel default of 2 is conservative; increasing to 4‑8 can reduce transmission latency.

Parallelism: Scaling from 16 to 64 parallelism increased latency in a test because more state shards caused longer checkpoint times and more back‑pressure. Choose parallelism based on Kafka partition count and actual QPS.

State Management: The Often‑Overlooked Bottleneck

State read/write dominates latency in many real‑time use cases, such as a de‑duplication check over a 5‑minute window with 100 k QPS, which may involve 30 million state entries.

Flink state backends:

HashMapStateBackend : Stores state in JVM heap, nanosecond‑level access, but limited by memory size and GC pauses; suitable for < few GB state.

EmbeddedRocksDBStateBackend : Persists state in RocksDB, supports incremental checkpoints, handles terabyte‑scale state with microsecond‑to‑millisecond latency; used by most production systems.

Practical advice: when using RocksDB, set state.backend.rocksdb.localdir to an SSD (NVMe). A misconfiguration to HDD caused P99 latency to jump from 80 ms to 600 ms.

Flink 2025 introduced ForSt (Fork of RocksDB) as a next‑generation state store optimized for streaming workloads; new projects should adopt it.

Output Layer and End‑to‑End Latency Optimizations

Choosing the sink depends on downstream consumption:

Real‑time dashboards: write to Apache Doris or ClickHouse (write < 10 ms, query < 100 ms).

Real‑time alerts: push directly to Kafka for alert services (latency < 20 ms).

Feature updates: write to Redis Cluster (latency < 5 ms).

Online query services: write to TiDB or ScyllaDB (latency < 10 ms).

Using MySQL as a sink often becomes a bottleneck; a pattern of Redis hot‑data buffering plus asynchronous batch persistence offers higher stability.

Batching sink writes matters: Redis SET per record vs. pipeline batch can differ by a factor of ten. Flink’s RedisSink defaults to per‑record writes, so custom pipeline implementation is required for low latency.

Monitoring and Back‑Pressure Mechanisms

Effective monitoring must track three key metrics:

Kafka Consumer Lag – growing lag indicates the compute layer cannot keep up.

Flink Back‑pressure – visible in the Flink Web UI; persistent HIGH state points to slow operators or sink bottlenecks.

End‑to‑End Event‑Time Latency – measured with OpenTelemetry instrumentation, not just Flink’s processing time.

Flink’s credit‑based back‑pressure propagates upstream when downstream stalls, but the propagation itself adds latency (hundreds of ms to seconds). Reserving extra Kafka partitions as buffers helps absorb traffic spikes before back‑pressure takes effect.

Conclusion

Designing a low‑latency real‑time system is about meticulously accounting for time at every stage—ingestion, computation, state handling, and output—while relying on comprehensive monitoring to validate performance.

Define latency budgets before selecting technologies.

Prefer asynchronous decoupling via message queues.

Treat state management as a first‑class concern.

Monitoring is mandatory; you cannot optimize what you cannot observe.

Adopt the 2026 mainstream stack: Kafka 4.0 (KRaft), Flink 2.x with ForSt, and OpenTelemetry for tracing; these have proven stability in large‑scale internet companies.

Finally, question whether millisecond‑level latency is truly needed for the business case; over‑design can be avoided in many scenarios.