Applying Flink State Management for Real-Time Recommendation Scenarios

Flink, as a pure stream‑processing big‑data engine, offers richer state management than Spark Streaming and Storm, making it well‑suited for real‑time recommendation use cases that require low‑latency aggregation of UV, clicks, exposures, and deliveries across multiple geographic dimensions.

The scenario faces two main problems: (1) the need to map a 13‑digit localId field to hierarchical region information (province, city, county) for multidimensional analysis, and (2) severe delay in real‑exposure data reporting, sometimes lasting several hours.

Although Spark is stable, its micro‑batch model and limited back‑pressure mechanisms make it unsuitable for the sub‑second processing required; Storm, while low‑latency, lacks built‑in state support. Flink’s stateful stream processing, broadcast state, and credit‑based flow control provide better latency, fault tolerance, and state consistency.

Flink’s broadcast state works like a distributed cache: a static region‑mapping table is read from HDFS, converted into a custom LocalContainerJava stream, and broadcast to all task managers via a MapStateDescriptor. Unlike Spark’s broadcast variables, Flink’s broadcast state is part of the runtime’s managed state and is checkpointed for fault recovery.

Flink distinguishes between managed (runtime‑controlled, stored in Java heap or RocksDB, checkpointed) and raw (operator‑managed) state. Managed state supports TTL, rebalancing, and various state types such as BroadcastState<K,V>, ValueState, MapState, etc., all accessed through a StateDescriptor.

The data pipeline follows a Lambda architecture: raw events are ingested into Kafka, Flink consumes them, enriches each event with the broadcast region map, and writes the results to another Kafka topic for downstream Druid ingestion. Druid stores the data as a time‑series database, enabling near‑real‑time aggregation and handling late‑arriving events.

Implementation steps include: (1) reading the region table from HDFS and creating a broadcast stream, (2) defining a MapStateDescriptor that matches the table schema, (3) connecting the broadcast stream with the main Kafka stream via BroadcastConnectedStream, (4) overriding processBroadcastElement to populate the broadcast state, and (5) overriding processElement to join incoming events with the broadcast state using a BroadcastProcessFunction. The processed results are then sunk to Kafka, Redis, MySQL, or directly to Druid.

Because some exposure data arrive with hours of delay, the solution avoids Flink‑side aggregation and instead relies on Druid’s event‑time segmentation to achieve accurate, incremental results.

In summary, Flink’s versatile state management simplifies data correlation in both real‑time and batch contexts; selecting the appropriate state type and combining Flink with complementary big‑data components like Druid can significantly improve development efficiency and system performance.

Reference: Apache Flink official documentation – State Management.