Understanding Flink Sliding Windows and Performance Optimizations

Flink's window mechanism is a core component for efficient stream processing, with WindowAssigner as the base class that determines which windows an element belongs to.

The article focuses on sliding windows, illustrating a typical 2‑minute size and 1‑minute slide example with a diagram.

Example code for assigning a sliding event‑time window:

dataStream
  .keyBy("userId")
  .window(SlidingEventTimeWindows.of(Time.minutes(2), Time.minutes(1)));

The processElement() method of WindowOperator assigns windows, updates state, triggers output, and registers cleanup timers, as shown in the provided source snippet.

@Override
public void processElement(StreamRecord<IN> element) throws Exception {
    final Collection<W> elementWindows = windowAssigner.assignWindows(
        element.getValue(), element.getTimestamp(), windowAssignerContext);
    boolean isSkippedElement = true;
    final K key = this.<K>getKeyedStateBackend().getCurrentKey();
    if (windowAssigner instanceof MergingWindowAssigner) {
        // session window logic omitted
    } else {
        for (W window : elementWindows) {
            if (isWindowLate(window)) {
                continue;
            }
            isSkippedElement = false;
            windowState.setCurrentNamespace(window);
            windowState.add(element.getValue());
            triggerContext.key = key;
            triggerContext.window = window;
            TriggerResult triggerResult = triggerContext.onElement(element);
            if (triggerResult.isFire()) {
                ACC contents = windowState.get();
                if (contents == null) {
                    continue;
                }
                emitWindowContents(window, contents);
            }
            if (triggerResult.isPurge()) {
                windowState.clear();
            }
            registerCleanupTimer(window);
        }
    }
    // side‑output late data logic omitted
}

The SlidingEventTimeWindows.assignWindows() method computes the start of each window based on the element timestamp and iterates to create size / slide windows, illustrating how granularity affects the number of windows an element belongs to.

@Override
public Collection<TimeWindow> assignWindows(Object element, long timestamp, WindowAssignerContext context) {
    if (timestamp > Long.MIN_VALUE) {
        List<TimeWindow> windows = new ArrayList<>((int) (size / slide));
        long lastStart = TimeWindow.getWindowStartWithOffset(timestamp, offset, slide);
        for (long start = lastStart; start > timestamp - size; start -= slide) {
            windows.add(new TimeWindow(start, start + size));
        }
        return windows;
    } else {
        throw new RuntimeException("Record has Long.MIN_VALUE timestamp (= no timestamp marker). " +
            "Is the time characteristic set to 'ProcessingTime', or did you forget to call " +
            "'DataStream.assignTimestampsAndWatermarks(...)'?");
    }
}

public static long getWindowStartWithOffset(long timestamp, long offset, long windowSize) {
    return timestamp - (timestamp - offset + windowSize) % windowSize;
}

Fine‑grained sliding windows (e.g., 3‑minute frequency for a 24‑hour PV/UV calculation) cause performance problems because each element must be written to many window states and many timers are registered, increasing memory and checkpoint overhead.

The recommended workaround is to replace the sliding window with a tumbling window of the slide length, perform aggregation inside each tumbling window, and store the intermediate results in an online store such as Redis or HBase. The final 24‑hour PV/UV can be obtained by reading and merging the stored partial results.

Using Redis, timestamps can be part of the key with a 24‑hour TTL; PV is stored as a numeric string, while UV uses HyperLogLog. Simple increment and pfmerge/pfcount operations yield the desired metrics.

Overall, the article provides a deep dive into Flink's window internals, highlights the cost of high‑granularity sliding windows, and offers a practical solution leveraging tumbling windows and external storage for scalable real‑time analytics.