Single-Task Recovery in Flink: Design and Implementation for Real‑Time Stream Processing

In ByteDance's real‑time computation environment, many streaming jobs (over 2k) serve online services directly, making output latency and stability critical. These jobs exhibit high traffic, massive parallelism, multi‑stream join topologies, and tolerate minimal data loss while demanding continuous output.

Under Flink's default architecture, a failure of a single task in a multi‑stream join triggers a full job redeployment, causing several minutes of output interruption—unacceptable for online products.

The proposed single‑task recovery approach enhances the network layer so that when a machine goes down or a task fails, only the failed task is failovered, while non‑failed tasks continue serving data. Upstream tasks discard data destined for the failed task, and downstream tasks clear incomplete buffers before re‑establishing connections after failover.

Solution Overview

1. Upstream sender optimization : Subpartitions receive a failure notification, become unavailable, and RecordWriter discards data for unavailable subpartitions. After failover, the subpartition is marked available again.

2. Downstream receiver optimization : InputChannels insert an UnavailableEvent into their buffer queue when a upstream task fails; the InputProcessor later clears the buffered data, avoiding cross‑thread calls.

3. JobManager restart strategy : Extends Flink's RestartIndividualStrategy to redeploy only the failed task and, using the execution graph, update downstream tasks via RPC. A CachedChannelProvider handles cases where channels are still consuming data, caching updates until buffers are drained.

Implementation Highlights

The design reuses Flink's existing Netty and task thread models, adding an isAvailable flag to Subpartitions and Channels while ensuring proper visibility across threads. A timer fallback forces failover if channel re‑initialization exceeds a configurable timeout.

Results

Deployed in over 1,000 production jobs, the single‑task recovery reduces failover time from 81 seconds to 5 seconds in a 4,000‑slot job, affecting only 0.1% of slots and dramatically improving downstream service continuity.

Conclusion

The solution provides a practical, low‑impact way to achieve fast, localized recovery in large‑scale Flink streaming jobs, significantly enhancing reliability for latency‑sensitive online services.