How AutoStream Scales Real‑Time Data Processing with Flink, Iceberg, and PyFlink

Application Scenarios

Our use cases are similar to other companies and cover real‑time metric statistics, monitoring alerts, real‑time data processing, user behavior tracking, data lake ingestion, and data transmission. Specific examples include traffic analysis dashboards, event impact monitoring, backend log alerts, video playback quality monitoring, content recommendation, and Kafka‑to‑ES data distribution.

Platform Evolution

Initially we used a Storm platform with pure Java code, which had high development overhead. In 2018‑19 we introduced Flink, building AutoStream 1.0 and moving to a SQL + UDF model, creating our own meta‑server for DDL. AutoStream 2.0 upgraded Flink to 1.9, added native DDL, user‑uploaded UDFs, health scoring, and online diagnostics. AutoStream 3.0 upgraded Flink to 1.12, enabling lakehouse integration, native Kubernetes deployment, PyFlink, and intelligent auto‑scaling.

Budget Resource Management

We enforce budget limits by linking teams to an internal asset‑cloud system; tasks exceeding the budget cannot start. Users must first optimize low‑utilization tasks (target ≤10% low‑utilization). If optimization fails, they can request inter‑team budget transfers or receive temporary whitelist support.

Resource‑utilization rules focus on CPU usage, memory usage, and idle slots, leveraging the health‑scoring data introduced in AutoStream 2.0 to identify low‑utilization jobs.

Automatic Scaling

Auto‑scaling reduces cost and improves resource utilization while lowering tuning effort and minimizing impact on job stability. Users can schedule scaling windows (e.g., off‑peak nights) and define CPU or memory scaling strategies. The system adds a RescaleCoordinator component in the JobManager, which periodically checks scaling policies, requests new TaskManager containers, releases old ones, rebuilds the execution graph, and persists the new configuration to ZooKeeper and HDFS.

Scaling decisions are based on CPU usage, parallelism, Kafka lag, and idle slots. Parallelism scaling increases parallelism when CPU is low but Kafka lag is high; down‑scaling removes idle slots.

Lakehouse Integration

We replaced the Hive‑based warehouse with an Iceberg‑based lakehouse to address timeliness (t+1), lack of upsert support, and immutable schema. Iceberg provides open table format, incremental snapshots, ACID semantics (v2), and seamless integration with Flink, Spark, and Hive.

Key implementation details include:

Defining bucket properties in Flink DDL to co‑locate rows with the same primary key.

Persisting watermarks into Iceberg metadata to track ingestion progress.

Introducing ZooKeeper‑based distributed locks to replace HiveMetaStore locks, solving lock‑hang issues.

Adding fault‑tolerant metadata updates to avoid accidental deletion of metadata files.

Implementing upsert semantics using sequence numbers, equality‑delete files, and position‑delete files.

Hourly small‑file merging for v2 tables while preserving sequence numbers to avoid transaction conflicts.

The lakehouse integration yielded faster data freshness (sub‑10‑minute SLA), improved feature‑engineering latency, and enabled near‑real‑time materialized views.

PyFlink Practice

We integrated PyFlink to expose Flink’s streaming capabilities to AI teams that primarily use Python. The architecture connects a Python VM with the JVM, allowing developers to write Python code that maps to Flink APIs.

Deployment runs on Kubernetes for resource isolation and easy inclusion of Python and ML dependencies. Dependency management supports Java JARs, Python files via the platform’s file service, and third‑party Python packages automatically installed before job start.

We also exposed the platform catalog to Python, enabling reuse of existing tables and UDFs, and provided a simple workflow for registering custom Python UDFs.

Future Plans

We will continue optimizing resource utilization through more granular Yarn scheduling and expanded auto‑scaling policies, further mixing real‑time and batch workloads. In the lakehouse area we will explore Flink batch capabilities, FLIP‑188 proposals for unified log‑store and file‑store writes, and broader support for stream‑batch convergence.