Alibaba Cloud Flink + Fluss: Streaming Agent & Lake-Stream Unity for Real-Time AI

As AI shifts from answering questions to continuously participating in business, the challenge evolves from understanding a single input to supporting 24/7 Streaming Agents that must keep pace with live score changes, detect anomalies in continuous video feeds, and make decisions combining latest orders with historical records. Supporting these scenarios requires both understanding multimodal data in motion and providing Agents with fresh, complete, and rapidly accessible business context.

From Lakehouse to Agentic Lake: AI-Driven Data Architecture Evolution

Traditional lakehouses serve human analysts with structured tables, reports, and batch analytics. As Agents become primary data consumers, platforms must support multimodal context understanding, latest context retrieval, and business decision-making. The Agentic Lake paradigm unifies continuous perception, multimodal processing, and real-time context serving.

Alibaba Cloud Flink Releases Streaming Agent

Streaming Agent connects continuous data ingestion, streaming AI processing, Agent orchestration, and business output. It targets continuously running business processes: new frames, documents, or events trigger timely understanding, context updates, and downstream actions.

1.1 Flink CDC: From Database Change Capture to Multimodal Continuous Perception

Flink CDC extends beyond database changelogs to three new categories:

OSS CDC : Captures object creation, update, deletion, and version changes in real time, passing object references, metadata, and timestamps downstream for understanding and lake ingestion. Supports images, video, audio, PDF, DOCX, JSON.

Doc CDC : Captures changes in collaborative documents (DingTalk, Yuque, Feishu), syncing only changed segments. Downstream recomputes embeddings only for changed fragments, improving real-time knowledge base updates and reducing build/maintenance cost.

Video Stream : Ingests live streams, camera feeds, audio via HLS, RTMP, RTSP. Performs slicing, keyframe extraction, audio extraction, forming events by time window, keyframe, or scene for lake persistence.

Thus Flink CDC becomes a continuous perception and real-time ingestion layer for multimodal content changes, while data understanding is handled by Flink AI's multimodal capabilities.

1.2 Flink AI: Encapsulating Multimodal Understanding as Standard Stream Operators

Flink AI adds three operator categories atop traditional structured operators:

Multimodal operators : Preprocessing — document parsing, codec, image resize, format conversion, video slicing, keyframe extraction — transforming raw content into model-ready inputs.

AI operators : Encapsulate model capabilities as orchestratable operators via SQL, Python, YAML. Tasks include classification, extraction, embedding, summarization, translation, tagging.

Agent operators : Scenario-oriented encapsulations involving multi-round LLM calls, context retrieval, tool use. Examples: sports commentary, store monitoring, highlight news, AI live streaming.

Processed multimodal data becomes structured records, semantic tags, summaries, vector representations persisted in the data lake — not just raw files but understood assets.

1.3 Live Commentary Case: CCTV Sports Continuous Perception & Event Production

For the 2026 Nagoya major games, CCTV Sports and Alibaba Cloud built a real-time AI commentary Agent for swimming and shooting. The pipeline ingests live video, schedules, rosters, official real-time events. Video is parsed, standardized, keyframe-sampled, timeline-normalized. Structured event data correlates with frames to identify current match, athlete, round. Observation plugins extract action changes, score overlays, result graphics.

Understanding combines official data, match phase, sport rules to distinguish replays from current results, provisional leads from final outcomes. Rankings, scores, medals rely on official confirmation; insufficient evidence withholds definite conclusions. Averages, differentials computed structurally, then phrased by model for factual grounding.

Flink state management maintains continuously updated match context: current phase, shot-by-shot scores, lead relationships, confirmed facts. Shooting compares recent five-shot average vs prior five, with rank changes to narrate chase; swimming tracks start, turn, exchange, touch, linking segment performance to final result. Continuous memory reduces repetitive scoring and inconsistency.

Output schedules content by event importance, fact reliability, available airtime: key scores first, gaps filled with rules/player background. Scripts pass fact/number verification and content review before TTS and broadcast arrangement; pending content withdrawn/updated per match progress. Three commentary styles (low steady, high energy, clear gentle) leverage live safety review window for near-sync AI commentary.

Confirmed facts also drive highlight production: key scoring, lead changes, eliminations, championships trigger highlight Agent to determine video boundaries, clip/merge, cover selection, title generation, upload. Highlight pipeline completes synthesis and upload within one minute of event (excluding manual review).

Stability: async model/TTS calls, concurrency control, timeout policies reduce backpressure; Flink checkpoints provide consistent snapshots of managed compute state with recoverable sources for context recovery. Executed actions recorded for idempotent/deduplicated output. Primary/standby failover, monitoring alerts, human on-call ensure continuous live operation, demonstrating production-grade collaboration of stream compute, models, media components.

1.4 Short-Video Marketing Case: Video Understanding, Semantic Search, Remix-to-Traffic in Real-Time Pipeline

A short-drama marketing company faces thousands of AI-generated dramas uploaded daily. Need faster, more accurate traffic investment. Thousands of clips per drama; operators must find highlights, assemble into traffic-ready videos.

Solution: Flink + Agentic Lake. Video uploaded to OSS triggers OSS CDC event, Flink slices and extracts keyframes. Flink AI calls Qwen, ASR to understand visuals and audio, extracts summaries/tags around characters, conflicts, plots, computes embeddings. All stored in Agentic Lake multimodal wide table managed by DLF Paimon. Operators query via StarRocks for desired shots, highlights, conflict plots — e.g., "protagonist humiliated then reversal highlights" recalls "present evidence", "situation reversal" scenes. Retrieved clips undergo post-processing/remix (montage, narration, plot compression) into multiple versions. Post-launch feedback (exposure, clicks) loops back for strategy optimization.

Results: video upload-to-searchable within 30 minutes, seizing traffic window, lowering operator barrier. Business value: faster output, lower threshold, higher per-person efficiency, multi-version per drama expanding revenue opportunity.

1.5 AI Store Inspection: Turning Surveillance Video into Actionable Store Events

Store cameras capture shelf stockouts, queue congestion, disputes, but recordings don't guarantee timely detection. Operations need actionable events: which store, when, what happened, still ongoing, supporting frames. Flink video stream processing converts continuous feeds into timestamped, stateful, evidenced event data, moving monitoring from post-playback into daily operations.

Two processing modes: Slow-changing shelves — low-frequency periodic frame capture, check obvious continuous gaps, combine with follow-up inspection to confirm persistent stockout. Event logic records detection, restock reminder, recovery, deduplicates notifications. Dynamic disputes — single frames lose context. Flink video operators detect activity, select representative keyframes from start, duration, end phases as chronological evidence for multimodal model. Model outputs structured "suspected", "needs attention", "no anomaly" for staff confirmation; it identifies risk clues, not liability judgment.

Both modes share same video source, split into two branches in one Flink job, merge into unified LLM call and event result table. Cameras ingest via RTSP, RTMP, HTTP/HLS through video-stream connector; Python DataFrame chains frame extraction, activity detection, evidence selection, model calls — no per-task stream pulling.

Cost control: filter evidence before model calls. One 25 fps stream yields ~2.16M frames/day, mostly static/duplicate. Periodic sampling + dynamic keyframe selection reduces invalid inputs. Flink AI Function uses llm.predict with business requirements and multiple evidence images, async calls, retries, batch inference. Optimized end-to-end test achieves ~1 RMB/day per stream; depends on sampling frequency, activity density, model choice, resource config — not a fixed service price.

Results written to DLF Paimon multimodal storage: event type, timestamp, processing status, evidence list, representative frames. Downstream supports alerts, timeline queries, historical retrospectives, manual confirmation/handling fed back into business flow. Ops track gap duration, restock response time; staff track valid alert rate, confirmation time. Scales via parallelism as stores/cameras grow. Value: reduces manual review burden, precipitates operational phenomena into queryable, retrospectable data assets for replenishment, scheduling, service checks, risk handling.

1.6 Other Industry Applications: Streaming Agent Across Verticals

Capabilities replicate across domains: CCTV sports (continuous feed + official data for AI commentary), store monitoring (real-time shelf stockout alerts), embodied intelligence, insurance claims, live operations. Eight application categories: embodied/autonomous driving video auto-annotation, short-drama assetization, store monitoring, sports events, insurance claims, IT operations, e-commerce product review, live operations. Common pattern: connect continuous data, stream processing, models, Skills, tools for content processing, issue identification, business actions.

1.7 Flink AI Ops: LLM-Powered Cloud Flink AI Companion

As multimodal processing and Agent apps run continuously on Flink, development, debugging, operations need upgrade. Latency, backpressure, checkpoint timeouts, resource utilization interrelate; single alerts rarely reveal root cause. Flink AI Ops connects intelligent assistant, intelligent inspection, open integration around same runtime data and permissions, supporting instant troubleshooting to continuous inspection.

Intelligent Assistant : User-initiated dev/ops queries. Correlates Metrics, Logs, Events, Traces across cluster, instance, job, operator levels, combines expert knowledge base and model reasoning. Answers Watermark, State, Connector concepts; generates/explains SQL for specific jobs, diagnoses anomalies, analyzes performance, recommends resource config.

Example : Sustained latency rise, frequent checkpoint failures → assistant correlates backpressure topology, checkpoint history, TaskManager memory/CPU, identifies hot-key-induced operator load imbalance, suggests aggregation optimization or parallelism adjustment. User confirms, ops pipeline executes config change, stateful restart, run-state verification, linking diagnosis to action.

Intelligent Inspection : Continuously discovers unraised issues. Periodic workspace job scans aggregate resource utilization, job health, checkpoint stability, backpressure/latency, connector/state management into health view, organized by risk level. Each anomaly links cause analysis and fix suggestions; resource analysis identifies over-provisioning or capacity gaps for scaling/cost optimization.

Boundaries : Distinguish auto-inspection, diagnostic suggestions, auto-repair; risk discovery ≠ default permission to modify production jobs.

Open Integration : Standard Skills for instance lifecycle, workspace ops, knowledge Q&A covering resource provisioning/query, draft/deploy/job/diagnosis management, SQL generation, parameter recommendation, doc retrieval. Enterprise Agents reuse via OpenAPI/Skill encapsulation, avoiding repeated interface integration and diagnostic flow reconstruction.

Guardrails : Parameter validation, state checks, credential protection, idempotency control. High-impact ops (stop job, delete resource) require explicit impact acknowledgement and confirmation; retries avoid duplicate changes. AI Ops connects runtime data, domain knowledge, controlled operations for continuous Streaming Agent and real-time data pipeline operation.

Alibaba Cloud Fluss Lake-Stream Unity Accelerates Real-Time Context for Agents

Multimodal solves what Agents can understand; real-time determines if information supports current action. Agents iterate observe-reason-act; data must meet three requirements:

Freshness : Latest state — latest orders, user behavior, device status — for correct, timely decisions. Offline snapshots insufficient.

Completeness : Not just current anomaly but past faults, maintenance records. Combining real-time state with history yields complete judgment basis for sound decisions.

Access Speed : Frequent point lookups and retrievals during multi-round processing; each access adds to task latency. Data must be fresh, complete, and fast.

Conceptual formula: Real-time Context = Freshness × Completeness × Access Speed — any deficiency impacts action quality.

2.1 Fluss Lake-Stream Unity: Organizing Real-Time and Historical Data as Unified Storage

Fluss (based on Apache Fluss) provides fully managed cloud stream storage, connects Flink, EMR Spark, StarRocks, forming lake-stream unified solution with Alibaba Cloud Data Lake Service. Lake-stream unity organizes data lake and data stream as a whole, not two independent stores each maintaining full data. Two layers: Data Lake (historical layer) — long-cycle, minute-level+ latency data; Fluss (real-time layer) — short-cycle, millisecond-latency needs. Layers coordinate: history mainly in lake, real-time layer retains short window, auto-tiering continuously sinks to historical layer. Reduces redundant storage, lowers cost, reduces management overhead and inconsistency.

Key read capability: Union Read automatically merges real-time and historical layers, coordinates read boundaries, guarantees logical no-duplicate-no-missing. Externally a single logical table, same usage as one table, no manual cold/hot stitching.

2.2 DLF (Paimon) One-Click Real-Time Acceleration: Keep Existing Tables and Query Patterns

DLF Data Lake provides one-click "real-time acceleration" entry. User finds target DLF table, enables "lake-stream unity" switch; DLF associates and manages Fluss real-time layer. After enable, table and schema unchanged, existing SQL preserved. Upper engines via Union Read auto-merge real-time and historical data, gaining second-level freshness while retaining full history. DLF unified manages Catalog, schema, cold/hot boundary, tiering — no historical data migration, no dual-table maintenance, no manual UNION, no upstream SQL rewrite. Emphasizes format compatibility, permission sync, batch/stream sharing, rollback capability; fully managed service provides permission control, elastic scaling, multi-AZ disaster recovery.

2.3 Multi-Engine Union Read: Stream Compute, Interactive Analytics, Batch Processing Share Latest Data

Fluss stores only recent data (controllable real-time layer cost); full history in Paimon. DLF Catalog presents both as unified storage, single logical table, manages read boundaries.

Flink : Starts from historical data, continuously consumes latest changes for Agent context continuous update. Pattern: "full snapshot → subsequent Changelog" — existing background and new events enter same stream processing pipeline.

StarRocks : Interactive OLAP SQL combines history and current data for second-freshness dashboards and ad-hoc analysis.

Spark : Batch analytics includes latest data, not just last completed historical snapshot.

Different engines read complete history and latest changes from same table per their compute mode. For Agents, recent user behavior and historical preferences jointly accessible via unified SQL entry.

2.4 Context Point-Lookup Acceleration: Tiered Caching Balances Access Latency and Storage Cost

Agents demand low-latency point queries. Frequent key lookups waiting on slow underlying reads prolong multi-round tasks. Fluss accelerates via lake-stream point-lookup caching.

Hot partitions : Full data cached in Fluss hot tier (memory/SSD) for low-latency return.

Cold partitions : Full data in Paimon; Fluss retains partial metadata cache and data block cache (Chunk-Level Caching).

On request: cache hit → direct return from Fluss memory/SSD; miss → load from Paimon, backfill cache for subsequent accesses. Long-cycle history stays in low-cost data lake; Agent context service gains low access latency via hot data and block caching. Lake-stream unity enhances data lake's online context query capability.

2.5 Kafka Producer Compatibility: Shortening Path from Existing Data into Agentic Lake

Fluss has native API/protocol; many enterprises have mature Kafka producers and log collectors. Upcoming Kafka Producer compatibility lets existing Kafka apps reuse familiar Producer interfaces to write to Fluss without code changes. Web apps, Log Proxy, Logstash, Fluent Bit shown. Fluss parses/structures writes, converts JSON to schema-bearing Arrow real-time tables, supports real-time consumption and online query. Data auto-tiered periodically to DLF Paimon for historical analysis and long-term storage. Value: not just replace producer interface, but shorten path from message generation to queryable lake table, connecting familiar Kafka ingestion with real-time tables and historical lake tables. Note: "upcoming" capability status as of sharing time.

2.6 Taobao Flash Sale: Lake-Stream Unity Improves Real-Time Business Data Timeliness

Taobao Flash Sale dashboards, order forecasting, canary monitoring need rapid change sensing and historical baselines. Original data lake latency became bottleneck; upgraded to lake-stream unity.

New architecture: orders, visits processed by Flink into Fluss real-time wide tables, auto-tiered to Paimon. Union Read unifies latest state and history for dashboards, forecasting, analysis.

Gains: real-time decision data latency from minutes to seconds; point-lookup P99 latency from 800ms to <5ms; monitoring scenario IO cost reduced ~67%; overall storage scale 1.4 PB.

Alibaba Cloud Flink + Fluss: Streaming Agent & Lake-Stream Unity

Combining Flink and Fluss lets Agentic Lake not only aggregate and understand multimodal data but continuously update context synchronized with business scene. From data ingestion, perception, understanding to historical persistence, real-time reading, business action — a complete data pipeline for Agents. Value beyond more data for Agents: every judgment has traceable history, every action has current grounding, pushing intelligence from "understanding data" to "continuously understanding business", foundation for Agents to penetrate business processes and create real value.