How Setats Unifies Stream, Batch, and Incremental Processing for Real‑Time Data Lakes

During the DA Data+AI Conference held on April 25, 2025 in Shanghai, Tencent Cloud officially introduced Setats, a unified engine that simultaneously supports stream, batch, and incremental computation, dramatically lowering system costs and enabling end‑to‑end second‑level data visibility and real‑time changelog generation.

Traditional big‑data architectures, such as the Lambda model, struggle with massive data volumes and AI‑driven workloads. They require multiple storage systems and separate real‑time and offline processing pipelines, leading to high latency for batch jobs and complex state management, resource overhead, and data redundancy for streaming jobs.

Setats’ underlying technology features a self‑developed row‑column mixed storage with hot‑cold tiering, delivering high‑performance real‑time data merging. This design unifies storage for data lakes, real‑time logs, and intermediate stream‑processing state, providing second‑level data visibility, efficient primary‑key lookup, and complete changelog generation, thereby eliminating data duplication and allowing seamless switching among processing modes.

For demanding real‑time scenarios, Setats incorporates a unified state store and lake‑warehouse design that resolves local state capacity limits and back‑fill difficulties, boosting task startup efficiency by 50%. The engine is deeply compatible with the Flink ecosystem, supports standardized SQL operations, and enables smooth migration without code refactoring, lowering technical barriers.

In a real‑world deployment, a leading automotive OEM rebuilt its connected‑vehicle data analysis pipeline with Setats. Handling several hundred terabytes of sensor data daily, the previous architecture suffered from high alert latency and 40% storage redundancy. After adopting Setats, vehicle‑status monitoring response time improved by 30%, storage and compute costs dropped by 33%, and overall data‑governance efficiency increased, supporting rapid business scaling.

At the conference, Tencent Cloud also announced an upgrade to its Data+AI development platform, WeData, with Setats as a core component. Combined with multimodal metadata catalogs and serverless resource scheduling, the platform offers a full‑stack solution covering data governance, AI model development, and LLM application building, breaking both “data walls” and “compute walls” and multiplying data‑value extraction efficiency.

WeData has already served retail, finance, manufacturing, and other sectors, helping more than 200 enterprises upgrade their data architectures and achieve efficient governance. Tencent Cloud plans to continue iterating Setats, aiming to provide an increasingly intelligent and integrated big‑data ecosystem that paves the way for enterprises to thrive in the AI era.