Professor Bian Haoqiong presents Retina, a new table format layer bringing row-level cross-table MVCC to lakehouses, achieving 10ms data freshness with only 5% query overhead by rethinking transaction replay, visibility, and compaction.
This article explains how EMR Serverless StarRocks leverages Paimon Global Index, including BTree for high-selectivity scalar filters and vector indexes for approximate nearest neighbor search, detailing the two-phase query architecture, index creation, SQL usage, and combined scalar-vector query execution.
Alibaba Cloud's EMR Serverless StarRocks now fully supports Apache Paimon 2.0, enabling unified multimodal analytics and retrieval by integrating vector search, full-text search, scalar filtering, and BLOB handling within a single lakehouse architecture, with best practices for data management and query optimization.
The article explains how Alibaba Cloud EMR Serverless StarRocks converges full-text, scalar, and vector search into a single engine using RRF for score fusion, In-Filter for predicate pushdown into vector search, and Partial Update for column-level annotation writes, reducing data movement and operational complexity.
This article explains how StarRocks integrates with Apache Fluss to enable SQL queries across real-time streaming data and historical lakehouse data, detailing the architecture, catalog setup, union read mechanics, performance optimizations, and future roadmap for unified analytics.
The article provides a decision framework for enterprises evaluating Lakehouse adoption, arguing architecture should evolve with business complexity rather than follow trends. It defines six evaluation dimensions, contrasts Lakehouse with traditional OLAP, outlines phased evolution from basic analytics to streaming Lakehouse and AI data infrastructure, and warns against five common over-engineering pitfalls.
This article clarifies the distinctions between data lakes, data warehouses, and lakehouse architectures, explaining their core philosophies, ideal use cases, common pitfalls like data swamps, and why stable data pipelines matter more than architectural buzzwords.
StarRocks integrates Paimon C++ (paimon-cpp) to enable native C++ execution paths for querying Apache Paimon tables, eliminating JNI overhead, supporting primary key merges and delete handling, and achieving 1.4-3.4x speedups in benchmarks while reusing StarRocks' caching, memory management, and query profiling.
StarRocks adopts Apache Paimon's C++ implementation (paimon-cpp) to replace JNI calls for primary-key merges and delete-vector handling, cutting cross-language overhead and reusing StarRocks' cache, memory management, and Query Profile; benchmarks show 1.4–3.4× speedups on primary-key counts and wide-table aggregations.
Alibaba Mama evolved its advertising real-time data warehouse from a message queue-based architecture to a Flink + Paimon streaming lakehouse, solving structural defects like lack of deduplication, aggregation layer support, and schema enforcement by leveraging Paimon's primary key upsert and changelog capabilities, while sharing production optimization practices and comparing storage choices for advertising vs. risk control scenarios.
This article outlines a phased roadmap for building lakehouse platforms — from 0 to 1 validation, 1 to N isolation, to storage-compute separation and cost governance — emphasizing that architecture must match the current business contradiction, not chase industry trends.
This article explains how to implement RAG within a lakehouse architecture by treating the vector database as a derived retrieval index rather than a separate system, covering embedding pipelines, Milvus index trade-offs, hybrid search, real-time updates via Flink, consistency governance, and troubleshooting common production issues.
This article presents a complete methodology for lakehouse performance stress testing and capacity planning, covering TPC-DS benchmark adaptation, end-to-end bottleneck identification across CPU, memory, disk, and network, a quantitative capacity derivation framework, and vertical versus horizontal scaling decision criteria for components like Kafka, Flink, Paimon, and StarRocks.
This article deep-dives into Apache Paimon's kernel source code, explaining the LSM write path from memory buffer to Sorted Runs, compaction scheduling trade-offs, four changelog production modes, and two-phase commit with Flink checkpoints, providing troubleshooting guidance for compaction backlog, changelog latency, and snapshot bloat.
StarRocks now fully supports Paimon 2.0, enabling multimodal lakehouse workloads where columnar scans, scalar/vector/full-text indexes, and BLOB fetches coexist in one SQL execution framework, backed by Row Tracking and Data Evolution for incremental feature backfill.
StarRocks now fully supports Paimon 2.0, integrating columnar analytics with vector, full-text, and scalar indexes into a single SQL execution framework for multimodal lakehouse workloads, enabling unified data management, retrieval, and relational analysis on object storage without data movement.
This article explains how StarRocks queries Paimon Variant data, covering the differences between regular Parquet JSON and Variant, the three-phase read architecture, Shredding optimization for hot paths, practical SQL examples using get_variant_* functions, and a decision framework for choosing between JSON, Plain Variant, Shredded Variant, and formal typed columns.
This article analyzes why identical StarRocks queries on Paimon external tables run fast in early morning but slow dramatically during daytime peaks, identifying five layered root causes from file fragmentation to storage pressure, and provides a systematic troubleshooting methodology plus a four-layer optimization framework covering source governance, compute caching, storage scaling, and business scheduling.
Apache Paimon combines a lakehouse table format with an LSM-tree update engine to handle continuous updates, historical snapshots, and multi-engine access in real-time data warehouses, reducing the need for separate Kafka, Hive, and OLAP systems.
This article details Apache Fluss's dual-table model where LogTable handles high-throughput append-only streaming with columnar storage while PrimaryKeyTable supports real-time updates, partial merges, and CDC changelogs via integrated RocksDB state storage, sharing unified tiering and lakehouse sinking.
At Apache COC Asia 2026, Didi’s engineers presented three deep‑dive sessions covering the evolution of massive‑scale object storage with Ozone, cost‑effective lakehouse migration using Iceberg and real‑time ingestion, and AI‑friendly enhancements to the open‑source Seata project, detailing architectural choices, performance gains, and open‑source contributions.
Enterprise data platforms face common questions—why add a data warehouse when MySQL or Oracle exist, why introduce a data lake after a warehouse, and what problems NoSQL, ClickHouse, or Doris solve—this article breaks down six storage technologies, their core use‑cases, and how to choose among them.
The article analyzes how StarRocks, Fluss, and Paimon can replace the traditional Lambda architecture with a unified lakehouse solution that delivers second‑level data freshness, consolidates storage and compute, and provides native read/write paths for high‑performance real‑time analytics and large‑scale ETL.
In July 2024 Apache Fluss received unanimous approval from the Apache Incubator IPMC and ASF board, graduating to a Top‑Level Project; the article details its community growth, core Lakestream architecture, real‑time streaming storage capabilities, adoption by major enterprises, and how it enables AI agents with fresh, unified data for real‑time decisions.
At AICon 2026, Ma Jin explained how Lance’s layered lakehouse format—spanning file, table, index and catalog layers—addresses AI workloads by supporting wide columns, cheap schema evolution, mixed storage and low‑latency random reads, enabling unified handling of training data, RAG retrieval and Agent memory.
This article (part 1) presents twelve high‑level interview questions from top tech firms covering the shift from AI‑Ready to Data‑Agent‑Ready data warehouses, AI‑centric metadata and lineage, lakehouse formats like Paimon, and key Iceberg features that empower AI‑driven analytics.
The article analyzes Databricks' shift from a unified Lakehouse architecture to building an "Agent operating system" that supplies context, state, tools, identity and governance for enterprise agents, outlines its four‑layer data stack, compares it with Palantir, Snowflake and Microsoft Fabric, and discusses the technical and strategic challenges ahead.
The article analyzes Databricks' shift from a pure Lakehouse data platform to an emerging Agent operating system, detailing a four‑layer architecture for facts, semantics, governance, and agent execution, and comparing its approach with Palantir, Snowflake and Microsoft Fabric.
The article explains that although modern data platforms are rebranded as lakehouses or AI data foundations, the core data‑warehouse tasks—ingestion, cleaning, modeling, metric alignment, quality, lineage and governance—remain unchanged, and practitioners often hide the term to avoid being labeled as using outdated technology.
Alibaba Cloud's EMR Serverless StarRocks Stella 2.2 unifies vector search, full-text search, and AI functions across both internal and lake tables (Paimon, Iceberg, Lance), enabling a single SQL interface for multimodal retrieval, AI-powered analytics, and RAG workloads without data movement.
The article provides a detailed 2026 H1 review of global database trends, highlighting the shift to AI‑native and lake‑house architectures, summarizing major version releases across RDBMS, NoSQL, big‑data ecosystems and domestic solutions, and analyzing their technical features, compliance, performance and market implications.
Milvus 3.0 introduces an External Collection that treats Parquet and other lakehouse files as a read‑only query view, detailing how external_source and external_spec are validated, how stable file lists are built, how the two‑stage refresh recomputes segment mappings, and how virtual primary keys and milvus‑table snapshots enable seamless querying without data duplication.
The article details how Taotian Group’s senior data engineer Zhu Ao designed a lake‑stream integrated architecture using Fluss for second‑level real‑time storage, Paimon for minute‑level lakehouse persistence, and StarRocks as a unified OLAP query layer, achieving over 50% faster development and more than 80% cost reduction.
Xiaohongshu transformed its data platform from a simple ClickHouse‑based stack to a Lambda‑enhanced architecture and finally to a Lakehouse with incremental compute, cutting architecture complexity, resource and development costs by two‑thirds while delivering second‑level analytics on petabyte‑scale data.
The article details Xiaohongshu’s journey from a simple ClickHouse‑based ad‑hoc analytics stack to a Lambda‑style architecture and finally to a lakehouse with generic incremental compute, cutting architecture complexity, resource cost and development effort each to roughly one‑third while achieving sub‑10‑second query latency on petabyte‑scale data.
Xiaohongshu transformed its data platform from a simple ClickHouse‑based ad‑hoc analysis system to a Lambda‑style architecture and finally to a lakehouse with incremental compute, cutting architecture complexity, resource and development costs by one‑third while delivering second‑level queries over petabyte‑scale data.
The article analyzes how OceanBase’s new AI‑native database tackles the data‑layer challenges of autonomous agents by introducing logical tables, multi‑modal storage, Fork DB sandboxes and lake‑warehouse integration, arguing that the product’s value lies beyond simple vector retrieval.
Lakebase, the core engine of OceanBase AI Database, unifies storage, management, computation and search for structured, unstructured and vector data, offering multimodal data capabilities, flexible deployment modes, and AI‑oriented use cases such as autonomous driving and financial document analytics.
Xiaohongshu, with over 3.5 billion monthly users and daily logs in the trillions, migrated 500 PB of data to Alibaba Cloud and iterated its data platform through four architecture generations—ClickHouse‑based ad‑hoc, Lambda, Lakehouse, and a unified incremental compute model—cutting resource, development, and storage costs to one‑third while delivering sub‑10‑second query latency at petabyte scale.
The article analyzes the May 2026 releases of Apache Iceberg V3 and Spark 4.0, explaining how Deletion Vectors, the native VARIANT type, and Spark Connect together transform semi‑structured data handling and real‑time upserts in lakehouse architectures.
Xiaohongshu, with over 350 million monthly users and daily logs in the billions, migrated its data platform from AWS to Alibaba Cloud and iterated four times—from a ClickHouse‑based ad‑hoc layer to a Lambda architecture and finally a Lakehouse with incremental compute—cutting architecture complexity, resource cost and development effort each to about one‑third while delivering second‑level analytics on trillion‑scale data.
The article details Xiaohongshu's step‑by‑step migration from a simple ClickHouse‑based analytics stack to a Lambda‑style 2.0 architecture and finally to a Lakehouse‑based 3.0 design, highlighting concrete performance numbers, cost reductions, and the definition of a generic incremental‑compute model (SPOT) that underpins the evolution.
This article presents a lake‑stream integrated solution that combines Apache Fluss, Apache Paimon, and StarRocks to achieve second‑level data freshness, tenfold storage cost reduction, and a single‑query access pattern for both real‑time and historical data, detailing its architecture, advantages, query modes, and future roadmap.
Xiaohongshu transformed its data platform from a simple ClickHouse‑based ad‑hoc analysis to a Lambda‑style architecture and finally to a lakehouse with generic incremental compute, cutting architecture complexity, resource and development costs by one‑third while delivering second‑level queries over trillions of rows.
This article analyzes Paimon Bloom Index internals, demonstrates a real-world 25x query acceleration (38s to 1.5s) on a 400GB partitioned table, explains false positive rates, optimal use cases, and provides production-ready configuration patterns for StarRocks+Paimon lakehouse architectures.
This article enumerates ten uncommon, compaction‑related problems encountered in large‑scale Paimon deployments, explains their root causes—such as RPC timeouts, snapshot expiration, file corruption, and write conflicts—and provides concrete configuration tweaks and operational steps to resolve each issue.
The article details Xiaohongshu's evolution from a simple ClickHouse‑based analytics layer to a Lambda‑enabled 2.0 stack and finally a Lakehouse‑based 3.0 architecture, showing how each iteration reduced infrastructure complexity, resource consumption and development effort by roughly one‑third while supporting trillions of daily events and AI‑driven use cases.
The article examines how the rise of AI agents is reshaping enterprise data infrastructure, tracing software evolution from rule‑based systems to lakehouses and arguing that real‑time OLAP engines with sub‑second latency, hybrid search, and semantic schemas will become the core of the new Agent‑centric stack.
Apache Doris 4.1 introduces a systematic evolution for AI and search workloads, adding low‑cost massive vector storage, unified structured, full‑text and vector search, 100 MB JSON document support, Segment V3 metadata decoupling, sparse column optimizations, lakehouse lifecycle management, and a suite of performance‑boosting features such as aggregate push‑down, condition cache, and spill‑to‑disk, all backed by detailed benchmark results.
The article examines the challenges of multimodal data in modern lakehouses and presents a three‑tool stack—Gravitino, Daft, and Lance—that provides unified metadata, distributed multimodal compute, and high‑performance storage, while detailing security governance, integration paths, and future directions.
Xiaohongshu transformed its data platform from a simple ClickHouse‑based ad‑hoc analysis to a Lambda‑style architecture and finally to a lakehouse built on Iceberg, StarRocks, Flink and Spark, cutting architecture complexity, resource and development costs by two‑thirds while supporting trillions of daily events with sub‑second query latency.
The article details Xiaohongshu's four‑stage data‑platform evolution—from a simple ClickHouse ad‑hoc setup to a Lambda‑based 2.0 design and finally a lakehouse‑driven 3.0 architecture—highlighting the adoption of general incremental compute, cost‑reduction to one‑third, performance gains of up to ten‑fold, and the SPOT standards that guide the new system.
Xiaohongshu transformed its data platform from a simple ClickHouse‑based analytics stack to a unified lakehouse with generic incremental compute, cutting architecture complexity, resource cost, and development effort by roughly one‑third while supporting petabyte‑scale, sub‑second queries across its 350 million‑user app.
This article details Xiaohongshu's journey from a ClickHouse‑based batch analytics stack to a unified lakehouse architecture powered by generic incremental computing, showing how the company reduced architecture complexity, resource consumption and development effort each to roughly one‑third while supporting trillions of daily events with sub‑10‑second query latency.
Traditional databases were built for low‑frequency, human‑driven queries, but AI agents generate dozens of concurrent, sub‑second queries that expose architectural limits, and StarRocks addresses these challenges with self‑healing optimization, real‑time data pipelines, extreme concurrency handling, and seamless lakehouse access.
This article details Xiaohongshu's data platform evolution from a simple ClickHouse‑based ad‑hoc system to a Lambda‑style architecture and finally a lakehouse solution, highlighting how the adoption of a new incremental computing model reduced architectural complexity, resource consumption and development effort each to roughly one‑third while delivering sub‑second query performance on petabyte‑scale data.
This article analyzes Xiaohongshu's data platform evolution—from a simple ClickHouse‑based analytics layer to a Lambda architecture and finally a lakehouse design—highlighting how adopting a new incremental computing model reduced architecture complexity, resource consumption, and development effort each to roughly one‑third while delivering sub‑second query performance on petabyte‑scale data.
This article analyzes the challenges of scaling ClickHouse within Baidu’s MEG data platform and details a lake‑house solution that decouples storage and compute, integrates a meta‑service for transparent data access, optimizes query performance through caching, data roll‑up and layout tuning, and introduces a unified query gateway that gracefully falls back to Spark for complex workloads.
The article outlines a full‑stack data architecture for 2026, covering edge‑preprocessed collection, unified observability with OpenTelemetry, CDC‑based real‑time sync, a Kafka‑Flink‑Iceberg processing stack, Data Mesh governance, and AI‑Agent powered insight, while offering practical adoption paths and pitfalls to avoid.
Fanatics unified its fragmented data stack by building a StarRocks‑powered Lakehouse on Apache Iceberg, replacing Redshift, Snowflake, Athena, and Druid, which cut costs by up to 95%, delivered sub‑second dashboard queries on petabyte‑scale data, and enabled real‑time and historical analytics on a single platform.
This article analyzes how Spark and Flink should divide responsibilities in lakehouse architectures by evaluating three core dimensions—data latency, computation accuracy, and resource cost—arguing that mature batch-stream integration balances business needs against technical overhead rather than pursuing universal real-time processing.
This article details the deep integration of StarRocks with Apache Paimon, describing the unified architecture, version evolution, performance enhancements, time‑travel queries, native readers/writers, distributed planning, and future roadmap for achieving lakehouse‑native analytics at scale.
This article details how Alibaba Cloud EMR Serverless Spark combined with the Paimon lakehouse framework enables Taobao Flash Sale’s retail data team to achieve low‑latency, high‑throughput real‑time analytics, batch processing, and feature generation, outlining architecture evolution, performance gains, and practical Spark tuning techniques.
This article details the evolution of a data warehouse at RenliJia from a MaxCompute‑centric setup to a modern lakehouse using StarRocks, Paimon, Flink, and Fluss, describing design goals, technical evaluations, implementation steps for offline, OLAP, and real‑time workloads, and the challenges and future plans that emerged.
The article outlines the evolution from traditional OLAP to an AI‑first Lakehouse, detailing unified multimodal storage, CPU/GPU heterogeneous scheduling, native vector search, in‑database AI inference, agent‑centric execution, and self‑evolving platform capabilities that together reshape modern data analytics.
The article explains that real‑time data processing requires not only speed but also accuracy, stability, and long‑term maintainability, compares Lambda, Kappa, and unified batch‑stream architectures, discusses lakehouse storage, and provides practical guidance on selecting the most suitable approach for different project needs.
This article details Gaode Maps' end‑to‑end lakehouse solution for massive, high‑frequency trajectory data, covering the challenges of real‑time visibility, query performance, and storage cost, and explaining how a hot‑warm‑cold tiering architecture built on Apache Flink, Paimon, StarRocks, Redis and Lindorm delivers millisecond‑level queries while cutting storage expenses.
This article provides a practical framework for SMEs to select and implement data platform architectures, comparing traditional warehouses, lakehouse, and multimodal data lakes across cost, ROI, and operational efficiency, with scenario-based recommendations and a lightweight lakehouse implementation guide using open-source stack Flink, Paimon, StarRocks, and MinIO.
Veteran data‑warehouse expert explains how the Paimon‑StarRocks stack solves the write‑read split, cuts storage costs, and delivers real‑time analytics, comparing it with Hudi, Iceberg, ClickHouse and Trino, and shows why leading Chinese tech firms are adopting this lakehouse architecture.
This article details Gaode Maps' end‑to‑end lakehouse solution for handling high‑frequency, high‑volume trajectory data, covering the challenges of real‑time visibility, multi‑scenario queries, storage cost, and data silos, and describing the layered storage architecture, performance validation, and future expansion plans.
This article examines Apache Paimon’s innovative lakehouse architecture, detailing its LSM‑Tree storage, flexible merge engine, and multi‑engine integration, and showcases two real‑world deployments—an operator’s real‑time fraud‑prevention system and a manufacturing firm’s unified data platform—highlighting performance gains and cost reductions.
StarRocks and Apache Paimon have been progressively integrated across multiple releases, enabling a unified lakehouse architecture that supports multi-source federated analysis, time-travel queries, native readers/writers, distributed planning, and advanced profiling, while delivering performance gains that bring Paimon query speed on par with native StarRocks tables.
This article explains the challenges of frequent column changes in AI feature engineering, introduces Paimon’s column‑separation storage with a global continuous Row ID, details its Blob data type for efficient multi‑modal handling, and outlines production results and future roadmap for building an AI‑native data lakehouse.
This article analyzes why StarRocks multi-table joins on Paimon/Iceberg tables remain significantly slower than internal tables even after cache warming, detailing six root causes across storage format, data layout, caching, execution engine, metadata, and version management, plus four optimization strategies to narrow the gap.
Facing Postgres overload and costly Snowflake queries, Fresha rebuilt its analytics platform by introducing StarRocks as a unified SQL entry point, combining federated lakehouse queries with high‑performance internal tables, which reduced homepage query latency to around 200 ms and achieved minute‑level data freshness across real‑time, historical, and search workloads.
The article recaps the Apache Hudi Asia Meetup hosted by JD, covering community updates, JD's data‑lake challenges, the upcoming Hudi 1.1 release, JD's architectural redesign, Kuaishou's real‑time lake adoption, and Huawei Cloud's deep optimizations, all aimed at building an AI‑native, real‑time lakehouse.
This article collection showcases seven advanced data engineering solutions—from Tencent Cloud's Iceberg batch‑stream integration and Apache Gravitino metadata lineage to Xiaohongshu's Lakehouse evolution and multimodal AI data lake implementations—highlighting architectural innovations, performance optimizations, and real‑world deployment insights for modern big‑data platforms.
The JD‑hosted Apache Hudi Meetup showcased the 1.1 release’s pluggable table format, Flink performance gains, LSM‑Tree MoR redesign, and AI‑native features such as vector indexing, illustrating how the open‑source lakehouse is evolving to meet BI and multimodal AI workloads.
Apache Paimon 1.3 introduces incremental clustering for 150x faster append-table queries, a pure Python SDK 95x faster than its JVM-based predecessor, data evolution cutting wide-table storage by 99.4%, a virtual file system for unified cross-engine access, and upcoming vector storage for AI workloads.
The 2025 Apache Paimon release brings major performance boosts, AI‑centric multimodal storage, deeper streaming‑batch integration, and broader engine compatibility, detailing query and write optimizations, memory management tweaks, and a unified lake format for structured and unstructured data.
The article compares data warehouses, data lakes, and lakehouse architectures, illustrating each with real‑world projects, highlighting their core concepts, strengths, limitations, key technologies, and a step‑by‑step migration path to help teams select the most suitable solution for their business needs.
This article explains why data migration is the essential first step for cloud modernization, outlines the technical challenges of moving terabytes to petabytes, compares physical and logical migration methods, and presents practical solutions and real‑world case studies across Hive, cloud warehouses, lake‑house formats and analytic databases.
The article benchmarks StarRocks internal tables against Paimon and Iceberg lake tables using TPCH 100G, shows internal tables excel at real-time analytics while lake tables optimize cold storage cost and multi-engine sharing, and details optimization strategies (metadata, data cache, predicate pushdown) achieving 5-10x speedup, illustrated by a retail case study cutting query latency from seconds to milliseconds and storage cost by 65%.
The article explains the concepts of data lakes and lakehouses, compares them with traditional data warehouses, outlines the reliability, performance, and security challenges of data lakes, and then details a practical lakehouse implementation using Apache Iceberg, Flink SQL, CDC pipelines, and supporting tools such as Hive Metastore and Trino.
This article reviews how Paimon and Doris combine to solve unified storage, data visibility, and performance challenges in modern lakehouse architectures, detailing their complementary features, integration capabilities, and real‑world use cases from leading companies.
The Apache Doris 3.1 release strengthens lake‑house capabilities with major upgrades to the VARIANT data type, vertical compaction, inverted index storage, new tokenizers, enhanced materialized view support for Iceberg/Paimon/Hudi, and numerous query‑performance optimizations such as faster partition pruning and dynamic partition clipping, offering smoother handling of thousands of columns and large‑scale semi‑structured data.
This article details how Xiaohongshu's massive user‑generated data platform evolved from a simple ClickHouse‑based architecture to a multi‑stage lakehouse design, adopting a new incremental computing model that reduced architecture complexity, resource and development costs by one‑third while boosting query performance for petabyte‑scale data.
This article traces the three‑generation evolution of data architecture—from the structured‑data era of data warehouses, through the flexible, multi‑format data lake, to the unified lakehouse model—explaining the drivers, benefits, challenges, and future trends shaping modern data platforms.
This article details Ctrip Business Travel’s implementation of a near‑real‑time data warehouse using Flink CDC and the Paimon lakehouse engine, covering order wide‑table construction, ticket refund alerts, ad attribution, batch‑stream integration, and practical lessons on Partial Update, Aggregation, and Tag‑based incremental processing.
In this detailed case study, Alibaba’s Ele.me team explains how they evolved from siloed, chimney‑style real‑time warehouses to a unified Flink‑Paimon lakehouse, highlighting the three development stages, technology evaluations, the Alake platform’s one‑stop capabilities, production results, and future directions such as Fluss and AI integration.
Facing minute‑level decision demands and billions of marketing events during Taobao's Flash Sale, the Ele.me data team built a real‑time lakehouse with StarRocks and Paimon, leveraging asynchronous materialized views, RoaringBitmap de‑duplication, and resource isolation to achieve sub‑second query latency, lower storage costs, and stable high‑concurrency.
Faced with billions of marketing events and minute‑level decision requirements during Taobao's flash‑sale campaign, the e‑commerce data team built a real‑time lakehouse using StarRocks and Paimon, leveraged asynchronous materialized views and RoaringBitmap deduplication, and achieved sub‑second query latency, massive cost savings, and stable high‑concurrency performance.
This article details Ctrip Business Travel’s implementation of a near‑real‑time data warehouse and lakehouse using Flink CDC and Apache Paimon, covering order wide‑table construction, automated ticket reminders, ad attribution, batch‑stream integration, and lessons on Partial Update, Aggregation, and Tag‑based incremental processing.
Facing daily data growth from millions to 150 billion records, Joydata‑U transformed its analytics platform through three architectural stages—Hadoop, Hadoop + Trino, and finally StarRocks—introducing resource isolation, Flat JSON acceleration, and Bitmap indexing to cut query latency by up to seven times and achieve sub‑2‑minute data freshness across BI, ad‑tech, game analytics, and CRM workloads.
This article explains the evolution from traditional data warehouses and data lakes to the unified Lakehouse architecture, detailing its design, benefits, challenges, and research directions for delivering high‑performance SQL and advanced analytics on open‑format storage.