RustFS evolved from a doubted README-only project into a production-ready, S3-compatible object storage written in Rust, now with 2.7M+ installations, seamless MinIO migration, and free Iceberg-powered S3 Tables that MinIO charges $80k/year for.
This article compares Flink CDC with traditional Canal-based architectures, explains Flink CDC's core principles including lock-free snapshots and exactly-once semantics, provides DataStream and SQL code examples for MySQL integration, and covers five high-frequency interview questions on DDL handling, DELETE capture, and large-table optimization.
This article dismantles twelve common data governance fallacies — from demanding instant metadata completion to treating governance as a cost center — arguing that effective governance requires confronting upstream data pollution, aligning incentives across departments, and prioritizing critical data paths over blanket coverage.
This article details how to build a production-ready real-time data warehouse using Flink, covering CDC ingestion, event-time processing, dimension joins, upsert aggregation, state TTL, checkpoint tuning, exactly-once verification, and a pre-launch checklist for reliability.
The 17th China Database Technology Conference (DTCC 2026) ran from August 20‑22 in Beijing, gathering top experts to discuss database kernel innovations, cloud‑native and distributed practices, AI‑driven data, vector databases, real‑time warehouses, and the emerging Agent era, while showcasing cutting‑edge solutions from Dameng, Tencent Cloud, Alibaba Cloud, OceanBase and GoldenDB.
The article outlines the strategic need for a unified data lake in a digital‑focused enterprise, details functional and non‑functional requirements such as linear scalability, real‑time and batch processing, multi‑tenant support, security and governance, and presents a comprehensive architecture design that integrates storage, compute, and management components.
The article analyzes how the rise of AI agents transforms data platforms from simple storage and query engines into AI‑native systems that provide trustworthy, real‑time context for autonomous decision‑making, outlining the three‑layer evolution of storage, compute, and application and the architectural upgrades required for modern data lakes.
Enterprise data platforms face common questions—why add a data warehouse when MySQL or Oracle exist, why introduce a data lake after a warehouse, and what problems NoSQL, ClickHouse, or Doris solve—this article breaks down six storage technologies, their core use‑cases, and how to choose among them.
The article explains how lance‑spark enables Lance's Blob V2 storage by marking a column with blob encoding and setting file_format_version ≥ 2.2, describes the metadata‑driven descriptor schema, the write path that still accepts Spark BINARY, and the size‑based placement of binary data into inline, packed, dedicated or external blob files.
As enterprises add more systems, data volumes explode while accessing it becomes harder; this article defines data architecture, explains how databases, data warehouses, data lakes, and data middle platforms each solve distinct layers, and outlines the key questions and challenges for building a unified, reusable data ecosystem.
The article reviews Tencent Cloud's AI DLC launch, detailing how the serverless Spark + Ray platform unifies data, compute, and agent workflows, introduces four architectural upgrades, showcases core engines (TCRay, Xpark, Meson, Open Engine), and presents benchmark results and real‑world practices from Bosch and WorkBuddy that demonstrate significant performance and productivity gains.
This tutorial explains how Apache Iceberg adds snapshot metadata to files on HDFS or object storage, enabling atomic writes, time‑travel queries, and schema evolution, and walks through a Docker‑based setup with a REST catalog, Spark 3.5.3, and hands‑on examples including table creation, data insertion, version queries, and troubleshooting tips.
This article explains why online transaction databases (OLTP) should not be used for multi‑year analytics, outlines the four core characteristics of a data warehouse, details the ETL process, compares star and snowflake schemas, contrasts data warehouses with data lakes, and guides you through selecting Lambda or Kappa architectures using a concrete retail‑store scenario.
The article explains how data warehouses, data marts, data lakes, and data middle platforms each address distinct problems—unified analytics, departmental needs, raw data storage, and governed reusable capabilities—while outlining their relationships, typical use cases, implementation considerations, and guidance on choosing the right architecture for a given business stage.
With agents moving from Q&A to execution, traditional architectures expose three critical flaws—data stored in lakes, models in the cloud, and split scheduling—forcing petabyte‑scale data movement; Tencent Cloud’s big data AI DLC resolves this by running Spark and Ray side‑by‑side on the same lake, enabling closed‑loop processing and automatic trajectory capture.
The article analyzes how traditional data‑lake‑model‑cloud architectures expose three critical flaws for agentic AI—massive data movement, fragmented logging, and split compute—then details Tencent Cloud's Big Data AI DLC solution that unifies Spark and Ray on a single lake to enable in‑place processing, closed‑loop training, and cost‑effective iteration.
The article explains how Tencent CodeBuddy’s AI DLC platform unifies Spark batch processing and Ray training to eliminate data movement, turning agent trajectories into reusable training fuel, which reduces monthly‑level training cycles to weekly, enables in‑place computation on billions of features, and cuts operational costs by 60%.
The article compares five data‑architecture concepts—data warehouse, big data platform, data lake, data middle platform, and lake‑warehouse integration—explaining the specific problems each solves, their core characteristics, advantages, risks, and guidance on when to adopt each solution.
The article compares four emerging data‑lake‑for‑PostgreSQL solutions—Databricks LTAP, EDB Fusion Analytics, Snowflake pg_lake, and pgEdge's ColdFront—highlighting ColdFront's unique transparent Iceberg layer, writable cold data, DuckDB integration, and the strategic trade‑offs developers must weigh when choosing a modern OLTP/OLAP/AI architecture.
This article presents an end‑to‑end design of a production‑grade observability platform that ingests millions of real‑time logs, metrics, and events, detailing functional and non‑functional requirements, capacity planning, component choices such as Kafka, Flink, Elasticsearch, object‑storage data lakes, and the trade‑offs involved.
Alibaba Cloud Message Queue for Kafka introduces a native message‑to‑lake capability that integrates Apache Iceberg with OSS Table Bucket, eliminating Spark/Flink/Kafka Connect, providing exactly‑once semantics, automatic schema management, dual write modes, smart partitioning, and up to ten‑fold performance gains across diverse real‑time analytics scenarios.
The article analyzes how the AI era shifts data from a passive by‑product to a core driver of large‑model performance, traces the evolution of data products from the DBA era through big‑data to AI‑native solutions, and details Volcano Engine’s four‑layer AI data platform that closes the data‑to‑model‑to‑Agent loop.
In the AI‑driven era, enterprises need a data foundation that supports both real‑time consumption and long‑term historical analysis, and the emerging "zero‑ETL" trend moves generic ingestion capabilities from external Flink/Spark jobs into a streamlined Kafka‑to‑Iceberg pipeline, reducing complexity while preserving low latency, consistency, schema evolution, CDC semantics and open‑ecosystem compatibility.
In the AI era, enterprises need a data foundation that supports both low‑latency streaming and long‑term analytics, and the combination of Kafka, Iceberg and object storage is emerging as a preferred solution; by moving ingestion capabilities closer to the message layer and eliminating external ETL jobs, a "zero‑ETL" approach reduces architectural complexity, improves consistency, and streamlines schema evolution and small‑file management.
This article details Fresha's real‑world integration of StarRocks with Apache Iceberg, covering metadata planning, distributed execution, adaptive metadata retrieval, hot‑cold data layering, missing statistics handling, catalog configuration, and performance optimizations that together demonstrate how federated queries can be efficiently executed over data‑lake tables.
This official technical reference details Paimon's file-level index architecture, covering BloomFilter, Bitmap V2, and Range Bitmap indexes with binary storage formats, configuration parameters, hash rules, supported data types, and SQL optimization examples for equality, range, AND/OR, and TOPN queries.
This article details Hisense's journey to create an AI‑ready multimodal data platform, covering the challenges of integrating diverse business systems, the shift from a Hadoop‑based architecture to a cloud‑native data lake, the JuData governance and development platform, and six practical scenarios that demonstrate unified ingestion, metadata management, rule‑based quality control, intelligent asset retrieval, and future AI‑driven DataOps capabilities.
Facing a projected 85% of enterprises deploying internal agents within two years, Alibaba Cloud proposes an Agent-Ready big‑data AI infrastructure—comprising a unified data lake, real‑time processing, high‑dimensional vector retrieval, elastic model serving, and comprehensive security governance—that has already cut data‑development cycles from hours to 5‑10 minutes in internal model‑training and Taobao flash‑sale scenarios.
The article explains how Kuaishou partnered with Apache Hudi to overhaul its ODS‑based data lake, addressing latency, storage cost, and complexity for AI and BI workloads, detailing the evolution from mysql‑to‑hive to mysql‑to‑hudi 1.0 and 2.0, the resulting performance gains, cost savings, and future roadmap.
The article analyzes why 74% of AI projects fail due to feedback gaps and data silos, explains how the open‑source Lance format addresses these issues with unified multimodal storage, outlines a layered Lance‑on‑Ray architecture, and details three real‑world practices—implicit feedback loops, GPU‑accelerated self‑evolution, and semantic knowledge‑graph evolution—to boost R&D efficiency.
Enterprises struggling with AI often hit a data wall; this article breaks down a practical five‑step data governance framework—inventory, classification, lineage, aggregation, and control—that transforms raw data into a reliable asset for AI and business initiatives.
The article presents Volcano Engine’s AI‑focused data lake built on the Lance format, detailing why traditional lakes fall short for multimodal data, the engineering enhancements such as Binary Copy Compaction, Lance Insight, distributed vector indexing, JSON‑based tagging, Row‑ID shuffle optimization, and real‑world case studies that demonstrate significant performance and cost gains.
The article explains how Data as a Service (DaaS) builds on data lakes and SaaS models to centralize governance, cut duplication and infrastructure costs, accelerate real‑time analytics, support cloud‑native deployments, and enable mobile/web applications through unified APIs.
This article systematically explains the definitions, architectures, advantages, disadvantages, and evolution of data warehouses, big‑data platforms, data lakes, data middle platforms, and lake‑warehouse integration, helping practitioners choose the right terminology and technology for their projects.
The article explains how fragmented ERP, MES, and CRM data create silos, outlines the five-layer data architecture lifecycle, identifies three common implementation challenges, and offers practical criteria for selecting the right storage and processing solutions to turn data into reliable business assets.
OPPO’s data‑lake team, led by David, detailed their transition from Hive‑Spark to a unified multi‑modal lake, leveraging Gravitino for cross‑engine metadata management and the open‑source Curvine cache to eliminate data silos, boost I/O performance, and support massive image, recommendation, and AI‑Agent workloads.
The article outlines Apache Doris's 2026 roadmap, detailing how the database will shift from pure analytics to a unified AI‑enabled platform with enhanced semi‑structured data support, vector and hybrid search, agent‑focused capabilities, and expanded storage and lakehouse integrations to meet emerging AI workloads.
The article explains how traditional Parquet‑based lakehouse architectures, optimized for large‑scale scans, struggle with AI workloads that need ultra‑low‑latency random access, and how Lance redesigns the storage format, indexing and write path to provide O(1) addressing, native vector support, and seamless integration with native execution engines.
This guide explains how to build a secure, multi‑tenant data lake on Alibaba Cloud by combining OSS Access Points with VPC Gateway Endpoints, covering architecture overview, step‑by‑step configuration, policy examples, and best‑practice considerations for private‑network access.
This article presents a series of seven technical case studies—including Tencent Cloud’s Iceberg‑based batch‑stream integration, AI‑driven data governance with Apache Gravitino, Xiaohongshu’s lakehouse evolution, and a multimodal data‑lake solution—detailing challenges, architectural designs, implementation steps, performance results, and future directions.
This article details the deep integration of StarRocks with Apache Paimon, describing the unified architecture, version evolution, performance enhancements, time‑travel queries, native readers/writers, distributed planning, and future roadmap for achieving lakehouse‑native analytics at scale.
As data volumes surge, the article compares six mainstream storage technologies—relational databases, data warehouses, NoSQL databases, search storage, object storage, and data lakes—explaining their core principles, ideal use cases, and trade‑offs to help you select the most suitable solution for your workload.
The article explains how Kuaishou modernized its data lake by partnering with Apache Hudi to address latency, storage cost, and consistency issues in both AI and BI pipelines, detailing architectural changes, new ingestion tools, partitioning strategies, compaction mechanisms, performance gains and future plans.
During Double‑11 mega‑sales, Taobao Group faced exploding OLAP query traffic, costly data sync pipelines, and slow near‑real‑time analytics, so they unified real‑time and batch data in Paimon, leveraged StarRocks for high‑performance lake queries, tuned cluster settings, and saved nearly ten‑million yuan annually while cutting refresh latency by 80%.
This article details the industry-standard dual pattern of file body storage and metadata registration for unstructured data lake ingestion, covering a three-layer architecture, batch and real-time implementation steps using Paimon and Flink, AI-specific optimizations like label standardization and format conversion, and key operational considerations for permissions, versioning, and cost control.
This article explains data architecture from the ground up, covering data sources, storage options such as operational databases, data warehouses and data lakes, ETL processing steps, layered data modeling, service delivery methods, and governance practices to ensure reliable, secure, and business‑driven data management.
This article provides a practical framework for SMEs to select and implement data platform architectures, comparing traditional warehouses, lakehouse, and multimodal data lakes across cost, ROI, and operational efficiency, with scenario-based recommendations and a lightweight lakehouse implementation guide using open-source stack Flink, Paimon, StarRocks, and MinIO.
The article details JD’s data lake architecture, its 500 PB scale, self‑developed Hudi extensions—including LSM‑Tree‑based MoR tables, custom indexing, IO optimizations, Flink stream scheduling, and NativeIO SDK—along with benchmarks, community contributions, and future roadmap for real‑time big‑data processing.
The article explains the distinct roles of data warehouses, data marts, and data lakes, illustrates their differences with analogies and real‑world cases, outlines a three‑step strategy for enterprises, highlights common pitfalls, and offers a decision guide to help organizations choose the right architecture for their data needs.
Haier’s digital transformation leverages a cloud‑native, open‑source‑based multi‑modal data lake that unifies structured and unstructured industrial data, uses metadata models and knowledge graphs for governance, and provides AI‑ready services that balance performance, cost, and real‑time requirements.
The article walks through the evolution of data management—from idealistic visions and messy “shit mountains” to the realities of data lakes, metadata layers, governance challenges, trust breakdowns, and finally the promise of DataOps as a hopeful path forward.
This article explains Apache Hudi’s core concepts, including its timeline architecture, file layout, indexing mechanisms, and the two primary table types—Copy on Write and Merge on Read—along with their trade‑offs and the various query modes such as snapshot, time‑travel, and incremental queries.
This article explains Apache Hudi’s core architecture, detailing the timeline mechanism, file layout, indexing strategies, the two main table types (Copy‑On‑Write and Merge‑On‑Read), and various query modes such as snapshot, time‑travel, read‑optimized and incremental queries.
Uber rebuilt its data‑lake ingestion pipeline with Apache Flink, replacing batch jobs with a streaming architecture that cuts data freshness from hours to minutes, lowers compute usage by 25%, and solves challenges like small‑file proliferation, partition skew, and checkpoint‑commit synchronization at petabyte scale.
The article recaps the Apache Hudi Asia Meetup hosted by JD, covering community updates, JD's data‑lake challenges, the upcoming Hudi 1.1 release, JD's architectural redesign, Kuaishou's real‑time lake adoption, and Huawei Cloud's deep optimizations, all aimed at building an AI‑native, real‑time lakehouse.
This article collection showcases seven advanced data engineering solutions—from Tencent Cloud's Iceberg batch‑stream integration and Apache Gravitino metadata lineage to Xiaohongshu's Lakehouse evolution and multimodal AI data lake implementations—highlighting architectural innovations, performance optimizations, and real‑world deployment insights for modern big‑data platforms.
Apache XTable introduces a lightweight metadata translation layer that decouples data storage from format metadata, enabling zero‑copy, omni‑directional conversion among Hudi, Iceberg, and Delta Lake, allowing organizations to write with one format and read with any engine without duplicating Parquet files.
The article compares data warehouses, data lakes, and lakehouse architectures, illustrating each with real‑world projects, highlighting their core concepts, strengths, limitations, key technologies, and a step‑by‑step migration path to help teams select the most suitable solution for their business needs.
This article series explores Tencent Cloud's Iceberg‑based batch‑stream integration, Apache Gravitino's unified metadata and lineage solution, Xiaohongshu's data‑architecture evolution for the Big AI Data era, and a practical Data+AI multimodal data‑lake implementation, highlighting challenges, architectural designs, and performance gains.
This article collection examines Tencent Cloud’s Iceberg batch‑stream integration, AI‑driven game data governance, Apache Gravitino unified metadata and lineage, Xiaohongshu’s multimodal data‑lake evolution, and Volcano Engine’s Data+AI multimodal lake, highlighting architectures, techniques, performance gains, and practical implementations.
This article chronicles a ten‑year evolution of a self‑built big data platform—detailing early Hadoop clusters, successive migrations to Spark, Hive, Hudi, and StarRocks, the operational challenges encountered, and the comprehensive shift to Alibaba Cloud EMR Serverless that delivered significant cost, performance, and stability gains while outlining future intelligent‑ecosystem plans.
The article explains the concepts of data lakes and lakehouses, compares them with traditional data warehouses, outlines the reliability, performance, and security challenges of data lakes, and then details a practical lakehouse implementation using Apache Iceberg, Flink SQL, CDC pipelines, and supporting tools such as Hive Metastore and Trino.
This article explains how small‑file issues in Paimon's streaming data lake architecture degrade system stability and query speed, and presents practical parameter‑tuning, table‑level settings, asynchronous compaction, and monitoring techniques to mitigate those problems.
This article reviews how Paimon and Doris combine to solve unified storage, data visibility, and performance challenges in modern lakehouse architectures, detailing their complementary features, integration capabilities, and real‑world use cases from leading companies.
This article explains Apache Hudi's architecture, including its timeline mechanism, file layout, indexing strategies, table types (COW and MOR), query options, storage format versioning, backward compatibility, and key configuration settings for managing data lake tables.
This article explains how Zilliz's Vector Lake extends traditional data lakes with a unified storage‑compute architecture optimized for massive unstructured and vector data, detailing its background, key data types, autonomous‑driving use case, data flow, architecture, and deployment options.
This article traces the three‑generation evolution of data architecture—from the structured‑data era of data warehouses, through the flexible, multi‑format data lake, to the unified lakehouse model—explaining the drivers, benefits, challenges, and future trends shaping modern data platforms.
This article examines the challenges of a legacy Lambda‑based membership data warehouse, introduces Apache Paimon’s lakehouse architecture and its key features, and showcases three real‑world implementations—partial‑update order wide tables, Bitmap‑based UV counting, and branch‑based data correction—while discussing benefits, remaining challenges, and future directions.
Xiaomi’s data‑lake team explains how they tackled small‑file issues, unified metadata with Gravitino, migrated Hive to Iceberg and Fileset, leveraged JuiceFS for multi‑cloud storage, and combined Iceberg and Paimon to achieve cost‑effective, high‑performance batch and real‑time analytics.
Xiaomi’s engineers explain how they tackled data‑lake challenges—small files, metadata latency, and multi‑cloud costs—by combining compact storage, Gravitino‑based metadata governance, Iceberg and Paimon formats, and JuiceFS abstraction, achieving lower storage expenses, faster queries, and a roadmap toward intelligent, real‑time, multimodal lakehouses.
This article provides a detailed technical comparison of Apache Hudi, Delta Lake, and Apache Iceberg, covering core features such as incremental pipelines, concurrency control, merge‑on‑read, partition evolution, multi‑mode indexing, ingestion tools, and real‑world use cases, and explains why Hudi often leads for heavy update workloads.
This article explains the evolution from traditional data warehouses and data lakes to the unified Lakehouse architecture, detailing its design, benefits, challenges, and research directions for delivering high‑performance SQL and advanced analytics on open‑format storage.
The article explains how Apache Iceberg v3 replaces the scalable‑limited positional‑delete mechanism in Merge‑on‑Read tables with compact Deletion Vectors, detailing the performance, I/O and metadata drawbacks of positional deletes and showing how the new bitmap‑based approach resolves them.
This guide presents a curated list of cutting‑edge lakehouse, real‑time analytics, and big‑data implementations from industry leaders, followed by a QR code that lets you download the full eBook for deeper insights.
This article reviews common data lake interview questions—covering problem definition, Flink-to-Hudi row_number deduplication, retract streams, pipeline architecture optimizations, and read/write best practices—providing concise explanations and practical insights for candidates.
This article outlines the overall architecture, data architecture, logical project structure, and the construction of a data resource center for a data lake and application platform, illustrated through a series of diagrams that depict each component and their interconnections.
This article presents a curated collection of cutting‑edge data lake and lakehouse case studies—including real‑time analytics, cloud‑native architectures, industry implementations from sales platforms to automotive IoT, and the latest advancements in open‑source projects—offering insights into modern big‑data strategies and governance.
In the era of rapid AI growth, Fluss offers a unified lake‑stream architecture that tackles data quality, timeliness, scale, and multimodal challenges by tightly integrating Flink streaming with a high‑performance data lake, enabling seamless real‑time and batch analytics for AI workloads.
The article reviews key topics from the FFA2025 Singapore conference, highlighting Flink’s new AI‑focused Agents framework, the breakthrough Flink 2.0 disaggregated state architecture, emerging lake storage solutions like Paimon, and the Fluss streaming table store, illustrating how big‑data platforms are evolving for AI workloads.
The Flink Forward Asia 2025 conference in Singapore showcased Apache Flink’s latest advances—including Flink Agents for system‑triggered AI, the cloud‑native Flink 2.0 with disaggregated state management, the multi‑modal lakehouse Paimon, and the Fluss table storage system—highlighting the ecosystem’s shift toward real‑time AI integration.
This article explains how Baidu’s next‑generation data platform Turing 3.0 integrates Apache Iceberg to solve the inefficiencies of the legacy MEG stack, detailing ecosystem components, migration strategies from Hive, table‑level optimizations, and future roadmap for high‑frequency, low‑latency analytics.
StarRocks 3.5 introduces a cluster‑level Snapshot backup for fast recovery, a bulk‑load optimization that reduces small files and compaction cost, smarter partition management with time‑based merging and TTL, multi‑statement transactions with full ACID guarantees, low‑cardinality dictionary support for lake tables, and several security and performance upgrades.
This article details Shopee's Data Infra team's use of the Paimon data lake to build near‑real‑time warehouses, accelerate ODS layers, implement a task‑diagnosis system, and create a reconciliation platform, while sharing future plans and a Q&A session.
This article explains the concept of a real‑time lakehouse, compares it with traditional batch warehouses, introduces Apache Paimon and its innovations such as native upserts, LSM storage, tags and branches, and showcases multiple enterprise use cases that demonstrate its low‑cost, low‑latency stream‑batch integration.
This article explores OpenLake's evolution as a data lake platform for AI, covering the transition from Hive to modern lake formats like Iceberg and Paimon, performance benchmarks, metadata management advances, intelligent storage optimization, and the integration of multimodal support with the Lance file format.
This article details how a leading automotive parts supply‑chain platform migrated from a traditional Hadoop stack to Alibaba Cloud EMR Serverless Spark and DataWorks, achieving faster, more elastic, and cost‑effective data processing, enhanced AI integration, and significant operational improvements across multiple business scenarios.
Coupang’s AI platform replaces costly data‑copy steps with a distributed cache that automatically pulls data from a central lake, boosts GPU utilization across regions, cuts storage and operational expenses, and speeds up model training by up to 40% while simplifying deployment via Kubernetes.
Facing billions of daily logs and the need for minute‑level experiment metrics, Xiaohongshu partnered with Yunqi Tech to design a generic incremental‑compute solution that delivers near‑real‑time data warehousing with lower cost, higher accuracy, simplified pipelines, and improved query performance.
Apache Gravitino is an open‑source metadata service platform that provides a unified, high‑performance, geographically distributed metadata lake, enabling end‑to‑end data governance, multi‑engine access, and direct management of both structured and unstructured data assets across diverse systems.
This article explains data architecture, its purpose, historical development, major enterprise frameworks, various data management system types, and the key advantages it brings to organizations, helping readers understand how to design and implement effective data solutions in modern environments.
At the 2025 DA Data+AI Conference in Shanghai, Tencent Cloud unveiled Setats—a unified stream‑batch‑incremental engine that cuts system costs, delivers second‑level data visibility and real‑time changelog generation, and demonstrates measurable performance gains in automotive IoT analytics while integrating tightly with the WeData platform.
This article explains how Huawei's terminal cloud adopts the Apache Iceberg table format to efficiently manage large-scale datasets, detailing its architecture, feature engineering, merge operations, LSM-based storage, schema versioning, AB testing support, catalog enhancements, and future roadmap for full lifecycle data governance.
The Kuaishou‑hosted Apache Hudi Asia technical salon gathered over 230 attendees and featured seven experts from Kuaishou, Meituan, TikTok, Huawei, JD and others, who shared best practices, architecture designs, and performance optimizations for large‑scale data lake applications across AI, BI, and real‑time workloads.
This article details the business background, technical challenges, and the step‑by‑step implementation of an intelligent data warehouse on Alibaba Cloud MaxCompute, covering offline data pipelines, metric calculation, data analysis, and future plans for data lake and AI‑driven analytics.
The Apache Hudi Asia technical salon held in Beijing on March 29 gathered over 230 on‑site participants and 16,000 online viewers, featuring expert talks from leading Chinese tech companies that showcased real‑world Hudi implementations, performance optimizations, and future roadmap for data‑lake technologies.
The first Apache Hudi Asia Summit in Beijing attracted over 230 attendees, featuring technical discussions on data lake optimization and case studies from companies like Fastly and Meituan.
The article explains how data lakes serve as a foundational component for observability by aggregating raw, diverse data for advanced analysis, while also outlining the technical, cost, and scalability challenges that make them unsuitable for every organization.
This article presents a comprehensive exploration of using Apache Paimon and Flink to design lake tables that support minute‑level latency, low cost, and unified batch‑stream processing for advertising data, covering schema design, partitioning strategies, performance trade‑offs, cost analysis, and operational best practices.
Alibaba Mama’s advertising platform migrated from a monolithic Flink‑Kafka pipeline to a layered Paimon lakehouse, adding DWS upsert support and multi‑layer storage, which delivers minute‑level data freshness, cuts latency by 2.5 hours, reduces resource use over 40 %, halves development effort and achieves ≥99.9 % availability.
This article examines how Apache Iceberg’s snapshot‑based ACID transactions, logical‑physical partition evolution, and COW/MOR update modes enable efficient real‑time data lake ingestion, and demonstrates AutoMQ’s Kafka‑to‑Iceberg Table Topic solution that simplifies schema management, reduces latency, and cuts operational costs.
