Xiaohongshu transformed its data platform from a simple ClickHouse‑based ad‑hoc analysis system to a Lambda‑style architecture and finally to a lakehouse with incremental compute, cutting architecture complexity, resource and development costs by one‑third while delivering second‑level queries over petabyte‑scale data.
This article walks through the essential Kafka command‑line tools, covering how to check the Kafka version, create, list, describe, alter, and delete topics, manage partitions, produce and consume messages, inspect consumer groups, reset offsets, query broker metadata, view log directories, and run built‑in performance tests for both producers and consumers.
The article analyzes how Taobao Live’s data team replaced traditional ETL bottlenecks with an AI‑native, three‑layer architecture built on DataWorks Data Agent, adopting NL2DSL2SQL, multi‑agent orchestration, and standardized pipelines to achieve near‑full automation and higher accuracy.
Litefuse open‑sources a single‑process, sub‑400 MB binary that deploys an Agent observability and evaluation platform in about 25 seconds, explains why Docker‑free deployment matters, and details how Apache Doris’s inverted index, VARIANT JSON type, and compute‑storage separation address the massive, long‑text, semi‑structured traces that differentiate Agent monitoring from traditional observability.
The article explains why enterprises should layer their data warehouse into ODS, DWD, DWS, and ADS, showing how each layer solves data chaos, inconsistent metrics, performance bottlenecks, duplicate development, and traceability, and how the layers form a clear path from raw records to business insights.
Xiaohongshu, with over 3.5 billion monthly users and daily logs in the trillions, migrated 500 PB of data to Alibaba Cloud and iterated its data platform through four architecture generations—ClickHouse‑based ad‑hoc, Lambda, Lakehouse, and a unified incremental compute model—cutting resource, development, and storage costs to one‑third while delivering sub‑10‑second query latency at petabyte scale.
The article systematically outlines DataWorks Data Agent’s progression from a Copilot‑assisted tool to human‑AI collaboration and finally AI‑driven autonomy, details its four‑agent product matrix covering data development, operations diagnostics, autonomous governance and ChatBI, describes three architecture iterations (Dify, AgentScope, QwenCode/OpenClaw) and a cloud‑managed deployment, and cites real‑world efficiency gains such as cutting development cycles from hours to minutes.
The article explains how Hologres AI Function and Skills transform traditional, slow, and fragmented ad‑creative production into a fully automated, SQL‑driven workflow that handles multimodal data ingestion, AI‑based labeling, video generation, and real‑time performance analysis in a single closed‑loop system.
The article explains why small‑to‑medium teams need a unified web‑based management console for Flink + Paimon real‑time data warehouses, outlines the operational pain points of current script‑driven workflows, and details the proposed solution’s core modules, tech stack, and role‑based access design.
The article walks through the architecture and core workflow of a lightweight real‑time data warehouse built on Flink and Paimon, covering service‑layer refactoring, CDC task lifecycle, dual Flink submission modes, metadata synchronization, and layered security measures for small‑to‑medium enterprises.
The Flink Forward Asia 2026 conference in Shenzhen showcased Apache Flink's evolution to Agentic Streaming for AI, introduced the multimodal Agentic Lake built on Apache Paimon 2.0, announced Fluss 1.0 as a real‑time context layer, and highlighted performance gains over competing stacks such as Ray and Daft.
Cainiao combines a decade of logistics data-warehouse experience with Alibaba Cloud’s DataWorks Data Agent to build the SuperETL intelligent system, which orchestrates nine fine-grained skills, enforces safety hooks, and boosts data-development efficiency by 2-3× while achieving over 80% AI automation in key scenarios.
The Flink Forward Asia 2026 conference in Shenzhen announced the evolution of Apache Flink toward Agentic Streaming for AI, unveiled multimodal data lake projects like Apache Paimon 2.0 and Fluss, highlighted performance gains over competing stacks, and showcased collaborations with NVIDIA to accelerate real‑time AI workloads.
This article explains Lance's row_offsets_to_row_addresses process, showing how logical row numbers are mapped to fragment‑specific RowAddress objects, handling deletion vectors, tombstone rows, and version‑dependent mappings with concrete code examples.
The article analyzes the May 2026 releases of Apache Iceberg V3 and Spark 4.0, explaining how Deletion Vectors, the native VARIANT type, and Spark Connect together transform semi‑structured data handling and real‑time upserts in lakehouse architectures.
The article details how Taobao Live’s data engineering team replaced traditional ETL bottlenecks with a three‑layer, AI‑native architecture built on DataWorks Data Agent, using NL2DSL2SQL, ontology‑driven knowledge bases, and multi‑agent collaboration to achieve near‑100% code generation and higher accuracy.
The article systematically outlines DataWorks Data Agent’s three‑stage evolution—from Copilot assistance to human‑AI collaboration and finally AI‑driven autonomy—details its four‑agent product matrix covering the full data lifecycle, describes the cloud‑managed engineering rollout, and presents a Taobao flash‑sale case where development cycles shrank from hours to minutes, highlighting efficiency gains, security measures, and architectural iterations.
This article walks through using DataWorks Data Agent to automate end‑to‑end data‑warehouse development—from preparing source tables and a structured requirement document, uploading it, crafting task commands, selecting execution modes and models, to the agent generating SQL, building workflows, publishing them, and producing a final report—all without writing SQL manually.
The article reviews four major distributed storage systems—HDFS, Ceph, GlusterFS, and FastDFS—detailing their architectures, core strengths such as HDFS’s batch processing, Ceph’s unified object/block/file capabilities, GlusterFS’s horizontal scalability, and FastDFS’s lightweight handling of small files, while also noting each solution’s limitations.
Xiaohongshu, with over 350 million monthly users and daily logs in the billions, migrated its data platform from AWS to Alibaba Cloud and iterated four times—from a ClickHouse‑based ad‑hoc layer to a Lambda architecture and finally a Lakehouse with incremental compute—cutting architecture complexity, resource cost and development effort each to about one‑third while delivering second‑level analytics on trillion‑scale data.
