Xiaohongshu transformed its data platform from a simple ClickHouse‑based stack to a Lambda‑enhanced architecture and finally to a Lakehouse with incremental compute, cutting architecture complexity, resource and development costs by two‑thirds while delivering second‑level analytics on petabyte‑scale data.
Polars outperforms Pandas on large‑scale ETL by using a multithreaded lazy execution model, columnar Arrow storage, and query optimizations, delivering up to 94× speedups on 10 GB workloads, while Pandas remains suitable for smaller datasets and tight ML ecosystem integration.
The article details Xiaohongshu’s journey from a simple ClickHouse‑based ad‑hoc analytics stack to a Lambda‑style architecture and finally to a lakehouse with generic incremental compute, cutting architecture complexity, resource cost and development effort each to roughly one‑third while achieving sub‑10‑second query latency on petabyte‑scale data.
The 2026 China University Computer Competition's Big Data Challenge revealed the monthly star award winners, each receiving a prize, and the top teams detailed their feature engineering, model selection, training validation, and portfolio combination tactics that led to their success.
The 2026 China University Computer Competition Big Data Challenge announced its Monthly Star winners, and the top teams detailed their data preparation, StockTransformer and LightGBM modeling pipelines, feature engineering, validation strategies, ensemble techniques, and key lessons learned from the competition.
By adopting Volcano Engine EMR's serverless Spark Bolt engine, YouJing Technology cut weekly compute consumption by 50% and reduced core task duration by 40%, while MiMan Technology lowered resource use over 60% and sped up key SQL jobs by more than 20%, all without code changes.
Xiaohongshu transformed its data platform from a simple ClickHouse‑based ad‑hoc analysis system to a Lambda‑style architecture and finally to a lakehouse with incremental compute, cutting architecture complexity, resource and development costs by one‑third while delivering second‑level queries over petabyte‑scale data.
The article analyzes how Taobao Live’s data team replaced traditional ETL bottlenecks with an AI‑native, three‑layer architecture built on DataWorks Data Agent, adopting NL2DSL2SQL, multi‑agent orchestration, and standardized pipelines to achieve near‑full automation and higher accuracy.
Litefuse open‑sources a single‑process, sub‑400 MB binary that deploys an Agent observability and evaluation platform in about 25 seconds, explains why Docker‑free deployment matters, and details how Apache Doris’s inverted index, VARIANT JSON type, and compute‑storage separation address the massive, long‑text, semi‑structured traces that differentiate Agent monitoring from traditional observability.
Xiaohongshu, with over 3.5 billion monthly users and daily logs in the trillions, migrated 500 PB of data to Alibaba Cloud and iterated its data platform through four architecture generations—ClickHouse‑based ad‑hoc, Lambda, Lakehouse, and a unified incremental compute model—cutting resource, development, and storage costs to one‑third while delivering sub‑10‑second query latency at petabyte scale.
The article systematically outlines DataWorks Data Agent’s progression from a Copilot‑assisted tool to human‑AI collaboration and finally AI‑driven autonomy, details its four‑agent product matrix covering data development, operations diagnostics, autonomous governance and ChatBI, describes three architecture iterations (Dify, AgentScope, QwenCode/OpenClaw) and a cloud‑managed deployment, and cites real‑world efficiency gains such as cutting development cycles from hours to minutes.
The article explains how Hologres AI Function and Skills transform traditional, slow, and fragmented ad‑creative production into a fully automated, SQL‑driven workflow that handles multimodal data ingestion, AI‑based labeling, video generation, and real‑time performance analysis in a single closed‑loop system.
The Flink Forward Asia 2026 conference in Shenzhen showcased Apache Flink's evolution to Agentic Streaming for AI, introduced the multimodal Agentic Lake built on Apache Paimon 2.0, announced Fluss 1.0 as a real‑time context layer, and highlighted performance gains over competing stacks such as Ray and Daft.
Cainiao combines a decade of logistics data-warehouse experience with Alibaba Cloud’s DataWorks Data Agent to build the SuperETL intelligent system, which orchestrates nine fine-grained skills, enforces safety hooks, and boosts data-development efficiency by 2-3× while achieving over 80% AI automation in key scenarios.
The Flink Forward Asia 2026 conference in Shenzhen announced the evolution of Apache Flink toward Agentic Streaming for AI, unveiled multimodal data lake projects like Apache Paimon 2.0 and Fluss, highlighted performance gains over competing stacks, and showcased collaborations with NVIDIA to accelerate real‑time AI workloads.
The article details how Taobao Live’s data engineering team replaced traditional ETL bottlenecks with a three‑layer, AI‑native architecture built on DataWorks Data Agent, using NL2DSL2SQL, ontology‑driven knowledge bases, and multi‑agent collaboration to achieve near‑100% code generation and higher accuracy.
The article systematically outlines DataWorks Data Agent’s three‑stage evolution—from Copilot assistance to human‑AI collaboration and finally AI‑driven autonomy—details its four‑agent product matrix covering the full data lifecycle, describes the cloud‑managed engineering rollout, and presents a Taobao flash‑sale case where development cycles shrank from hours to minutes, highlighting efficiency gains, security measures, and architectural iterations.
This article walks through using DataWorks Data Agent to automate end‑to‑end data‑warehouse development—from preparing source tables and a structured requirement document, uploading it, crafting task commands, selecting execution modes and models, to the agent generating SQL, building workflows, publishing them, and producing a final report—all without writing SQL manually.
The article reviews four major distributed storage systems—HDFS, Ceph, GlusterFS, and FastDFS—detailing their architectures, core strengths such as HDFS’s batch processing, Ceph’s unified object/block/file capabilities, GlusterFS’s horizontal scalability, and FastDFS’s lightweight handling of small files, while also noting each solution’s limitations.
Xiaohongshu, with over 350 million monthly users and daily logs in the billions, migrated its data platform from AWS to Alibaba Cloud and iterated four times—from a ClickHouse‑based ad‑hoc layer to a Lambda architecture and finally a Lakehouse with incremental compute—cutting architecture complexity, resource cost and development effort each to about one‑third while delivering second‑level analytics on trillion‑scale data.
