BestHub
Sign in

Big Data

Showing 100 articles max
Latest Popular
dbaplus Community
dbaplus Community
Jun 23, 2026 · Big Data

From Hand‑Written SQL to One‑Click Validation: Alibaba’s Verify‑Data Agent Skill Design Review

The article details how Alibaba’s production‑grade Verify‑Data Agent Skill replaces manual, multi‑SQL data validation with a single natural‑language command, automating table discovery, SQL generation, execution, and review‑level reporting, achieving up to 30‑minute turnaround, comprehensive coverage, and robust risk controls for big‑data pipelines.

Big DataData QualityData Validation
0 likes · 28 min read
From Hand‑Written SQL to One‑Click Validation: Alibaba’s Verify‑Data Agent Skill Design Review
dbaplus Community
dbaplus Community
Jun 23, 2026 · Big Data

How AI‑Powered Skills Cut 70% of Repetitive Data Development Work

A real‑world incident where an ADS table stopped updating triggered a three‑second root‑cause discovery and a three‑hour data‑warehouse rebuild using a Claude‑based Skill that eliminated about 70% of the manual, repetitive steps traditionally required in data development, testing, deployment, and operations.

AI AutomationClaudeData Development
0 likes · 12 min read
How AI‑Powered Skills Cut 70% of Repetitive Data Development Work
Past Memory Big Data
Past Memory Big Data
Jun 22, 2026 · Big Data

What’s New in Apache Spark 4.2? Core Features and Architecture Evolution

Apache Spark 4.2 introduces a lightweight Spark Connect architecture, native AI integration, enhanced Metrics View for unified semantics, Arrow‑first performance gains, advanced SQL extensions like vector search and QUALIFY, robust geospatial support, and a revamped streaming engine with auto CDC and sub‑millisecond state cleanup.

Apache SparkArrowMetrics View
0 likes · 13 min read
What’s New in Apache Spark 4.2? Core Features and Architecture Evolution
DataFunTalk
DataFunTalk
Jun 21, 2026 · Big Data

How Zhihu Optimized Spark Jobs with Gluten: A Practical Deep‑Dive

This article details Zhihu's end‑to‑end experience of migrating Spark SQL workloads to the open‑source Gluten framework, covering background performance benchmarks, the architecture of Gluten and Velox, consistency and performance challenges encountered during migration, the concrete fixes applied, and the resulting resource savings and future plans.

Big DataGlutenOptimization
0 likes · 22 min read
How Zhihu Optimized Spark Jobs with Gluten: A Practical Deep‑Dive
DataFunSummit
DataFunSummit
Jun 20, 2026 · Big Data

Building an Agentic Analytics Platform for the Gaming Industry with SelectDB

The article analyzes the fourfold challenges of game‑industry data analysis—high timeliness, massive concurrency, heterogeneous sources, and petabyte‑scale volumes—and explains how SelectDB’s evolution to an AI‑Ready, Agentic platform with MCP and a semantic layer addresses these issues through real‑time OLAP, multimodal processing, and autonomous decision loops.

AI-ReadyAgentic AIBig Data
0 likes · 16 min read
Building an Agentic Analytics Platform for the Gaming Industry with SelectDB
DataFunTalk
DataFunTalk
Jun 20, 2026 · Big Data

How Xiaohongshu Evolved Its Data Architecture for the Big AI Data Era

The article details Xiaohongshu's step‑by‑step migration from a simple ClickHouse‑based analytics stack to a Lambda‑style 2.0 architecture and finally to a Lakehouse‑based 3.0 design, highlighting concrete performance numbers, cost reductions, and the definition of a generic incremental‑compute model (SPOT) that underpins the evolution.

Big DataClickHouseData Architecture
0 likes · 22 min read
How Xiaohongshu Evolved Its Data Architecture for the Big AI Data Era
DataFunSummit
DataFunSummit
Jun 19, 2026 · Big Data

Near‑Real‑Time Data Warehousing with Yunqi Lakehouse: Cases from Xiaohongshu, Kuaishou, Meituan

The article examines how Xiaohongshu, Kuaishou and Meituan adopted Yunqi Lakehouse’s General Incremental Computing and Single‑Engine architecture to achieve near‑real‑time data warehouses, cutting resource usage to as low as 1/20 of full‑batch jobs, reducing data latency from days to minutes, and improving query performance.

Big DataGeneral Incremental ComputingReal-Time Data Warehouse
0 likes · 12 min read
Near‑Real‑Time Data Warehousing with Yunqi Lakehouse: Cases from Xiaohongshu, Kuaishou, Meituan
Alibaba Cloud Native
Alibaba Cloud Native
Jun 19, 2026 · Big Data

Why Real-Time Data Lake Ingestion Is Dropping ETL in the AI Era: Architecture Simplification from Kafka to Iceberg

In the AI‑driven era, enterprises need a data foundation that supports both real‑time consumption and long‑term historical analysis, and the emerging "zero‑ETL" trend moves generic ingestion capabilities from external Flink/Spark jobs into a streamlined Kafka‑to‑Iceberg pipeline, reducing complexity while preserving low latency, consistency, schema evolution, CDC semantics and open‑ecosystem compatibility.

Data LakeIcebergKafka
0 likes · 25 min read
Why Real-Time Data Lake Ingestion Is Dropping ETL in the AI Era: Architecture Simplification from Kafka to Iceberg
Alibaba Cloud Developer
Alibaba Cloud Developer
Jun 18, 2026 · Big Data

How AI-Driven Real-Time Data Lakes Are Ditching ETL: A Kafka‑to‑Iceberg Architecture Simplification

In the AI era, enterprises need a data foundation that supports both low‑latency streaming and long‑term analytics, and the combination of Kafka, Iceberg and object storage is emerging as a preferred solution; by moving ingestion capabilities closer to the message layer and eliminating external ETL jobs, a "zero‑ETL" approach reduces architectural complexity, improves consistency, and streamlines schema evolution and small‑file management.

Data LakeIcebergKafka
0 likes · 27 min read
How AI-Driven Real-Time Data Lakes Are Ditching ETL: A Kafka‑to‑Iceberg Architecture Simplification
DataFunTalk
DataFunTalk
Jun 16, 2026 · Big Data

How MaxCompute Evolves Data Platforms for AI: Architecture, Features, and Real‑World Cases

The article explains how Alibaba Cloud's MaxCompute transforms a traditional data warehouse into a cloud‑native, multimodal Data+AI platform by introducing a four‑layer architecture, SQL‑based AI functions, the Python‑native MaxFrame framework, and a series of industry case studies that demonstrate performance gains and flexible resource scheduling.

Big DataData+AIMaxCompute
0 likes · 11 min read
How MaxCompute Evolves Data Platforms for AI: Architecture, Features, and Real‑World Cases
Linyb Geek Road
Linyb Geek Road
Jun 14, 2026 · Big Data

How to Solve Data Ordering Issues in Apache Kafka

This article explains how Kafka maintains order within partitions using keys and offsets, why ordering across partitions can break, and how to preserve strict sequencing through producer configuration, idempotent producers, and exactly‑once transactional processing.

Kafkadata orderingexactly-once
0 likes · 9 min read
How to Solve Data Ordering Issues in Apache Kafka
dbaplus Community
dbaplus Community
Jun 14, 2026 · Big Data

Why Big Data Is Falling Silent: When Scale Can’t Fake Value Anymore

Although national data production reached 52.26 ZB in 2025 and keeps growing, the term “big data” is disappearing because it no longer serves as an organizational credit that hides the need for real value, responsibility, and measurable business impact, especially in the AI era.

AI impactBig DataData Governance
0 likes · 13 min read
Why Big Data Is Falling Silent: When Scale Can’t Fake Value Anymore
StarRocks
StarRocks
Jun 12, 2026 · Big Data

Building a Millisecond-Responsive Real-Time Data Engine with StarRocks, Fluss, and Paimon

This article presents a lake‑stream integrated solution that combines Apache Fluss, Apache Paimon, and StarRocks to achieve second‑level data freshness, tenfold storage cost reduction, and a single‑query access pattern for both real‑time and historical data, detailing its architecture, advantages, query modes, and future roadmap.

Cost ReductionFlussLakehouse
0 likes · 13 min read
Building a Millisecond-Responsive Real-Time Data Engine with StarRocks, Fluss, and Paimon
DataFunSummit
DataFunSummit
Jun 11, 2026 · Big Data

How MaxCompute Enables Multimodal Storage and Hybrid Computing for Powerful Digital Agents

The article details MaxCompute's three‑stage approach—production‑ready Agent access via MCP and Skill, a business‑oriented semantic layer, and multimodal Blob storage with hybrid compute—culminating in a CPU‑only home‑design demo that showcases end‑to‑end Agent workflows, security controls, and mobile integration.

BLOBDigital AgentHybrid Computing
0 likes · 11 min read
How MaxCompute Enables Multimodal Storage and Hybrid Computing for Powerful Digital Agents
iQIYI Technical Product Team
iQIYI Technical Product Team
Jun 11, 2026 · Big Data

How iQIYI’s QBFS Enables Seamless Hybrid‑Cloud Storage and Cuts Big‑Data Costs by Over 30%

iQIYI’s big‑data team built a self‑developed QBFS virtual file system that unifies private and multiple public clouds, providing transparent routing, automatic migration, intelligent caching and fine‑grained governance, which together reduce storage and compute costs by more than 30 % while supporting scalable analytics.

Big DataCachingData Migration
0 likes · 21 min read
How iQIYI’s QBFS Enables Seamless Hybrid‑Cloud Storage and Cuts Big‑Data Costs by Over 30%
Architect Chen
Architect Chen
Jun 9, 2026 · Big Data

How Kafka Prevents Duplicate Consumption: Three Main Solutions

The article explains why Kafka does not guarantee exactly‑once delivery and presents three practical approaches—business‑level idempotence, manual offset management, and Kafka’s transaction/EOS features—to reliably avoid duplicate message processing.

ConsumerKafkaOffset Management
0 likes · 4 min read
How Kafka Prevents Duplicate Consumption: Three Main Solutions