BestHub
Sign in

Big Data

Showing 100 articles max
Latest Popular
Data Integration and Governance
Data Integration and Governance
Aug 28, 2026 · Big Data

What Does Metric Management Actually Manage? The Full Lifecycle Explained

The article explains that metric management is not just cataloging metrics but governing their full lifecycle—business definitions, calculation rules, responsibility assignment, data lineage, and versioned change control—to ensure metrics are trustworthy, traceable, and sustainably governed across business, data, and technical layers.

FineDataLinkResponsibility Assignmentcalculation rules
0 likes · 19 min read
What Does Metric Management Actually Manage? The Full Lifecycle Explained
IT Services Circle
IT Services Circle
Aug 27, 2026 · Big Data

10 Python Libraries That Actually Make Data Professionals Stronger

The article presents a curated list of ten Python libraries—Polars, Pandera, DuckDB, Rich, Pydantic, MLflow, tqdm, pyinstrument, RapidFuzz, and sqlite-utils—each chosen for its ability to eliminate specific bottlenecks, reduce errors, and turn guesswork into verifiable steps in data workflows, complete with concrete code examples and practical trade‑offs.

DuckDBMLflowPolars
0 likes · 15 min read
10 Python Libraries That Actually Make Data Professionals Stronger
Baidu Intelligent Cloud Tech Hub
Baidu Intelligent Cloud Tech Hub
Aug 27, 2026 · Big Data

PFS L3 Architecture Deep Dive: Redesigning Parallel File Storage for AI Production Workloads

Baidu's PFS L3 rearchitects parallel file storage for AI production loads with a hybrid data engine (Aries + BlockServer), scalable metadata foundation (MetaDB), high-performance client (RapidFC), RDMA network tuning, transparent tiering to object storage, and AgenticFS for multi-tenant Agent workloads, proven in RL training saving 30k GPU-hours weekly.

AI storageAgenticFSAries
0 likes · 29 min read
PFS L3 Architecture Deep Dive: Redesigning Parallel File Storage for AI Production Workloads
Ray's Galactic Tech
Ray's Galactic Tech
Aug 26, 2026 · Big Data

Layered Real‑Time Data Warehouse with Flink CDC, Kafka & Doris

The article explains why real‑time data‑warehouse projects often fail in production and presents a complete, production‑ready solution that layers ODS, DWD, DWS and ADS using Flink CDC to capture MySQL changes, Kafka for buffering and replay, and Doris for OLAP storage, with detailed guidance on architecture, state handling, fault‑tolerance and operations.

CDCDorisFlink
0 likes · 35 min read
Layered Real‑Time Data Warehouse with Flink CDC, Kafka & Doris
Lakehouse Research Base
Lakehouse Research Base
Aug 25, 2026 · Big Data

StarRocks on Paimon: Morning Fast, Daytime Slow - Root Cause & Layered Optimization

This article analyzes why identical StarRocks queries on Paimon external tables run fast in early morning but slow dramatically during daytime peaks, identifying five layered root causes from file fragmentation to storage pressure, and provides a systematic troubleshooting methodology plus a four-layer optimization framework covering source governance, compute caching, storage scaling, and business scheduling.

CompactionDelete VectorPaimon
0 likes · 21 min read
StarRocks on Paimon: Morning Fast, Daytime Slow - Root Cause & Layered Optimization
Smart Sea Tide
Smart Sea Tide
Aug 25, 2026 · Big Data

Data Governance & Metrics Library Planning: A Symbiotic Framework for Business Value

This article explains how data governance establishes reliable data standards and quality to support metrics library planning, while metrics libraries define business-driven KPIs that guide governance priorities, creating a virtuous cycle where governance ensures metric accuracy and metric outcomes measure governance effectiveness, ultimately maximizing data value.

KPIbusiness valuedata governance
0 likes · 6 min read
Data Governance & Metrics Library Planning: A Symbiotic Framework for Business Value
Lakehouse Research Base
Lakehouse Research Base
Aug 24, 2026 · Big Data

Apache Fluss: Four Merge Engines Explained for Real-Time Analytics

This article systematically analyzes Apache Fluss's four merge engines—Default (LastRow), FirstRow, Versioned, and Aggregation—covering core principles, supported operations, behavior examples, applicable scenarios, and a production selection guide for real-time data warehousing.

Apache FlussFirstRowLastRow
0 likes · 14 min read
Apache Fluss: Four Merge Engines Explained for Real-Time Analytics
Smart Sea Tide
Smart Sea Tide
Aug 24, 2026 · Big Data

Big Data Governance Platform: 5-Layer Architecture, 9 Modules & 3-Phase Implementation

This article presents a comprehensive framework for building a big data governance platform, covering a five-layer architecture, nine core governance modules, a three-phase implementation roadmap, organizational structure, success metrics, and industry-specific selection guidance to transform chaotic data into trusted, findable, and controllable assets.

Implementation Roadmapbig data governancedata architecture
0 likes · 10 min read
Big Data Governance Platform: 5-Layer Architecture, 9 Modules & 3-Phase Implementation
Niu Liu
Niu Liu
Aug 23, 2026 · Big Data

Paimon: Unifying Storage for Real-Time Data Warehouses

Apache Paimon combines a lakehouse table format with an LSM-tree update engine to handle continuous updates, historical snapshots, and multi-engine access in real-time data warehouses, reducing the need for separate Kafka, Hive, and OLAP systems.

Apache PaimonCDCFlink
0 likes · 23 min read
Paimon: Unifying Storage for Real-Time Data Warehouses
Lakehouse Research Base
Lakehouse Research Base
Aug 23, 2026 · Big Data

Apache Fluss Dual-Table Model: LogTable and PrimaryKeyTable Explained

This article details Apache Fluss's dual-table model where LogTable handles high-throughput append-only streaming with columnar storage while PrimaryKeyTable supports real-time updates, partial merges, and CDC changelogs via integrated RocksDB state storage, sharing unified tiering and lakehouse sinking.

Apache FlussCDCLogTable
0 likes · 19 min read
Apache Fluss Dual-Table Model: LogTable and PrimaryKeyTable Explained
Lakehouse Research Base
Lakehouse Research Base
Aug 22, 2026 · Big Data

Apache Fluss: Dual Engines & Tiered Storage Power Real-Time Lakehouse

This article deep-dives into Apache Fluss architecture, detailing its Master-Worker design, dual LogStore and KvStore engines, tablet-based sharding, tiered hot/cold storage on remote object stores, and Flink-centric client integration, explaining how these components unify streaming and lakehouse workloads.

Apache FlussFlink connectorKvStore
0 likes · 14 min read
Apache Fluss: Dual Engines & Tiered Storage Power Real-Time Lakehouse
DataFunTalk
DataFunTalk
Aug 22, 2026 · Big Data

Next‑Gen MaxCompute: Full‑Modal Engine and Agentic Upgrade Overview

The article outlines MaxCompute’s evolution into a Data+AI platform with a unified model management layer, heterogeneous compute quotas, Blob‑type full‑modal storage, AI Function APIs, and an Agentic interface built on MCP, Skills, and metadata, highlighting real‑world pipelines and performance gains.

AI FunctionAgenticData+AI
0 likes · 12 min read
Next‑Gen MaxCompute: Full‑Modal Engine and Agentic Upgrade Overview
Niu Liu
Niu Liu
Aug 21, 2026 · Big Data

Real-Time Data Warehouse Evolution: Flink Writes, Paimon Stores, Doris Queries, One Platform Manages

This article details a lightweight real-time data warehouse architecture using Apache Flink 2.0+ for streaming writes, Apache Paimon for lakehouse storage, Apache Doris for interactive queries, and a custom RT-DWH management platform to unify metadata, quality, permissions, and observability — reducing component count and operational complexity for small-to-medium teams.

Apache DorisApache FlinkApache Paimon
0 likes · 21 min read
Real-Time Data Warehouse Evolution: Flink Writes, Paimon Stores, Doris Queries, One Platform Manages
Smart Sea Tide
Smart Sea Tide
Aug 21, 2026 · Big Data

Designing Scalable Data Warehouses: Architecture, Modeling, Scheduling, and Metric Construction

The article explains how the surge of data in the DT era makes traditional storage insufficient, defines a data warehouse as a subject‑oriented, integrated, stable collection for decision support, outlines its development lifecycle—including integration, modeling, services, scheduling, metadata and quality management—and stresses that building a warehouse is an ongoing, iterative process driven by evolving business needs.

Data ModelingETLMetrics
0 likes · 3 min read
Designing Scalable Data Warehouses: Architecture, Modeling, Scheduling, and Metric Construction
Didi Tech
Didi Tech
Aug 20, 2026 · Big Data

Didi’s Three Sessions at COC Asia 2026: Object Storage, Lakehouse, AI‑Friendly

At Apache COC Asia 2026, Didi’s engineers presented three deep‑dive sessions covering the evolution of massive‑scale object storage with Ozone, cost‑effective lakehouse migration using Iceberg and real‑time ingestion, and AI‑friendly enhancements to the open‑source Seata project, detailing architectural choices, performance gains, and open‑source contributions.

AI‑Friendly Open‑SourceApache OzoneSeata
0 likes · 11 min read
Didi’s Three Sessions at COC Asia 2026: Object Storage, Lakehouse, AI‑Friendly
Tencent Technical Engineering
Tencent Technical Engineering
Aug 20, 2026 · Big Data

Tencent SuperSQL Sets New TPC‑DS World Record, Leads in Performance and Cost Efficiency

Tencent's SuperSQL achieved a 654 million composite score on the TPC‑DS 100 TB benchmark—nearly ten times the previous record—while cutting performance‑per‑dollar cost to 11.04 CNY (about one‑sixth of earlier results), thanks to innovations in query optimization, a vectorized execution engine, adaptive memory and shuffle mechanisms, and AI‑driven diagnostics that together deliver superior speed, scalability, and business value.

AI diagnosticsDistributed ComputingSuperSQL
0 likes · 18 min read
Tencent SuperSQL Sets New TPC‑DS World Record, Leads in Performance and Cost Efficiency
Lakehouse Research Base
Lakehouse Research Base
Aug 20, 2026 · Big Data

Big Data Across 10 Industries: Finance, New Energy & AI — Who Reigns as Data King?

This article maps big data applications across ten industries — finance, retail, manufacturing, healthcare, government, transportation, new energy, AI, agriculture, and property — detailing use cases, technical architectures, and measurable value, concluding that AI and new energy form a mutually reinforcing data-driven loop.

Artificial IntelligenceIndustry ApplicationsMLOps
0 likes · 32 min read
Big Data Across 10 Industries: Finance, New Energy & AI — Who Reigns as Data King?
Architecture Digest
Architecture Digest
Aug 19, 2026 · Big Data

How to Deduplicate 4 Billion QQ IDs with Only 1 GB Memory

The article explains how to solve the classic massive‑data deduplication problem of 4 billion QQ numbers within a 1 GB memory limit by analyzing memory requirements, comparing BitMap and Bloom filter approaches, and detailing a step‑by‑step BitMap implementation with its pros and cons.

Bitmapalgorithmdeduplication
0 likes · 5 min read
How to Deduplicate 4 Billion QQ IDs with Only 1 GB Memory