BestHub
Sign in

Big Data

Showing 100 articles max
Latest Popular
IT Services Circle
IT Services Circle
Aug 27, 2026 · Big Data

10 Python Libraries That Actually Make Data Professionals Stronger

The article presents a curated list of ten Python libraries—Polars, Pandera, DuckDB, Rich, Pydantic, MLflow, tqdm, pyinstrument, RapidFuzz, and sqlite-utils—each chosen for its ability to eliminate specific bottlenecks, reduce errors, and turn guesswork into verifiable steps in data workflows, complete with concrete code examples and practical trade‑offs.

Data EngineeringDuckDBMLflow
0 likes · 15 min read
10 Python Libraries That Actually Make Data Professionals Stronger
Ray's Galactic Tech
Ray's Galactic Tech
Aug 26, 2026 · Big Data

Layered Real‑Time Data Warehouse with Flink CDC, Kafka & Doris

The article explains why real‑time data‑warehouse projects often fail in production and presents a complete, production‑ready solution that layers ODS, DWD, DWS and ADS using Flink CDC to capture MySQL changes, Kafka for buffering and replay, and Doris for OLAP storage, with detailed guidance on architecture, state handling, fault‑tolerance and operations.

CDCData EngineeringDoris
0 likes · 35 min read
Layered Real‑Time Data Warehouse with Flink CDC, Kafka & Doris
DataFunTalk
DataFunTalk
Aug 22, 2026 · Big Data

Next‑Gen MaxCompute: Full‑Modal Engine and Agentic Upgrade Overview

The article outlines MaxCompute’s evolution into a Data+AI platform with a unified model management layer, heterogeneous compute quotas, Blob‑type full‑modal storage, AI Function APIs, and an Agentic interface built on MCP, Skills, and metadata, highlighting real‑world pipelines and performance gains.

AI FunctionAgenticBig Data
0 likes · 12 min read
Next‑Gen MaxCompute: Full‑Modal Engine and Agentic Upgrade Overview
Smart Sea Tide
Smart Sea Tide
Aug 21, 2026 · Big Data

Designing Scalable Data Warehouses: Architecture, Modeling, Scheduling, and Metric Construction

The article explains how the surge of data in the DT era makes traditional storage insufficient, defines a data warehouse as a subject‑oriented, integrated, stable collection for decision support, outlines its development lifecycle—including integration, modeling, services, scheduling, metadata and quality management—and stresses that building a warehouse is an ongoing, iterative process driven by evolving business needs.

ArchitectureBig DataData Modeling
0 likes · 3 min read
Designing Scalable Data Warehouses: Architecture, Modeling, Scheduling, and Metric Construction
Didi Tech
Didi Tech
Aug 20, 2026 · Big Data

Didi’s Three Sessions at COC Asia 2026: Object Storage, Lakehouse, AI‑Friendly

At Apache COC Asia 2026, Didi’s engineers presented three deep‑dive sessions covering the evolution of massive‑scale object storage with Ozone, cost‑effective lakehouse migration using Iceberg and real‑time ingestion, and AI‑friendly enhancements to the open‑source Seata project, detailing architectural choices, performance gains, and open‑source contributions.

AI‑Friendly Open‑SourceApache OzoneData Migration
0 likes · 11 min read
Didi’s Three Sessions at COC Asia 2026: Object Storage, Lakehouse, AI‑Friendly
Tencent Technical Engineering
Tencent Technical Engineering
Aug 20, 2026 · Big Data

Tencent SuperSQL Sets New TPC‑DS World Record, Leads in Performance and Cost Efficiency

Tencent's SuperSQL achieved a 654 million composite score on the TPC‑DS 100 TB benchmark—nearly ten times the previous record—while cutting performance‑per‑dollar cost to 11.04 CNY (about one‑sixth of earlier results), thanks to innovations in query optimization, a vectorized execution engine, adaptive memory and shuffle mechanisms, and AI‑driven diagnostics that together deliver superior speed, scalability, and business value.

AI DiagnosticsBig DataDistributed Computing
0 likes · 18 min read
Tencent SuperSQL Sets New TPC‑DS World Record, Leads in Performance and Cost Efficiency
Architecture Digest
Architecture Digest
Aug 19, 2026 · Big Data

How to Deduplicate 4 Billion QQ IDs with Only 1 GB Memory

The article explains how to solve the classic massive‑data deduplication problem of 4 billion QQ numbers within a 1 GB memory limit by analyzing memory requirements, comparing BitMap and Bloom filter approaches, and detailing a step‑by‑step BitMap implementation with its pros and cons.

AlgorithmBitMapdeduplication
0 likes · 5 min read
How to Deduplicate 4 Billion QQ IDs with Only 1 GB Memory
DaTaobao Tech
DaTaobao Tech
Aug 19, 2026 · Big Data

Zero‑Code Real‑Time Data Pipelines Powered by an AI‑Driven Metric Engine

The article details how a live‑streaming data team built an AI‑assisted, metric‑driven system that lets users describe a real‑time data task in under 200 characters, automatically generates DSL and Flink SQL, and supports safe incremental adjustments via a Hook protocol, cutting development time from days to minutes.

AI-assisted data pipelineDSLFlink
0 likes · 17 min read
Zero‑Code Real‑Time Data Pipelines Powered by an AI‑Driven Metric Engine
Smart Sea Tide
Smart Sea Tide
Aug 18, 2026 · Big Data

How to Choose and Architect a Data Lake Platform for Enterprise Digital Transformation

The article outlines the strategic need for a unified data lake in a digital‑focused enterprise, details functional and non‑functional requirements such as linear scalability, real‑time and batch processing, multi‑tenant support, security and governance, and presents a comprehensive architecture design that integrates storage, compute, and management components.

Big DataCloud NativeData Governance
0 likes · 19 min read
How to Choose and Architect a Data Lake Platform for Enterprise Digital Transformation
Big Data Technology Tribe
Big Data Technology Tribe
Aug 14, 2026 · Big Data

How lance‑spark Implements Blob V2 Support: A Deep Dive

The article explains how lance‑spark enables Lance's Blob V2 storage by marking a column with blob encoding and setting file_format_version ≥ 2.2, describes the metadata‑driven descriptor schema, the write path that still accepts Spark BINARY, and the size‑based placement of binary data into inline, packed, dedicated or external blob files.

ArrowBlob V2Spark
0 likes · 12 min read
How lance‑spark Implements Blob V2 Support: A Deep Dive
Data Integration and Governance
Data Integration and Governance
Aug 11, 2026 · Big Data

What Is Data Architecture? Clarifying Databases, Data Warehouses, Lakes, and Middle Platforms

As enterprises add more systems, data volumes explode while accessing it becomes harder; this article defines data architecture, explains how databases, data warehouses, data lakes, and data middle platforms each solve distinct layers, and outlines the key questions and challenges for building a unified, reusable data ecosystem.

Big DataDatabasedata architecture
0 likes · 13 min read
What Is Data Architecture? Clarifying Databases, Data Warehouses, Lakes, and Middle Platforms
ITPUB
ITPUB
Aug 9, 2026 · Big Data

Why a CK+Kafka+Filebeat Stack Beats ELK for Log Analytics

The article compares Elasticsearch and ClickHouse for log storage, analyzes cost and performance, and provides a step‑by‑step guide to deploying a Zookeeper‑Kafka‑Filebeat‑ClickHouse pipeline, including common pitfalls and their solutions, showing how the combo reduces server costs by up to half.

ClickHouseElasticSearchFilebeat
0 likes · 15 min read
Why a CK+Kafka+Filebeat Stack Beats ELK for Log Analytics
DeepHub IMBA
DeepHub IMBA
Aug 7, 2026 · Big Data

Pandas vs Polars vs DuckDB: Benchmarking Performance on a 1.2M‑row CSV

A head‑to‑head benchmark on a 2.3 GB CSV (~1.2 million rows) shows Pandas exhausting memory, Polars completing the pipeline in 8.7 seconds with modest RAM, and DuckDB answering the same query in just 12 milliseconds, highlighting distinct trade‑offs for Python data processing.

BenchmarkCSVDuckDB
0 likes · 11 min read
Pandas vs Polars vs DuckDB: Benchmarking Performance on a 1.2M‑row CSV
DataFunSummit
DataFunSummit
Aug 7, 2026 · Big Data

Apache Fluss Graduates to Top‑Level Project, Launching Agentic Lake’s Full Real‑Time Era

In July 2024 Apache Fluss received unanimous approval from the Apache Incubator IPMC and ASF board, graduating to a Top‑Level Project; the article details its community growth, core Lakestream architecture, real‑time streaming storage capabilities, adoption by major enterprises, and how it enables AI agents with fresh, unified data for real‑time decisions.

AI AgentApache FlussLakehouse
0 likes · 14 min read
Apache Fluss Graduates to Top‑Level Project, Launching Agentic Lake’s Full Real‑Time Era
AntData
AntData
Aug 7, 2026 · Big Data

Designing Lampara: Ant Group’s Real‑Time Data Processing System for End‑to‑End SLA Guarantees

The article details Ant Group’s Lampara, a next‑generation real‑time data processing platform that embeds end‑to‑end SLA guarantees, active disaster‑recovery, scenario‑driven development and enhanced operators, showing how it improves reliability, efficiency and cost while supporting large‑scale business scenarios such as flash sales, AI agents and marketing campaigns.

Big DataDistributed SystemsLampara
0 likes · 18 min read
Designing Lampara: Ant Group’s Real‑Time Data Processing System for End‑to‑End SLA Guarantees