BestHub
Sign in

Big Data

Showing 100 articles max
Latest Popular
ITPUB
ITPUB
Sep 14, 2026 · Big Data

Flink CDC vs Canal: Building 99.99% Consistent Real-Time Data Pipelines

This article compares Flink CDC with traditional Canal-based architectures, explains Flink CDC's core principles including lock-free snapshots and exactly-once semantics, provides DataStream and SQL code examples for MySQL integration, and covers five high-frequency interview questions on DDL handling, DELETE capture, and large-table optimization.

DebeziumExactly OnceFlink CDC
0 likes · 12 min read
Flink CDC vs Canal: Building 99.99% Consistent Real-Time Data Pipelines
Smart Sea Tide
Smart Sea Tide
Sep 14, 2026 · Big Data

Metabase: Open-Source BI & Embedded Analytics for Everyone

Metabase is an open-source business intelligence platform that empowers non-technical users with no-code visual querying and AI-assisted natural language analysis, while providing data professionals a full SQL editor, interactive dashboards, multi-database connectivity, and embeddable analytics with granular access controls.

BIMetabasePermissions
0 likes · 3 min read
Metabase: Open-Source BI & Embedded Analytics for Everyone
DataFunTalk
DataFunTalk
Sep 11, 2026 · Big Data

Ant Group's Financial Data Ontology: Automating Business Semantics with LangGraph

Ant Group solves inconsistent business definitions across thousands of tables by building a financial data knowledge ontology using a six-node LangGraph state machine that automates schema perception, entity resolution, and conflict marking without forced merging, enabling a three-layer retrieval architecture that cuts cross-opportunity identification from days to hours and achieves 85% anomaly analysis accuracy.

Automated Ontology ConstructionConflict ResolutionFinancial Data Ontology
0 likes · 4 min read
Ant Group's Financial Data Ontology: Automating Business Semantics with LangGraph
Smart Sea Tide
Smart Sea Tide
Sep 11, 2026 · Big Data

Data Planning Blueprint: From Real Needs to DRD and Collection

This article outlines a comprehensive methodology for data planning, covering historical context, requirement analysis with subtractive thinking, Data Requirements Document (DRD) design, data flow diagrams, data dictionaries, and internal versus external data collection strategies including web scraping considerations.

DRDRequirement Analysisbig data
0 likes · 15 min read
Data Planning Blueprint: From Real Needs to DRD and Collection
iQIYI Technical Product Team
iQIYI Technical Product Team
Sep 10, 2026 · Big Data

Inside iQIYI's Agent Team: Automating Cross-Service Big Data Diagnosis

iQIYI built a Big Data Assistant using an Agent Team architecture where a Coordinator delegates tasks to domain-specific Service Agents (Scheduling, Spark, Flink, ML, Data Lake, StarRocks) that collaborate via a shared blackboard to diagnose cross-service issues, evolving from rule-based workflows to LangGraph-based agents with Harness engineering for context management, tool control, and observability.

Agent TeamCross-Service DiagnosisHarness Engineering
0 likes · 21 min read
Inside iQIYI's Agent Team: Automating Cross-Service Big Data Diagnosis
Frontline Investigation
Frontline Investigation
Sep 7, 2026 · Big Data

Why More Data Interfaces Make Quality Issues Harder to Trace

As data interfaces proliferate, quality issues shift from simple errors to semantic drift across systems; this article analyzes meaning, time, and responsibility drift, proposes a four-question lineage framework, and advocates lightweight change records to maintain trust in evolving data ecosystems.

GB/T 34960change managementdata governance
0 likes · 12 min read
Why More Data Interfaces Make Quality Issues Harder to Trace
Lakehouse Research Base
Lakehouse Research Base
Sep 6, 2026 · Big Data

StarRocks Query Acceleration on Paimon: Production Tuning & Best Practices

This article details production practices for accelerating StarRocks queries on Paimon external tables, covering architecture, table design (partitioning, bucketing, compaction), dirty data handling, catalog configuration, predicate pushdown verification, SQL optimization with partition/bucket pruning, query method selection (direct, async materialized views, hot/cold tiering), and BE-level tuning.

Bucket PruningCompactionMaterialized View
0 likes · 22 min read
StarRocks Query Acceleration on Paimon: Production Tuning & Best Practices
Niu Liu
Niu Liu
Sep 6, 2026 · Big Data

Real-Time Data Quality: Handling Late Data, Reconciliation, and Alerting

This article explores data quality challenges in real-time data warehouses, covering timing agreements for late-arriving data, reconciliation techniques using Flink and Paimon, SQL-based checking strategies, alerting with actionable details, and iterative quality validation processes.

AlertingCDCDoris
0 likes · 15 min read
Real-Time Data Quality: Handling Late Data, Reconciliation, and Alerting
DataFunSummit
DataFunSummit
Sep 4, 2026 · Big Data

Tencent Cloud Big Data Computing Intelligence: Unifying SQL and Multimodal AI Workloads

Tencent Cloud's big data platform introduces computing intelligence as a unified paradigm for structured SQL and multimodal AI workloads, detailing engines Meson, Xpark, TCRay, and ES that enable GPU-accelerated SQL, distributed multimodal processing, unified CPU/GPU scheduling, and evaluatable RAG for agent-ready data platforms.

Computing IntelligenceMesonRAG
0 likes · 17 min read
Tencent Cloud Big Data Computing Intelligence: Unifying SQL and Multimodal AI Workloads
ITPUB
ITPUB
Sep 4, 2026 · Big Data

Sentinel-2 End-to-End Python Workflow: 47 Indices & 8-Class Land Classification

This article presents a complete Python workflow for Sentinel-2 L2A imagery, covering automated download, preprocessing, 47 remote sensing indices calculation, index applications, and GB/T 21010-2017 compliant 8-class land cover classification with a PySide6 visualization interface.

GB/T 21010-2017PySide6Python
0 likes · 16 min read
Sentinel-2 End-to-End Python Workflow: 47 Indices & 8-Class Land Classification
Data Integration and Governance
Data Integration and Governance
Sep 4, 2026 · Big Data

ETL vs ELT vs ETLT: Where Should Data Transformation Live?

This article explains the core differences between ETL, ELT, and ETLT data integration patterns, emphasizing that the key distinction is where transformation occurs—before or after loading—and provides five decision criteria for choosing the right approach based on data latency, raw data retention, compute costs, and business logic volatility.

ELTETLETLT
0 likes · 16 min read
ETL vs ELT vs ETLT: Where Should Data Transformation Live?
Code Farming
Code Farming
Sep 4, 2026 · Big Data

How Big Data Platforms Turn Dormant Data into User Value: A Ride-Hailing Case Study

This article details a three-stage big data pipeline—collection, computation, and application—used by a ride-hailing platform to build user and vehicle profiles, enable intelligent order matching that boosts click-through rates by 30%, and create a self-reinforcing data flywheel that continuously improves service quality.

Sparkarchitecture designbig data platform
0 likes · 8 min read
How Big Data Platforms Turn Dormant Data into User Value: A Ride-Hailing Case Study
Huolala Tech
Huolala Tech
Sep 2, 2026 · Big Data

Building DataLeap's AI-Ready Data Knowledge Base: From Scattered Metrics to Reliable Conversational Analytics

DataLeap consolidates fragmented metric definitions, business rules, and SQL templates into a governed, AI-understandable knowledge base, enabling conversational analytics with 90% recall and stability through semantic integration, layered knowledge engines, and an evaluatable data flywheel.

AI analyticsDataLeapconversational analytics
0 likes · 10 min read
Building DataLeap's AI-Ready Data Knowledge Base: From Scattered Metrics to Reliable Conversational Analytics
Data Integration and Governance
Data Integration and Governance
Sep 1, 2026 · Big Data

Building a Real-Time Data Warehouse: End-to-End Pipeline from Kafka to BI

This article details the end-to-end architecture of a real-time data warehouse, covering data ingestion via CDC and Kafka, stream processing challenges like deduplication and event-time handling, layered modeling (ODS/DWD/DWS/ADS), unified metric definitions, and critical production monitoring for latency, lag, data reconciliation, and consistency with offline results.

CDCData ModelingFlink
0 likes · 20 min read
Building a Real-Time Data Warehouse: End-to-End Pipeline from Kafka to BI
DataFunTalk
DataFunTalk
Sep 1, 2026 · Big Data

From Three Months to Two Weeks: DataWorks Data Agent Revamps Xinfēi Technology’s Multi‑Country Data‑Warehouse Delivery

Xinfēi Technology, a fintech data‑intelligence provider, cut its multi‑country data‑warehouse rollout from three months to about two weeks by adopting Alibaba Cloud’s DataWorks Data Agent, which unified data ingestion, task development, quality configuration, lineage analysis and release verification, introduced Skills and an enterprise semantic layer, and achieved up to six‑fold efficiency gains across ODS, DWD, DWS, DQC and reporting.

Data AgentDataWorksEfficiency
0 likes · 10 min read
From Three Months to Two Weeks: DataWorks Data Agent Revamps Xinfēi Technology’s Multi‑Country Data‑Warehouse Delivery
Lakehouse Research Base
Lakehouse Research Base
Aug 31, 2026 · Big Data

Factory Lines & Data Pipelines: The Shared Orchestration DNA

This article reveals the deep structural isomorphism between manufacturing assembly lines and data pipeline orchestration, mapping five shared design philosophies, a 15-dimension logical correspondence, bidirectional lessons, and six practical mental models for data engineers to adopt factory-floor thinking.

assembly linedata engineeringdata orchestration
0 likes · 22 min read
Factory Lines & Data Pipelines: The Shared Orchestration DNA
Data Integration and Governance
Data Integration and Governance
Aug 31, 2026 · Big Data

Data Agent Architecture: 7 Layers to Connect Enterprise Data Beyond Text-to-SQL

This article argues that enterprise Data Agents require a seven-layer data architecture—source, preparation, object, semantic, permission, execution, and verification—rather than simply connecting databases to LLMs, detailing how each layer resolves ambiguity, ensures stability, enforces security, and enables trustworthy analytical reasoning.

AI analyticsData AgentEnterprise Data
0 likes · 18 min read
Data Agent Architecture: 7 Layers to Connect Enterprise Data Beyond Text-to-SQL
Smart Sea Tide
Smart Sea Tide
Aug 31, 2026 · Big Data

B2B vs B2C Data Development: Core Differences in Metrics, Modeling & Team Structure

A data developer with two years each in B2C and B2B e-commerce compares how user focus, metric priorities, team structures, and data warehouse modeling differ between consumer-facing and enterprise-facing products, illustrating with concrete project examples like APP health dashboards and lead allocation systems.

B2B data developmentB2C data developmentETL
0 likes · 17 min read
B2B vs B2C Data Development: Core Differences in Metrics, Modeling & Team Structure