BestHub
Sign in
Tagged articles

data warehouse

644 articles · Page 3 of 7
DataFunSummit
DataFunSummit
Aug 26, 2024 · Big Data

Building a Doris‑Based Lakehouse Integrated Analytics System at Kuaishou

This article presents Kuaishou's experience of designing and implementing a Doris‑driven lakehouse integrated analytics system, covering the current OLAP landscape, challenges of data duplication and governance, the new architecture with caching and auto‑materialization, implementation details, performance impact, and future work.

Auto MaterializationDorisLakehouse
0 likes · 24 min read
Building a Doris‑Based Lakehouse Integrated Analytics System at Kuaishou
Bilibili Tech
Bilibili Tech
Aug 23, 2024 · Big Data

Accelerating Multi‑Dimensional OLAP Queries in ClickHouse with Grouping Sets, RBM, and Dense Dictionary Encoding

To achieve sub‑second, multi‑dimensional analytics on Bilibili’s hundred‑million‑row datasets, the team built a ClickHouse‑based acceleration layer that combines grouping‑set pre‑aggregation, bitmap (RBM) distinct handling, and a dense dictionary encoding service, dramatically cutting CPU, memory and query latency versus traditional OLAP pipelines.

BitmapClickHouseDictionary Service
0 likes · 28 min read
Accelerating Multi‑Dimensional OLAP Queries in ClickHouse with Grouping Sets, RBM, and Dense Dictionary Encoding
Data Thinking Notes
Data Thinking Notes
Aug 15, 2024 · Big Data

How to Build a Scalable Data Warehouse: Theory, Architecture, and Best Practices

This article outlines practical approaches to data warehouse construction, covering dimensional modeling, layered architecture, capability development, real‑time and batch processing with technologies like Hive, Spark, Flink, Iceberg, and discusses governance, security, and future trends toward data value and real‑time metrics.

Lambda ArchitectureReal-time Analyticsdata governance
0 likes · 13 min read
How to Build a Scalable Data Warehouse: Theory, Architecture, and Best Practices
21CTO
21CTO
Aug 13, 2024 · Databases

How PostgreSQL Can Replace Kafka, Redis, MongoDB and More in Your Stack

This article explores how PostgreSQL’s advanced features—UNLOGGED tables, JSONB, SKIP LOCKED, TimescaleDB, pg_cron, PostGIS, full‑text search, JSON generation, pgaudit, and GraphQL adapters—can replace specialized tools like Kafka, Redis, MongoDB, and others, simplifying the tech stack while boosting performance and maintainability.

CachingGraphQLPostgreSQL
0 likes · 23 min read
How PostgreSQL Can Replace Kafka, Redis, MongoDB and More in Your Stack
DataFunSummit
DataFunSummit
Aug 13, 2024 · Big Data

Data Cost Reduction and Efficiency: Qichacha's Data Architecture and Multi‑Cloud Unified Design

This article presents Qichacha's comprehensive data‑cost‑reduction strategy, detailing its Hadoop‑based three‑pillar architecture, layered data warehouse, Hive upgrades, unified metadata across multi‑cloud clusters, middleware choices such as Alluxio and JuiceFS, version‑compatible hybrid clouds, and Kubernetes‑driven resource orchestration to achieve scalable, low‑cost data processing.

HadoopMulti-Cloudbig data
0 likes · 16 min read
Data Cost Reduction and Efficiency: Qichacha's Data Architecture and Multi‑Cloud Unified Design
DataFunSummit
DataFunSummit
Jul 20, 2024 · Databases

Real-time Data Update Solutions in TCHouse‑C: Architecture, Schema‑less Design, and Performance Evaluation

This article presents TCHouse‑C, a cloud‑native ClickHouse service, detailing its real‑time data update architecture, schema‑less ingestion, various update strategies such as Delete‑Insert and lightweight‑update/delete, and comprehensive performance tests comparing UniqueMergeTree with standard ClickHouse engines across import, query, and update workloads.

ClickHouseDelete-InsertReal-time Updates
0 likes · 32 min read
Real-time Data Update Solutions in TCHouse‑C: Architecture, Schema‑less Design, and Performance Evaluation
Data Thinking Notes
Data Thinking Notes
Jul 11, 2024 · Big Data

How to Build a Robust Data Lineage Foundation for Scalable Business Insights

This article explains how to construct a full‑chain data lineage system, covering its overall architecture, quality measurement framework, and application layer, and demonstrates practical use cases such as handling data growth, monitoring warehouse changes, accelerating development, ensuring consistency, and automating metric decomposition in real‑world business scenarios.

big datadata governancedata lineage
0 likes · 14 min read
How to Build a Robust Data Lineage Foundation for Scalable Business Insights
Baidu Tech Salon
Baidu Tech Salon
Jul 11, 2024 · Industry Insights

How Baidu Feed Evolved Its Data Warehouse with Multi‑Version Wide Tables

This article outlines the step‑by‑step evolution of Baidu's Feed data warehouse—from traditional layered modeling to hour‑level core tables, then real‑time wide tables, and finally a flow‑batch integrated multi‑version wide‑table architecture—highlighting the motivations, design choices, challenges, and resulting benefits.

Data ModelingReal-time Analyticsbig data
0 likes · 10 min read
How Baidu Feed Evolved Its Data Warehouse with Multi‑Version Wide Tables
DataFunSummit
DataFunSummit
Jul 1, 2024 · Big Data

Optimizing JD Retail Data Architecture: From Lambda to Real‑time Unified Processing with Flink, Hudi, and StarRocks

This article details JD Retail's transition from a complex Lambda architecture to a unified real‑time data pipeline using Flink, Hudi, and StarRocks, addressing data completeness versus latency, reducing maintenance costs, improving storage efficiency, and delivering faster, more consistent analytics for business users.

FlinkHudiJD Retail
0 likes · 13 min read
Optimizing JD Retail Data Architecture: From Lambda to Real‑time Unified Processing with Flink, Hudi, and StarRocks
Big Data Technology & Architecture
Big Data Technology & Architecture
Jul 1, 2024 · Big Data

Applying Data Lake (Hudi) at Kuaishou: Architecture Evolution, Use Cases, and Practice

This article details Kuashou's journey of adopting the Hudi data lake, covering business challenges, migration from Hive to Hudi, architectural redesign, promotion strategy, real‑world use cases such as CDC sync and batch‑stream integration, and key lessons learned for large‑scale data engineering.

Big Data ArchitectureHudiKuaishou
0 likes · 11 min read
Applying Data Lake (Hudi) at Kuaishou: Architecture Evolution, Use Cases, and Practice
Past Memory Big Data
Past Memory Big Data
Jun 27, 2024 · Big Data

Inside Presto 2.0: The Native C++ Query Engine Explained

This article provides a detailed technical overview of Presto 2.0, the native C++ query engine built on the Velox library, covering its motivation, vectorized architecture, memory management, performance benchmarks from Meta and IBM, and deployment practices for large‑scale data warehouses.

C++SQLTPC-DS
0 likes · 15 min read
Inside Presto 2.0: The Native C++ Query Engine Explained
DataFunTalk
DataFunTalk
Jun 27, 2024 · Big Data

Data Warehouse Construction and Data Governance Practices at Wing Payment

This presentation by senior data warehouse engineer Huang Luo details Wing Payment’s end‑to‑end data warehouse build, covering background challenges, governance framework, platform architecture, layered modeling, naming standards, asset management, monitoring, and future plans, illustrating how systematic data governance drives cost reduction, efficiency, and security.

AnalyticsData Modelingbig data
0 likes · 14 min read
Data Warehouse Construction and Data Governance Practices at Wing Payment
DataFunTalk
DataFunTalk
Jun 19, 2024 · Big Data

Evolution and Practices of E‑commerce Data Warehouse Governance

This article analyzes the current state, development stages, and comprehensive solutions of e‑commerce data‑warehouse governance, covering data quality, cost, security, and efficiency requirements, and presents a roadmap from early‑stage standardization to mature tool‑driven governance with future outlooks.

Cost ManagementE‑commercebig data
0 likes · 13 min read
Evolution and Practices of E‑commerce Data Warehouse Governance
Data Thinking Notes
Data Thinking Notes
Jun 18, 2024 · Big Data

How to Build a Robust Data Metric System: From Atomic to Derived Indicators

This article explains what a metric (indicator) is, distinguishes atomic, derived and composite indicators, outlines the OSM and UJM modeling methods, describes the steps for constructing a metric system, its integration with data warehouses, and details the design and technical implementation of a metric management platform.

Business IntelligenceData MetricsMetric System
0 likes · 13 min read
How to Build a Robust Data Metric System: From Atomic to Derived Indicators
ITPUB
ITPUB
Jun 9, 2024 · Databases

Doris vs ClickHouse: Which Database Fits Your Workload?

This article compares Doris and ClickHouse across architecture, table creation, ecosystem integration, management tools, query performance, and join capabilities, offering practical guidance on how to choose the right database based on your specific data processing and operational requirements.

ClickHouseDorisSQL
0 likes · 10 min read
Doris vs ClickHouse: Which Database Fits Your Workload?
DataFunTalk
DataFunTalk
Jun 4, 2024 · Databases

From Lambda Architecture to an All‑in‑One Apache Doris Real‑Time/Offline Data Platform for 5G Connected Factories

The article explains how China Unicom transformed its 5G fully‑connected factory data pipeline from a complex Lambda architecture into a streamlined, real‑time and offline‑integrated solution built on Apache Doris, detailing system requirements, architectural redesign, performance gains, and future plans.

5GApache DorisReal-time Analytics
0 likes · 15 min read
From Lambda Architecture to an All‑in‑One Apache Doris Real‑Time/Offline Data Platform for 5G Connected Factories
DataFunTalk
DataFunTalk
Jun 1, 2024 · Big Data

Ant Group's Real-Time Data Warehouse Architecture, Solutions, and Data Lake Outlook

This article presents Ant Group's recent explorations and practices in real-time data warehousing, covering the system architecture, streaming data quality assurance, flow‑batch integrated applications, and future data lake integration, while sharing technical details and operational insights for large‑scale data processing.

FlinkReal-time Datadata warehouse
0 likes · 16 min read
Ant Group's Real-Time Data Warehouse Architecture, Solutions, and Data Lake Outlook
Data Thinking Notes
Data Thinking Notes
May 30, 2024 · Databases

Why Your Data Team Is Drowning in Requests—and How OLAP Can Save You

This article examines why data departments get overwhelmed by massive data‑retrieval requests, identifies root causes such as mindset, requirement handling, and lack of tools, and presents a technical solution centered on dimensional modeling and OLAP multi‑dimensional reporting to streamline data access and empower teams.

Data ModelingOLAPbig data
0 likes · 12 min read
Why Your Data Team Is Drowning in Requests—and How OLAP Can Save You
DataFunTalk
DataFunTalk
May 28, 2024 · Big Data

Building and Managing a Metric System in Data Warehouse: Practices from Dongchedi

This article details how the Dongchedi business team designs, implements, and monitors a comprehensive metric system within its data warehouse, covering metric standards, model construction, metadata management, quality monitoring, application scenarios, and future directions using the DataLeap platform.

Data ModelingMetric Systembig data
0 likes · 18 min read
Building and Managing a Metric System in Data Warehouse: Practices from Dongchedi
Alibaba Cloud Developer
Alibaba Cloud Developer
May 27, 2024 · Big Data

How MaxCompute’s New Offline‑Near‑Real‑Time Architecture Revolutionizes Big Data Workloads

This article explains how MaxCompute’s integrated offline‑and‑near‑real‑time architecture, built on Delta Table, solves complex big‑data scenarios by providing unified storage, ACID transactions, upsert, time‑travel, automatic data‑file governance and low‑latency query capabilities while reducing cost and operational complexity.

Delta TableMaxComputeNear Real-Time
0 likes · 27 min read
How MaxCompute’s New Offline‑Near‑Real‑Time Architecture Revolutionizes Big Data Workloads
Alibaba Cloud Big Data AI Platform
Alibaba Cloud Big Data AI Platform
May 9, 2024 · Big Data

How RoaringBitmap Supercharged Lazada’s Selection Platform and Cut Processing Time by 99%

This article explains how Lazada’s internal selection platform leveraged Hologres and the RoaringBitmap compression algorithm to dramatically reduce storage costs, accelerate set operations, and break the 200,000‑item pool limit, achieving up to a 99% speed improvement in scheduling.

Bitmap CompressionHologresPerformance Optimization
0 likes · 16 min read
How RoaringBitmap Supercharged Lazada’s Selection Platform and Cut Processing Time by 99%
Past Memory Big Data
Past Memory Big Data
Apr 19, 2024 · Databases

How ByteHouse Achieves Hundred‑Fold OLAP Performance Gains

ByteHouse, a cloud‑native data warehouse built on ClickHouse, redesigns storage‑compute separation, introduces a new MPP architecture, rule‑based and cost‑based optimizers, exchange runtime filters, and parallelism techniques, delivering 10‑200× faster query performance on TPC‑DS, TPC‑H and SSB benchmarks and boosting point‑lookup QPS to 32,000.

BenchmarkByteHouseOLAP
0 likes · 19 min read
How ByteHouse Achieves Hundred‑Fold OLAP Performance Gains
DataFunTalk
DataFunTalk
Apr 15, 2024 · Databases

ByteHouse Cloud‑Native Data Warehouse Performance Whitepaper: Architecture, Optimizations, and Benchmark Results

The ByteHouse performance whitepaper details the cloud‑native data warehouse’s architecture, rule‑based and cost‑based optimizer enhancements, exchange runtime, runtime filters, parallelism and wide‑table optimizations, and presents benchmark comparisons on TPC‑DS, TPC‑H and SSB datasets demonstrating orders‑of‑magnitude query speed improvements.

BenchmarkByteHouseOLAP
0 likes · 17 min read
ByteHouse Cloud‑Native Data Warehouse Performance Whitepaper: Architecture, Optimizations, and Benchmark Results
DataFunTalk
DataFunTalk
Apr 14, 2024 · Big Data

Third‑Generation Metric Platform: Enabling a Light Data Warehouse with NoETL

This article explains how a third‑generation metric platform replaces traditional ETL‑heavy data‑warehouse pipelines with a semantic‑driven NoETL approach, reducing cost, improving quality and efficiency, and delivering automated, self‑service analytics for both IT and business users.

Metric PlatformNoETLbig data
0 likes · 16 min read
Third‑Generation Metric Platform: Enabling a Light Data Warehouse with NoETL
DataFunTalk
DataFunTalk
Apr 12, 2024 · Big Data

Building and Managing an Indicator System in a Data Warehouse: Practices from the Dongchedi Business

This article explains how the Dongchedi team designed, implemented, and monitored a comprehensive indicator system within a petabyte‑scale data warehouse, covering standards, metadata management, model construction, quality monitoring, and diverse application scenarios to improve data reliability and business insight.

Indicator ManagementMetric Systembig data
0 likes · 18 min read
Building and Managing an Indicator System in a Data Warehouse: Practices from the Dongchedi Business
Didi Tech
Didi Tech
Mar 28, 2024 · Big Data

How We Unified Real‑Time and Batch Features with StarRocks in Financial Risk Control

This article analyzes the challenges of building real‑time and batch risk‑control features, compares Lambda and Kappa architectures, evaluates storage‑unified and compute‑unified solutions, and details how StarRocks was selected, validated, and deployed to achieve high‑performance, low‑latency feature serving in a financial context.

Real-time AnalyticsStarRocksbig data
0 likes · 19 min read
How We Unified Real‑Time and Batch Features with StarRocks in Financial Risk Control
DataFunSummit
DataFunSummit
Mar 14, 2024 · Big Data

Tencent Game Data Analysis: Lakehouse Integration Practice

This article presents Tencent Game's comprehensive lakehouse integration practice, detailing the project background, storage‑compute separation, data layering, unified DDL/DML operations, performance optimizations, and future plans, illustrating how StarRocks, Iceberg, and Spark are combined to achieve scalable, cost‑effective analytics for massive game data.

Compute-Storage SeparationLakehousePerformance Optimization
0 likes · 16 min read
Tencent Game Data Analysis: Lakehouse Integration Practice
Xiaohongshu Tech REDtech
Xiaohongshu Tech REDtech
Mar 4, 2024 · Big Data

Integrating Data Lake Technologies with Data Warehouse Architecture at Xiaohongshu: Practices and Performance Optimizations

Xiaohongshu’s data‑warehouse team integrated Apache Iceberg‑based data‑lake techniques into its existing warehouse, replacing the legacy Hive/Spark stack with global sorting, Z‑order, and upsert‑enabled tables, which cut query latency by up to 90 %, boosted data freshness by 50 %, slashed storage costs by 83 % and saved tens of thousands of GB‑hours of compute daily.

Apache IcebergData LakeFlink
0 likes · 19 min read
Integrating Data Lake Technologies with Data Warehouse Architecture at Xiaohongshu: Practices and Performance Optimizations
Volcano Engine Developer Services
Volcano Engine Developer Services
Feb 29, 2024 · Big Data

How MetaApp Cut Data Warehouse Costs by 50% with ByConity

MetaApp replaced ClickHouse with the open‑source cloud‑native data warehouse ByConity, achieving over 50% cost reduction and faster, more stable OLAP queries by separating storage and compute, simplifying scaling, and improving resource utilization across a range of analytics workloads such as deduplication, retention, conversion and point‑lookup.

ByConityClickHouseOLAP
0 likes · 13 min read
How MetaApp Cut Data Warehouse Costs by 50% with ByConity
Baidu Geek Talk
Baidu Geek Talk
Feb 28, 2024 · Big Data

How Baidu’s Fusion Compute Engine Cuts Query Time to Seconds on Petabyte‑Scale Data

This article analyzes Baidu's fusion compute engine for its data warehouse, detailing its architecture, optimization techniques such as data skipping, Parquet column indexing, ProjectLimit and CodeGen, and demonstrates how these innovations reduce query latency to seconds while cutting storage costs by about 30% on multi‑petabyte workloads.

BaiduFusion Compute EnginePerformance Optimization
0 likes · 12 min read
How Baidu’s Fusion Compute Engine Cuts Query Time to Seconds on Petabyte‑Scale Data
DataFunSummit
DataFunSummit
Feb 19, 2024 · Big Data

Yipay Data Warehouse Construction and Data Governance Practices

This presentation by senior data warehouse engineer Huang Luo details Yipay's end‑to‑end data warehouse build, covering background challenges, governance framework, platform development, layered architecture, naming standards, monitoring, and future plans, offering practical insights for data engineers, architects, and business stakeholders.

Data Qualitybig datadata architecture
0 likes · 14 min read
Yipay Data Warehouse Construction and Data Governance Practices
DataFunSummit
DataFunSummit
Feb 7, 2024 · Big Data

Evolution of OLAP with Apache Doris at Xingyun Retail Credit

Facing rapid data growth, Xingyun Retail Credit transitioned from traditional OLTP systems to an Apache Doris‑based OLAP solution, detailing the data demand generation, OLAP engine selection challenges, multi‑stage implementation, performance gains, data‑warehouse construction, and future roadmap for scalable analytics.

Apache DorisOLAPPerformance Optimization
0 likes · 17 min read
Evolution of OLAP with Apache Doris at Xingyun Retail Credit
DataFunSummit
DataFunSummit
Jan 25, 2024 · Big Data

Best Practices of Jushuitan Cloud‑Native OLAP Architecture and Logistics Warning

This article presents Jushuitan's cloud‑native OLAP architecture, covering business background, data‑warehouse evolution, real‑time processing with Flink, Hologres, and Aerospike, and detailed logistics‑warning use cases, followed by technical challenges, future outlook, and a Q&A on implementation details.

FlinkLogistics WarningOLAP
0 likes · 20 min read
Best Practices of Jushuitan Cloud‑Native OLAP Architecture and Logistics Warning
DataFunTalk
DataFunTalk
Jan 3, 2024 · Databases

ClickHouse 2024 Core New Features and Product Development Directions

This article introduces ClickHouse, an open‑source columnar OLAP database, outlines its architecture, advantages, self‑hosted and cloud deployment models, highlights recent product features such as async inserts, JSON support, Parquet acceleration, query caching, and summarizes a Q&A covering semi‑structured data, MPP, virtual columns, and future roadmap.

ClickHouseColumnar DatabaseOLAP
0 likes · 12 min read
ClickHouse 2024 Core New Features and Product Development Directions
Architects Research Society
Architects Research Society
Jan 2, 2024 · Big Data

Understanding Data Lakes: Concepts, Benefits, Challenges, and Comparison with Data Warehouses

This article explains what a data lake is, its origins, key characteristics such as collecting all data, enabling diverse user access, and flexible processing, compares it with traditional data warehouses, discusses cost advantages, potential pitfalls like data swamps, and outlines best‑practice considerations for enterprise adoption.

AnalyticsData LakeHadoop
0 likes · 10 min read
Understanding Data Lakes: Concepts, Benefits, Challenges, and Comparison with Data Warehouses
Weimob Technology Center
Weimob Technology Center
Jan 2, 2024 · Big Data

How to Efficiently Test BI Reports in a Hive‑StarRocks Data Warehouse

This article details practical methods for testing BI reports built on Hive and StarRocks, covering the report creation workflow, testing characteristics, SQL writing techniques, impact analysis, data warehouse simplification, and the application of data quality tools to ensure accurate and efficient reporting.

BI testingData QualityHive
0 likes · 9 min read
How to Efficiently Test BI Reports in a Hive‑StarRocks Data Warehouse
DataFunTalk
DataFunTalk
Jan 1, 2024 · Big Data

MaxCompute Semi-Structured Data: Concepts, Solutions, and Benefits

This article explains the nature of semi‑structured data, compares traditional schema‑on‑read and schema‑on‑write approaches, and details MaxCompute's columnar storage solution that balances flexibility, performance, and cost for large‑scale data warehouses.

MaxComputeSemi-structured Databig data
0 likes · 19 min read
MaxCompute Semi-Structured Data: Concepts, Solutions, and Benefits
Data Thinking Notes
Data Thinking Notes
Dec 28, 2023 · Big Data

How Xiaomi Built a Scalable Metric System: Best Practices and Methodology

This article explains Xiaomi's end‑to‑end metric system construction, covering the definition of metrics, business pain points, the OSM (Object‑Strategy‑Measure) model, MECE principle, model design guidelines, data‑warehouse implementation, metric management, and the resulting data‑driven workflow across the company.

MECE principleMetric SystemOSM model
0 likes · 10 min read
How Xiaomi Built a Scalable Metric System: Best Practices and Methodology
Zhuanzhuan Tech
Zhuanzhuan Tech
Dec 27, 2023 · Big Data

Implementing Self-Service OLAP Analytics with Quick BI and StarRocks: Architecture, Optimizations, and Lessons Learned

This article presents a comprehensive case study of building a self‑service OLAP analytics platform at ZhaiZhai using Quick BI and StarRocks, covering background motivations, technical architecture, implementation details, performance‑optimizing case studies, and the resulting business impact.

OLAPPerformance OptimizationQuick BI
0 likes · 16 min read
Implementing Self-Service OLAP Analytics with Quick BI and StarRocks: Architecture, Optimizations, and Lessons Learned
DataFunSummit
DataFunSummit
Dec 7, 2023 · Databases

Apache Doris: A High‑Performance Real‑Time Analytical Database for Online High‑Concurrency Reporting

This article introduces Apache Doris, a real‑time analytical database built on an MPP architecture, explains its suitability for massive data workloads and online high‑concurrency reporting scenarios, and details the core technologies—storage models, vectorized query engine, materialized views, partitioning, indexing, row‑store and prepared statements—that enable sub‑second query latency and high QPS, while also showing a real‑world case study and how to join the Doris community.

Apache DorisMaterialized ViewsReal-time Analytics
0 likes · 13 min read
Apache Doris: A High‑Performance Real‑Time Analytical Database for Online High‑Concurrency Reporting
DataFunTalk
DataFunTalk
Nov 28, 2023 · Big Data

Xiaomi Metric System Construction and Management Best Practices

This article presents Xiaomi's comprehensive metric system framework, covering its definition, business pain points, the OSM and MECE methodologies, model design principles, data warehouse construction, metric management, and future outlook, illustrating how a unified data platform drives efficient business decision‑making.

Business IntelligenceMECEMetric System
0 likes · 10 min read
Xiaomi Metric System Construction and Management Best Practices
Architects Research Society
Architects Research Society
Nov 26, 2023 · Big Data

Data Lake vs Data Warehouse: Key Differences and How to Choose

Data lakes and data warehouses serve different purposes in big‑data architectures; this article explains their definitions, core attributes, five major distinctions—including data retention, type support, user coverage, adaptability, and insight speed—and offers guidance on selecting or combining the two approaches.

AnalyticsData LakeHadoop
0 likes · 12 min read
Data Lake vs Data Warehouse: Key Differences and How to Choose
Ctrip Technology
Ctrip Technology
Nov 23, 2023 · Big Data

Optimizing Data Warehouse Timeliness Using Metadata Lineage

This article presents a metadata‑driven approach to improve data warehouse timeliness by extracting upstream lineage, identifying over‑layered, duplicate, and critical‑path tasks, and applying targeted scheduling and code‑level optimizations, demonstrated with a hotel order wide‑table case study.

DAGData PipelineLineage
0 likes · 7 min read
Optimizing Data Warehouse Timeliness Using Metadata Lineage
DataFunSummit
DataFunSummit
Nov 22, 2023 · Big Data

Bilibili Data Quality Assurance System: Architecture, Practices, and Case Study

This article presents Bilibili's data quality assurance system, detailing its evolution across four stages, the architectural framework, core capabilities such as a quality data warehouse, monitoring, collaborative safeguards, digital-driven optimization, and efficient incident handling, along with practical case studies and future outlooks.

Data Qualitybig datadata warehouse
0 likes · 22 min read
Bilibili Data Quality Assurance System: Architecture, Practices, and Case Study
HomeTech
HomeTech
Nov 15, 2023 · Industry Insights

How to Build Accurate Data Asset Lineage for Data Warehouse Governance

This article explains the challenges of data asset lineage in large data warehouses, presents a comprehensive approach using business‑level instrumentation, SQL interceptor plugins, and ETL script parsing to generate fine‑grained lineage graphs, and demonstrates measurable improvements in coverage and zombie‑table cleanup.

Data QualityETL parsingSQL interceptor
0 likes · 18 min read
How to Build Accurate Data Asset Lineage for Data Warehouse Governance
Data Thinking Notes
Data Thinking Notes
Nov 9, 2023 · Big Data

How to Build a Scalable Data Governance System for Massive E‑Commerce Warehouses

This article outlines the challenges of ultra‑large e‑commerce data warehouses—such as SLA pressure, model instability, soaring resource costs, low governance efficiency, and fragmented processes—and presents a one‑stop, tiered data‑governance framework with stability, cost, and efficiency subsystems that drives distributed autonomous governance and measurable business value.

automationbig datacost optimization
0 likes · 11 min read
How to Build a Scalable Data Governance System for Massive E‑Commerce Warehouses
Volcano Engine Developer Services
Volcano Engine Developer Services
Nov 9, 2023 · Databases

How ByteHouse Redefines ELT for Cloud‑Native Data Warehousing

This article explains how ByteHouse, a cloud‑native data warehouse, shifts traditional ETL to ELT, simplifies data pipelines, enhances scalability, and introduces advanced features such as stage‑by‑stage scheduling, adaptive resource management, async execution, and future roadmap for big‑data workloads.

ByteHouseELTSQL optimization
0 likes · 16 min read
How ByteHouse Redefines ELT for Cloud‑Native Data Warehousing
dbaplus Community
dbaplus Community
Nov 8, 2023 · Big Data

Choosing Between Data Warehouse, Data Lake, and Lakehouse: When to Use Each

This article compares traditional data warehouses, modern data lakes, and emerging lakehouse architectures, explaining their design patterns, advantages, disadvantages, and suitable use cases, while detailing implementation considerations such as schema design, ETL/ELT processes, file formats like Delta, Iceberg, and Hudi, and factors influencing platform selection.

Apache SparkData LakeDelta Lake
0 likes · 20 min read
Choosing Between Data Warehouse, Data Lake, and Lakehouse: When to Use Each
Data Thinking Notes
Data Thinking Notes
Nov 2, 2023 · Operations

How Bilibili Built a Scalable Data Quality Assurance System for Its Data Warehouse

This article details Bilibili's data quality assurance framework, covering its evolution across four data platform stages, the architecture of its quality data warehouse, core capabilities such as a complete assurance system, digital‑driven continuous optimization, and efficient incident handling, plus case studies, future plans, and a Q&A session.

BilibiliData Qualitybig data
0 likes · 27 min read
How Bilibili Built a Scalable Data Quality Assurance System for Its Data Warehouse
DataFunTalk
DataFunTalk
Oct 25, 2023 · Databases

Apache Doris Summit Asia 2023: Highlights, Innovations, and Industry Use Cases

The Apache Doris Summit Asia 2023 showcased the milestone 2.0 release, impressive performance gains, rapid community growth, and diverse industry deployments, while outlining future cloud‑native and unified analytics directions that position Doris as a leading real‑time data warehouse solution.

Apache DorisReal-time AnalyticsSelectDB
0 likes · 13 min read
Apache Doris Summit Asia 2023: Highlights, Innovations, and Industry Use Cases
DataFunTalk
DataFunTalk
Oct 23, 2023 · Big Data

Alibaba Cloud DataWorks Intelligent Data Modeling: Practices, Challenges, and Solutions

This article introduces Alibaba Cloud DataWorks' intelligent data modeling tool, outlines the data demand flow, shares best practices and hands‑on demonstrations for data warehouse modeling, discusses common challenges and their solutions, and provides Q&A and product details for developers and data engineers.

Alibaba CloudData ModelingDataWorks
0 likes · 12 min read
Alibaba Cloud DataWorks Intelligent Data Modeling: Practices, Challenges, and Solutions
Big Data Technology & Architecture
Big Data Technology & Architecture
Oct 23, 2023 · Big Data

Bilibili Data Quality Assurance: Architecture, Goals, Core Capabilities, and Future Outlook

This article outlines Bilibili's data quality assurance framework, detailing its evolution across four development stages, the current data platform architecture, identified pain points, four key quality objectives, core capabilities such as a quality data warehouse, comprehensive monitoring, digital optimization, fault handling, and future directions.

Data Qualitybig datadata governance
0 likes · 22 min read
Bilibili Data Quality Assurance: Architecture, Goals, Core Capabilities, and Future Outlook
DataFunSummit
DataFunSummit
Oct 22, 2023 · Big Data

How Kuaishou E‑commerce Leverages OLAP and a Unified Data Architecture to Solve Business Data Challenges

This article explains how Kuaishou's e‑commerce team built a unified OLAP‑based data platform—covering data ingestion, consistent dimensional and fact layers, metric management, and real‑time services—to address rapid growth, metric inconsistency, and operational inefficiencies across multiple business scenarios.

E‑commerceOLAPbig data
0 likes · 20 min read
How Kuaishou E‑commerce Leverages OLAP and a Unified Data Architecture to Solve Business Data Challenges
DataFunTalk
DataFunTalk
Oct 22, 2023 · Operations

Bilibili Data Quality Assurance System: Architecture, Practices, and Case Study

This article presents Bilibili's data quality assurance system, detailing its evolution across four data platform stages, the multi‑layer architecture, core capabilities such as a quality data warehouse, digital‑driven continuous optimization, and efficient incident handling, and concludes with a real‑world case study and future outlook.

big datadata warehousequality assurance
0 likes · 21 min read
Bilibili Data Quality Assurance System: Architecture, Practices, and Case Study
DataFunSummit
DataFunSummit
Oct 16, 2023 · Big Data

Elegant Dimensional Modeling and Multi‑Dimensional Analysis Design Practice

In this presentation, Qiu Shengchang shares his 13‑year experience designing elegant data‑warehouse architectures, detailing a highly generic dimensional model, extreme partitioned tables, and a universal multi‑dimensional analysis framework that enables rapid, comprehensive reporting on massive datasets.

Multi-dimensional Analysisbig datadata warehouse
0 likes · 3 min read
Elegant Dimensional Modeling and Multi‑Dimensional Analysis Design Practice
Past Memory Big Data
Past Memory Big Data
Oct 10, 2023 · Big Data

2023 Big Data Interview Guide: Hadoop, Hive, Doris, Data Warehouse Essentials

This comprehensive 2023 guide covers essential big‑data interview topics, providing detailed explanations and step‑by‑step processes for Hadoop HDFS read/write, YARN, Hive table types and optimizations, Doris architecture and data models, data‑warehouse layers, modeling techniques, quality monitoring, and classic algorithm design questions such as TOP‑K and duplicate detection.

DorisHadoopHive
0 likes · 54 min read
2023 Big Data Interview Guide: Hadoop, Hive, Doris, Data Warehouse Essentials
Big Data Technology & Architecture
Big Data Technology & Architecture
Sep 18, 2023 · Big Data

Unified Real‑Time and Batch Data Warehouse Architecture with Hudi Lakehouse

The article explains the mainstream Lambda data‑warehouse architecture, its benefits and challenges, then introduces Hudi as a lake‑house solution that unifies real‑time and offline storage, describes the multi‑layer service design, and showcases three practical scenarios—stream processing, real‑time multidimensional analysis, and stream‑batch data reuse—demonstrating how the integrated architecture improves latency, cost, and operational complexity.

Batch ProcessingHudiLakehouse
0 likes · 13 min read
Unified Real‑Time and Batch Data Warehouse Architecture with Hudi Lakehouse
dbaplus Community
dbaplus Community
Sep 10, 2023 · Big Data

Master Data Warehouse Architecture: Layers, Naming Rules, and Lifecycle Tips

This guide outlines a comprehensive data model architecture for data warehouses, detailing layer definitions (ODS, CDM, DWD, DWS, ADS), naming conventions, data type standards, partitioning strategies, redundancy rules, and lifecycle management policies to ensure consistency, performance, and maintainability across big‑data environments.

Partitioningdata warehousenaming conventions
0 likes · 28 min read
Master Data Warehouse Architecture: Layers, Naming Rules, and Lifecycle Tips
DataFunSummit
DataFunSummit
Sep 8, 2023 · Big Data

Tianqiong OLAP Real‑time Lakehouse Fusion Platform Architecture Practice

This article explains why lake‑warehouse fusion is needed, describes the challenges of integrating real‑time data warehouses with data lakes, introduces a new StarRocks‑based architecture that supports real‑time ingestion, cooling, offline loading, and adaptive hot‑cold query rewriting, and outlines future plans and Q&A.

Data IntegrationLakehouseReal-time OLAP
0 likes · 21 min read
Tianqiong OLAP Real‑time Lakehouse Fusion Platform Architecture Practice
DataFunTalk
DataFunTalk
Sep 6, 2023 · Databases

Large Model + OLAP: Enabling a New Data Service Platform

This article details how Tencent Music combines large language models with an Apache Doris‑based OLAP engine, introduces a semantic layer, manual‑experience routing, schema mapping and plugin integration, and outlines the evolution of its data architecture through four versions to achieve real‑time, cost‑effective, and scalable intelligent data services.

Apache DorisOLAPSemantic Layer
0 likes · 24 min read
Large Model + OLAP: Enabling a New Data Service Platform
JD Retail Technology
JD Retail Technology
Sep 4, 2023 · Big Data

JD Mini Program Data Center: Architecture, Milestones, and Real‑time Analytics Solutions

The article details the JD Mini Program platform, its data‑center development milestones, comprehensive business panorama, technical architecture, data collection, storage, and analysis pipelines—including Flink‑based real‑time monitoring, ClickHouse custom analytics, and Elasticsearch user‑behavior insights—while outlining current challenges and future AI‑driven enhancements.

ClickHouseElasticsearchFlink
0 likes · 16 min read
JD Mini Program Data Center: Architecture, Milestones, and Real‑time Analytics Solutions
DataFunTalk
DataFunTalk
Sep 4, 2023 · Big Data

Unified Batch‑Stream Storage with Hudi and LAS: Architecture, Design, and Deployment

This article presents a comprehensive overview of a batch‑stream unified storage solution built on Hudi and the Lakehouse Analysis Service (LAS), covering background challenges, architectural design, data organization, read/write mechanisms, BTS architecture, real‑world deployment scenarios, and future development plans.

Batch-StreamHudiLAS
0 likes · 22 min read
Unified Batch‑Stream Storage with Hudi and LAS: Architecture, Design, and Deployment
DataFunTalk
DataFunTalk
Sep 3, 2023 · Big Data

Evolution of OLAP at Xingyun Retail Credit Using Apache Doris

This article details how Xingyun Retail Credit transitioned from traditional data warehouses to an Apache Doris‑based OLAP solution, covering data demand generation, OLAP engine selection challenges, multi‑stage implementation, performance optimizations, data‑warehouse construction, real‑world use cases, and future roadmap.

Apache DorisETLOLAP
0 likes · 16 min read
Evolution of OLAP at Xingyun Retail Credit Using Apache Doris
Didi Tech
Didi Tech
Aug 31, 2023 · Big Data

Data Stability Construction and Fault Governance Practices at Didi Customer Service

Didi’s multi‑year data‑stability program for its customer‑service platform progressed through fault‑centered engineering, business‑aligned cross‑team work, and capability normalization, instituting pre‑, mid‑ and post‑fault safeguards, clear ownership, automated alerts and repair tools, which cut fault count by 42 % and more than doubled mean‑time‑to‑repair while boosting team communication and satisfaction.

Data ReliabilityODSautomation
0 likes · 16 min read
Data Stability Construction and Fault Governance Practices at Didi Customer Service
DataFunTalk
DataFunTalk
Aug 29, 2023 · Big Data

MaxCompute Incremental Update, Processing Architecture, and Intelligent Data Warehouse Optimizations

This article presents a comprehensive overview of MaxCompute's incremental update and processing architecture, the design of intelligent materialized views, and the engine's adaptive execution optimizations, detailing the integrated near‑real‑time and batch pipelines, transactional table 2.0, and practical Q&A.

MaxComputebig datadata warehouse
0 likes · 21 min read
MaxCompute Incremental Update, Processing Architecture, and Intelligent Data Warehouse Optimizations
DataFunTalk
DataFunTalk
Aug 28, 2023 · Big Data

Practical Experience of an E‑commerce Platform’s Offline and Real‑time Data Warehouse

This article shares the practical architecture, technology selection, implementation details, and evolution of an e‑commerce platform’s offline and real‑time data warehouses, covering data modeling, processing pipelines, system components such as Hive, Spark, Flink, ClickHouse, Doris, and Hudi, and the lessons learned from multiple production deployments.

ClickHouseDorisE‑commerce
0 likes · 18 min read
Practical Experience of an E‑commerce Platform’s Offline and Real‑time Data Warehouse
Huawei Cloud Developer Alliance
Huawei Cloud Developer Alliance
Aug 25, 2023 · Databases

Unlock GaussDB(DWS) Performance: Expert Tips for Resource Management

This announcement introduces Huawei Cloud’s DTT live session on August 29, where expert Lv Pengbo will explain GaussDB(DWS) resource control principles and demonstrate practical techniques—such as CPU usage analysis, memory tuning, and queue issue resolution—to help developers efficiently manage data‑warehouse resources and boost performance.

GaussDBHuawei CloudSQL Performance
0 likes · 3 min read
Unlock GaussDB(DWS) Performance: Expert Tips for Resource Management
Tencent Cloud Developer
Tencent Cloud Developer
Aug 23, 2023 · Big Data

WeChat Experiment Platform: Architecture Design and Iceberg Lakehouse Optimization

The WeChat Experiment Platform migrated its 60,000 metric, 200,000 core, 30 PB plus data pipeline to an Iceberg based lakehouse, leveraging three layer metadata, fine grained partitioning, MERGE into writes, time travel snapshots and skew handling UDFs, which cut core time by 69%, saved ~100 PB storage, and reduced latency by up to 70%.

LakehouseMerge IntoMetric Computation
0 likes · 18 min read
WeChat Experiment Platform: Architecture Design and Iceberg Lakehouse Optimization
DataFunTalk
DataFunTalk
Aug 14, 2023 · Big Data

Data Warehouse Modeling Platform: Exploration and Practice at NetEase Yanxuan

This article details NetEase Yanxuan’s exploration and practice of a data warehouse modeling platform, covering background, current challenges, a comprehensive solution, step‑by‑step implementation, and the resulting improvements in model standardization, automation, and business value.

Platformautomationbig data
0 likes · 18 min read
Data Warehouse Modeling Platform: Exploration and Practice at NetEase Yanxuan
DataFunTalk
DataFunTalk
Jul 25, 2023 · Databases

Building an Integrated Metric Data Service Platform with Apache Doris: Architecture Evolution and Millisecond‑Level Query Performance

This article describes how Financial One Account, a technology service arm of Ping An, migrated from a Hadoop‑Presto‑Kylin stack to an Apache Doris‑based data platform, detailing the architectural evolution, OLAP engine selection, metric system design, performance optimizations, and future roadmap for real‑time analytics.

Apache DorisOLAPbig data
0 likes · 15 min read
Building an Integrated Metric Data Service Platform with Apache Doris: Architecture Evolution and Millisecond‑Level Query Performance