big data platform

Bilibili Tech

Jul 19, 2024 · Big Data

Bilibili's One-Stop Big Data Cluster Management Platform (BMR) - Architecture and Implementation

Bilibili’s one‑stop Big Data Cluster Management Platform (BMR) consolidates HDFS, Spark, Flink, ClickHouse, Kafka and other services into a unified system that evolved through four stages—standardization, metadata‑driven construction, containerization, and observability—addressing node consistency, scaling, fault self‑healing, and resource optimization while delivering elastic scaling, automated start/stop, and future cost‑saving and stability enhancements.

Cluster ManagementContainerizationbig data platform

0 likes · 12 min read

Bilibili's One-Stop Big Data Cluster Management Platform (BMR) - Architecture and Implementation

DataFunTalk

Jan 22, 2024 · Big Data

Design and Implementation of Bilibili's Big Data Development Governance Platform

This article details Bilibili's five‑year development of a comprehensive big‑data governance platform, covering its usage scenarios, product positioning, data map and governance solutions, abstract configuration approach, operational mechanisms, and future plans, highlighting significant improvements in data efficiency and value assessment.

Bilibilibig data platformdata governance

0 likes · 10 min read

Design and Implementation of Bilibili's Big Data Development Governance Platform

DataFunTalk

Aug 10, 2023 · Big Data

iQIYI Magic Mirror: Evolution of a Big Data Analysis Platform

The article details how iQIYI's Magic Mirror platform evolved from a simple single‑table reporting tool to a multi‑engine, self‑service big data analysis system that improves data access speed, reduces operational costs, and supports comprehensive business analytics across the company.

Data EngineeringData VisualizationMagic Mirror

0 likes · 17 min read

iQIYI Magic Mirror: Evolution of a Big Data Analysis Platform

DataFunSummit

Jul 12, 2023 · Big Data

Data Development Production Environment Isolation: Xiaomi's Experience, Technical Choices, and Implementation

This article explains Xiaomi's approach to isolating production environments for data development, covering the evolution of its data platform, the trade‑offs between physical and logical isolation, the productized workflow and security measures, and real‑world outcomes from the deployment.

Data IsolationData Securitybig data platform

0 likes · 18 min read

Data Development Production Environment Isolation: Xiaomi's Experience, Technical Choices, and Implementation

DataFunSummit

Nov 11, 2022 · Big Data

Tencent Oula Data Governance Platform: Architecture, Practices, and Solutions

The article presents an in‑depth overview of Tencent's Oula data governance platform, describing its construction goals, core capabilities, DataOps‑driven development workflow, unified metric store, data map services, and practical Q&A on asset health scoring and data lineage, illustrating a comprehensive end‑to‑end big‑data governance solution.

Data ModelingDataOpsTencent

0 likes · 17 min read

Tencent Oula Data Governance Platform: Architecture, Practices, and Solutions

Baidu Geek Talk

Jul 1, 2022 · Big Data

Evolution of Data Platform Technology: From Data Warehouse to Lakehouse Architecture

The article traces the evolution of data platforms from early data warehouses—using schema‑on‑write, columnar storage, and MPP engines—to data lakes that retain raw data with schema‑on‑read, and finally to lakehouse architectures that merge storage and compute, offering unified metadata, versioning, and support for BI, big‑data, AI, and HPC workloads.

Data WarehouseLakehouseMPP

0 likes · 25 min read

Evolution of Data Platform Technology: From Data Warehouse to Lakehouse Architecture

DataFunSummit

May 4, 2022 · Big Data

NetEase Big Data Platform: HDFS Optimization and Practices

NetEase’s senior big‑data engineer shares how the company’s large‑scale data platform leverages Hadoop, HDFS, YARN and related technologies, detailing multi‑layer architecture, cross‑cloud deployment, storage optimizations, NameNode performance enhancements, RPC prioritization, and practical lessons from operating petabyte‑scale clusters.

Cluster OptimizationHDFSStorage Management

0 likes · 23 min read

NetEase Big Data Platform: HDFS Optimization and Practices

Beike Product & Technology

Oct 14, 2020 · Artificial Intelligence

AI Engineering Architecture Practice Conference

This article introduces a technical conference on AI engineering architecture practices hosted by Beike's AI Technology Center, featuring expert speakers and detailed agenda covering big data platforms, OLAP systems, DMP platforms, recommendation systems, and commercialization algorithms.

AI conferenceData AnalyticsEngineering Architecture

0 likes · 4 min read

AI Engineering Architecture Practice Conference

DataFunTalk

Aug 1, 2019 · Big Data

Streaming Data Platform Practices and Challenges at Beike Real Estate

This article presents an in‑depth overview of Beike's four‑layer streaming data platform, covering the foundational infrastructure, capability aggregation, data content, and output layers, as well as the challenges of metadata management, real‑time processing, and productization through the Ark and Tianyan systems.

Ark platformBeikeTianyan

0 likes · 14 min read

Streaming Data Platform Practices and Challenges at Beike Real Estate