BestHub
Sign in
Tagged articles

data platform

386 articles · Page 2 of 4
Data Thinking Notes
Data Thinking Notes
Apr 9, 2024 · Big Data

What Is a Data Middle Platform and Why It’s Essential for Modern Enterprises

Data middle platforms transform raw enterprise data into reusable assets by integrating collection, storage, processing, governance, and service layers, enabling faster deployment, consistent metrics, improved data quality, and business value across digital transformation, while addressing challenges like siloed data, low efficiency, and inconsistent standards.

Big DataData GovernanceData Integration
0 likes · 23 min read
What Is a Data Middle Platform and Why It’s Essential for Modern Enterprises
DataFunSummit
DataFunSummit
Apr 4, 2024 · Big Data

Design Principles and Future Directions of DataOps

This article outlines the core capabilities of data-driven development, the background and architecture of DataOps, its research challenges and focus areas, and explores future directions such as data virtualization, platform governance, and data value assessment, providing a comprehensive overview of DataOps practices.

Big Datadata platform
0 likes · 8 min read
Design Principles and Future Directions of DataOps
DataFunSummit
DataFunSummit
Apr 1, 2024 · Big Data

DataOps at ByteDance: Challenges, Implementation, and Future Outlook

This article examines ByteDance's DataOps journey, detailing the data‑engineering challenges faced, the concrete solutions and productization through the DataLeap platform, the metrics and best‑practice framework adopted, and the future directions involving AI‑assisted development and open‑source collaboration.

Big Datadata platformmetrics
0 likes · 20 min read
DataOps at ByteDance: Challenges, Implementation, and Future Outlook
MaGe Linux Operations
MaGe Linux Operations
Mar 8, 2024 · Cloud Computing

Choosing Between Cloud, On-Premises, and Cloud‑Near Storage: Which Wins?

This article compares the advantages and disadvantages of cloud storage, on‑premises storage, and the hybrid cloud‑near storage model, explaining how each impacts scalability, cost, control, security, and integration for modern data platforms, and helps organizations select the most suitable solution.

Cloud StorageHybrid storagecost optimization
0 likes · 12 min read
Choosing Between Cloud, On-Premises, and Cloud‑Near Storage: Which Wins?
Baidu Tech Salon
Baidu Tech Salon
Mar 7, 2024 · Artificial Intelligence

How Generative AI is Transforming Business Intelligence: Inside Baidu’s ChatBI

This article examines the evolution of BI through generative AI, outlines the design and implementation of Baidu’s ChatBI platform, and discusses technical challenges such as NL2SQL integration, performance, accuracy, and user experience improvements that enable intelligent, low‑cost data analysis.

AIBusiness IntelligenceChatBI
0 likes · 14 min read
How Generative AI is Transforming Business Intelligence: Inside Baidu’s ChatBI
DataFunTalk
DataFunTalk
Mar 5, 2024 · Big Data

Changan Automotive Big Data Platform: Challenges and Practices in Connected Vehicle Scenarios

This article outlines the rapid growth of data in the smart automotive sector and details Changan's big data platform challenges—high cost, data accessibility, and operational complexity—and the practical migration from a Lambda to a unified Kappa architecture that delivers significant storage, compute, and maintenance efficiencies.

Big DataConnected VehiclesKappa Architecture
0 likes · 14 min read
Changan Automotive Big Data Platform: Challenges and Practices in Connected Vehicle Scenarios
DataFunTalk
DataFunTalk
Mar 4, 2024 · Big Data

Design and Implementation of a Lakehouse‑Integrated Data Platform for Financial Innovation by Shuxin Network

This article presents Shuxin Network's practical experience in building a cloud‑native, lakehouse‑integrated data platform for the financial sector, covering architecture evolution, challenges of domestic‑innovation (信创), the DataCyber solution, core components, deployment roadmap, and real‑world case studies.

Big DataCloud NativeFinancial Innovation
0 likes · 21 min read
Design and Implementation of a Lakehouse‑Integrated Data Platform for Financial Innovation by Shuxin Network
JD Retail Technology
JD Retail Technology
Feb 22, 2024 · Big Data

JD Retail Data Platform: Data Asset Governance, Metric Middle Platform, Visualization Tools, and AI‑Driven Applications

This technical article details JD Retail’s 2023 data platform advancements, covering data‑asset certification and governance, a metric middle‑platform for unified indicator management, sophisticated visualization components, low‑code orchestration, and large‑model AI applications that together improve data retrieval efficiency, reduce storage‑compute costs, and support rapid business decision‑making.

AIData GovernanceLow-code
0 likes · 39 min read
JD Retail Data Platform: Data Asset Governance, Metric Middle Platform, Visualization Tools, and AI‑Driven Applications
NetEase Cloud Music Tech Team
NetEase Cloud Music Tech Team
Feb 21, 2024 · Artificial Intelligence

Cloud Music Public Opinion Analysis Platform: Architecture and GPT-Based Implementation

The article describes NetEase Cloud Music’s public‑opinion analysis platform, which integrates external and internal data streams into a layered architecture—ingestion, processing, storage in Elasticsearch, visualization, and monitoring—and employs GPT‑based analyzers for clustering, sentiment, summarization, and intelligent alerts while optimizing costs and planning automated GPT‑driven reports.

ElasticsearchGPT analysisSentiment Analysis
0 likes · 13 min read
Cloud Music Public Opinion Analysis Platform: Architecture and GPT-Based Implementation
DataFunTalk
DataFunTalk
Feb 8, 2024 · Big Data

Design and Practice of Ant Group's Metric System

This talk by Ant Group’s senior technical expert Wang Gaohang details the definition, design, mechanism, productization, and future outlook of the company’s metric system, covering concept consensus, semantic layers, workflow, AI assistance, performance optimization, and practical case studies.

AIBig DataData Modeling
0 likes · 28 min read
Design and Practice of Ant Group's Metric System
DataFunSummit
DataFunSummit
Jan 31, 2024 · Big Data

iQIYI Magic Mirror: Evolution of a Big Data Analysis Platform

iQIYI's Magic Mirror platform, evolving from 1.0 to 3.0, addresses the growing data analysis demands of the internet industry by empowering self‑service analytics, introducing multi‑stage architectures, advanced computation engines, customizable SQL, and visual dashboards, thereby improving efficiency, scalability, and data security for business users.

Big DataSQLSelf‑service analytics
0 likes · 18 min read
iQIYI Magic Mirror: Evolution of a Big Data Analysis Platform
DataFunTalk
DataFunTalk
Jan 28, 2024 · Databases

Practical Experience of StarRocks Materialized Views at Didi

This article presents Didi's practical experience with StarRocks materialized views, covering the evolution of its OLAP architecture, the challenges of previous engines, the adoption of StarRocks, the design of materialized view acceleration for real‑time dashboards, and future optimization directions.

Big DataStarRocksdata platform
0 likes · 17 min read
Practical Experience of StarRocks Materialized Views at Didi
DataFunTalk
DataFunTalk
Jan 21, 2024 · Cloud Native

Building a System Observability Framework with YHP: Practices, Challenges, and Integrated Solutions

This article explains how YHP enables cloud‑native systems to achieve comprehensive observability by defining the three core signals—metrics, traces, and logs—addressing common enterprise pain points, and presenting an integrated platform that unifies data collection, storage, analysis, and visualization for efficient fault diagnosis and performance monitoring.

Cloud Nativedata platformlogs
0 likes · 22 min read
Building a System Observability Framework with YHP: Practices, Challenges, and Integrated Solutions
DataFunTalk
DataFunTalk
Jan 12, 2024 · Big Data

Building a Unified Data Empowerment Layer with Apache Kyuubi at GF Securities

The article describes how GF Securities designed and implemented a unified big‑data empowerment layer based on Apache Kyuubi to address data‑centric challenges, improve efficiency, ensure controllable governance, and support agile data scenarios across ingestion, processing, storage, and security.

Apache KyuubiBig DataData Empowerment
0 likes · 33 min read
Building a Unified Data Empowerment Layer with Apache Kyuubi at GF Securities
DataFunSummit
DataFunSummit
Jan 9, 2024 · Big Data

Introducing Yunqi Lakehouse: An Integrated Cloud‑Native Data Platform with Incremental Computing and Auto Materialized Views

This article introduces Yunqi's self‑developed Lakehouse product, explaining its cloud‑native, one‑stop data platform architecture, incremental computing that balances freshness, performance and cost, and the autoMV feature that automatically creates materialized views to boost query speed up to nine times.

Auto Materialized ViewBig DataLakehouse
0 likes · 14 min read
Introducing Yunqi Lakehouse: An Integrated Cloud‑Native Data Platform with Incremental Computing and Auto Materialized Views
DataFunSummit
DataFunSummit
Jan 4, 2024 · Big Data

YY Live Business Metric Governance Practice

This presentation details YY Live’s data product team’s end‑to‑end business metric governance practice, covering problem background, analysis, governance objectives, multi‑team collaboration, implementation steps, achieved efficiencies, and future directions leveraging large language models.

Big DataLLMSelf‑service analytics
0 likes · 16 min read
YY Live Business Metric Governance Practice
Goodme Frontend Team
Goodme Frontend Team
Jan 1, 2024 · Frontend Development

How Guming’s Front‑End Data Center Enables Real‑Time Monitoring for Web, Mini‑Programs, Flutter & Node.js

Guming’s Front‑End Data Center integrates monitoring, performance, logging, and analytics for web, mini‑programs, Flutter clients, and Node.js services, offering real‑time alerts, high availability, sampling, multi‑channel data pipelines, custom charting, and detailed CPU/GC profiling to streamline issue diagnosis and business insights.

FrontendMonitoringdata platform
0 likes · 10 min read
How Guming’s Front‑End Data Center Enables Real‑Time Monitoring for Web, Mini‑Programs, Flutter & Node.js
Zuoyebang Tech Team
Zuoyebang Tech Team
Dec 28, 2023 · Big Data

How We Scaled Our Data Platform by Migrating to Apache DolphinScheduler

Facing growing task volumes and diverse workload types, we upgraded our data development platform's scheduling engine to Apache DolphinScheduler, detailing the migration process, architectural enhancements, stability and observability improvements, multi‑tenant support, and the resulting performance gains and future roadmap.

Apache DolphinSchedulerBig DataMigration
0 likes · 12 min read
How We Scaled Our Data Platform by Migrating to Apache DolphinScheduler
DataFunTalk
DataFunTalk
Dec 28, 2023 · Product Management

Building an Effective Data Platform: Insights and Practices from Tencent's Senior Product Manager

Senior Tencent product manager He Zhichao shares his experience and methodology for creating a high‑quality data platform, covering the transition from technical roles to product, understanding data users’ needs, the Euler asset‑factory implementation, product‑manager best practices, and solutions to common data‑engineering challenges.

Data EngineeringData GovernanceDataOps
0 likes · 16 min read
Building an Effective Data Platform: Insights and Practices from Tencent's Senior Product Manager
Alibaba Cloud Big Data AI Platform
Alibaba Cloud Big Data AI Platform
Dec 26, 2023 · Big Data

How Panasonic Overcame Data Silos: A Big Data Governance Journey

Panasonic's digital transformation case study details the challenges of fragmented data across 64 subsidiaries, the strategic adoption of a serverless big‑data platform, governance milestones from 2021 to 2023, tool comparisons, standardization efforts, talent development, and future outlook driven by five core values.

Big DataCloud ComputingData Governance
0 likes · 15 min read
How Panasonic Overcame Data Silos: A Big Data Governance Journey
58 Tech
58 Tech
Dec 22, 2023 · Big Data

Design and Implementation of the JiShi BI Data Visualization Platform

This article details the architecture, core processes, and module designs of the JiShi BI platform, a self‑built data visualization and analysis system that integrates data ingestion, processing, enhanced AI‑driven analytics, and multi‑dimensional dashboard capabilities to support enterprise decision‑making.

BIData VisualizationEnhanced Analytics
0 likes · 12 min read
Design and Implementation of the JiShi BI Data Visualization Platform
Zhuanzhuan Tech
Zhuanzhuan Tech
Dec 14, 2023 · Big Data

Design and Implementation of a Data Service Platform for New Media Business

This article details the background, challenges, design principles, and implementation of a unified data service platform—including data modeling, multi-source governance, real-time processing, and a Doris-based storage solution—to support large‑scale video data for a new media operation.

Apache DorisData GovernanceData Modeling
0 likes · 7 min read
Design and Implementation of a Data Service Platform for New Media Business
DataFunTalk
DataFunTalk
Dec 5, 2023 · Big Data

Design and Practice of Xiaomi’s One‑Stop Data Production Platform

This article presents a comprehensive overview of Xiaomi’s data production platform, detailing the full data lifecycle, the technical‑driven product design methodology, the platform’s architecture and core capabilities, as well as real‑world case studies and a Q&A session that illustrate how the system improves data collection, storage, processing, and usage across the organization.

Data EngineeringETLdata lifecycle
0 likes · 17 min read
Design and Practice of Xiaomi’s One‑Stop Data Production Platform
Sohu Tech Products
Sohu Tech Products
Nov 22, 2023 · Big Data

Field Extraction and Read‑Time Modeling in the Honghu Data Platform

The Honghu data platform delivers a unified, UI‑driven environment that uses read‑time modeling to dynamically structure massive heterogeneous logs during queries, replacing pre‑defined schemas and ETL pipelines with rule‑based field extraction (regex, JSON, key‑value, IP) bound to data‑source types, trading CPU cycles for flexible, accurate analysis.

Regexdata platformfield extraction
0 likes · 17 min read
Field Extraction and Read‑Time Modeling in the Honghu Data Platform
Data Thinking Notes
Data Thinking Notes
Nov 2, 2023 · Operations

How Bilibili Built a Scalable Data Quality Assurance System for Its Data Warehouse

This article details Bilibili's data quality assurance framework, covering its evolution across four data platform stages, the architecture of its quality data warehouse, core capabilities such as a complete assurance system, digital‑driven continuous optimization, and efficient incident handling, plus case studies, future plans, and a Q&A session.

Big DataBilibiliData Warehouse
0 likes · 27 min read
How Bilibili Built a Scalable Data Quality Assurance System for Its Data Warehouse
Big Data Technology & Architecture
Big Data Technology & Architecture
Oct 23, 2023 · Big Data

Bilibili Data Quality Assurance: Architecture, Goals, Core Capabilities, and Future Outlook

This article outlines Bilibili's data quality assurance framework, detailing its evolution across four development stages, the current data platform architecture, identified pain points, four key quality objectives, core capabilities such as a quality data warehouse, comprehensive monitoring, digital optimization, fault handling, and future directions.

Big DataData GovernanceData Warehouse
0 likes · 22 min read
Bilibili Data Quality Assurance: Architecture, Goals, Core Capabilities, and Future Outlook
DataFunSummit
DataFunSummit
Oct 15, 2023 · Big Data

Construction and Architecture of JD One-Service Data Service System

This article details JD's three‑stage evolution of its data service platform, explains thematic (topic‑based) data services, introduces the One‑Service unified architecture, and outlines future plans for standardization, low‑code front‑end, and operational improvements.

Big DataData ServiceOne-Service
0 likes · 13 min read
Construction and Architecture of JD One-Service Data Service System
JD Tech
JD Tech
Oct 10, 2023 · Operations

Technical Case Study of JDV Visual Dashboard Platform for the 618 Promotion

This article details how JDV, JD.com’s internal visual dashboard platform, tackled the massive data‑intensive 618 promotion by implementing real‑time updates, cross‑midnight count stops, request‑state control, heartbeat monitoring, proxy data sources, and a suite of developer tools to ensure stability, performance, and rapid feature delivery.

Monitoringdata platformlarge-scale
0 likes · 18 min read
Technical Case Study of JDV Visual Dashboard Platform for the 618 Promotion
Architects Research Society
Architects Research Society
Sep 26, 2023 · Big Data

From a Single Data Lake to a Decentralized Data Mesh: A Step‑by‑Step Migration Guide

This article explains why traditional centralized data lakes hinder modern software development, introduces the data‑mesh concept as a decentralized alternative, and walks through an e‑commerce microservice example with concrete steps, data‑API designs, and migration tactics to transition from a monolithic lake to a distributed data mesh.

Data LakeData MeshData Product
0 likes · 22 min read
From a Single Data Lake to a Decentralized Data Mesh: A Step‑by‑Step Migration Guide
iQIYI Technical Product Team
iQIYI Technical Product Team
Sep 22, 2023 · Big Data

Data Lake: Concepts, Architecture, and Application in iQIYI's Data Platform

iQIYI’s data‑middle‑platform team built a four‑zone data lake—raw, product, work, and sensitive—integrated with unified ODS/DWD/MID layers, a metadata catalog, and self‑service tools, leveraging HDFS, Hive/Iceberg, Spark/Trino, and Flink, migrated to Apache Iceberg for real‑time freshness, and now aims to further streamline modules and adopt new technologies.

Apache IcebergData GovernanceData Lake
0 likes · 13 min read
Data Lake: Concepts, Architecture, and Application in iQIYI's Data Platform
Data Thinking Notes
Data Thinking Notes
Sep 6, 2023 · Big Data

How to Build an Effective Tagging System for Data Platforms

This article explains what objects and tags are, distinguishes physical, network and electronic tags, outlines how to construct and manage a comprehensive tag taxonomy for user profiling, product labeling, and data platforms, and details quality assessment criteria for tags in DMP, CDP, and recommendation systems.

CDPDMPdata platform
0 likes · 13 min read
How to Build an Effective Tagging System for Data Platforms
Efficient Ops
Efficient Ops
Aug 30, 2023 · Operations

How New Oriental Built a Scalable DevOps Platform to Cut Costs and Boost Security

New Oriental’s recent DevOps transformation details how the company tackled siloed platforms, built a unified service‑tree‑driven infrastructure, created a real‑time data processing platform, and implemented comprehensive security measures—including red‑blue exercises, penetration testing, sensitive data monitoring, and CA/KMS—to boost efficiency and reduce costs.

DevOpsOperationscost reduction
0 likes · 7 min read
How New Oriental Built a Scalable DevOps Platform to Cut Costs and Boost Security
Big Data Technology & Architecture
Big Data Technology & Architecture
Aug 22, 2023 · Big Data

DataOps Practices and Challenges at ByteDance: From Model to Productization

The article summarizes ByteDance's DataOps journey, detailing its mid‑platform tool and Data BP model, core performance metrics, quality, hardware and human efficiency challenges, concrete DataOps implementation, productization through DataLeap, best‑practice promotion, and future outlook for data‑driven business value.

Big DataByteDanceData Governance
0 likes · 17 min read
DataOps Practices and Challenges at ByteDance: From Model to Productization
DataFunSummit
DataFunSummit
Aug 20, 2023 · Big Data

Kuaishou Data Service System: Modeling, Architecture, and Future Directions

This article presents Kuaishou's comprehensive data service system, covering its domain modeling, evolution from custom to unified services, the Octo query engine and data preparation platform architecture, the dual data API and analysis services, and future plans for intelligence and serverless high‑performance capabilities.

Big DataData ServiceQuery Engine
0 likes · 16 min read
Kuaishou Data Service System: Modeling, Architecture, and Future Directions
Data Thinking Notes
Data Thinking Notes
Aug 13, 2023 · Big Data

How to Successfully Deliver a Data Governance Project: Step‑by‑Step Guide

This article outlines a comprehensive methodology for delivering a data governance project, covering planning, blueprint design, implementation, and acceptance phases, with detailed guidance on team formation, stakeholder roles, requirement analysis, platform architecture, management processes, and post‑deployment operations.

Big DataData GovernanceProject Management
0 likes · 12 min read
How to Successfully Deliver a Data Governance Project: Step‑by‑Step Guide
DataFunSummit
DataFunSummit
Aug 13, 2023 · Big Data

KwaiBI: Evolution of Kuaishou’s One‑Stop Business Intelligence Platform from 1.0 to 2.0

The article details Kuaishou’s KwaiBI business intelligence platform evolution, covering its 1.0 tool‑based implementation, the 2.0 standardized architecture built on an indicator middle‑platform, core processes, data integration, self‑service features, and future directions for self‑service and intelligent analytics.

BIBig DataData Integration
0 likes · 22 min read
KwaiBI: Evolution of Kuaishou’s One‑Stop Business Intelligence Platform from 1.0 to 2.0
DataFunSummit
DataFunSummit
Aug 8, 2023 · Artificial Intelligence

Xiaomi’s Experience in Deploying Intelligent Analytics: Productization, Challenges, and Future Plans

The article shares Xiaomi’s practical experience in building and productizing intelligent analytics, explaining why it is needed, how it integrates with BI, the essential prerequisites, staged implementation, technical challenges, and future roadmap including smart alerts, automated insights, and data Q&A.

AIBI IntegrationBig Data
0 likes · 15 min read
Xiaomi’s Experience in Deploying Intelligent Analytics: Productization, Challenges, and Future Plans
Didi Tech
Didi Tech
Jul 31, 2023 · Big Data

Data Serviceization at Didi: Architecture, Phases, and Standard Metric Service

Didi’s data serviceization converts raw business data into consumable services through a four‑stage pipeline—integration, development, production, and back‑flow—while the Data Dream Factory and Shu‑Chain platform automate synchronization, provide a unified access gateway for thousands of APIs, and introduce a standard metric service that abstracts storage complexities and ensures high‑performance, secure data delivery.

Data Integrationdata platformdata serviceization
0 likes · 16 min read
Data Serviceization at Didi: Architecture, Phases, and Standard Metric Service
DataFunSummit
DataFunSummit
Jul 30, 2023 · Big Data

Data Platform Evolution and Digital Practice in the FMCG Industry: A Baicaowei Case Study

This article presents a comprehensive case study of Baicaowei's data platform evolution, digital workflow, metric governance, business modeling, and BI insights, illustrating how big‑data technologies and rational architecture simplification empower the fast‑moving consumer goods sector to enhance operational efficiency and decision‑making.

BIFMCGdata platform
0 likes · 10 min read
Data Platform Evolution and Digital Practice in the FMCG Industry: A Baicaowei Case Study
DataFunSummit
DataFunSummit
Jul 24, 2023 · Big Data

Design and Practice of OPPO Big Data Diagnostic Platform

This article presents the background, technical architecture, feature set, workflow, and practical results of OPPO's big data diagnostic platform, illustrating how intelligent, non‑intrusive task analysis improves efficiency, stability, and cost across massive offline and real‑time workloads.

OPPOTask Optimizationdata platform
0 likes · 10 min read
Design and Practice of OPPO Big Data Diagnostic Platform
Architects Research Society
Architects Research Society
Jul 16, 2023 · Big Data

Four Innovation Phases of Netflix’s Trillion‑Scale Real‑Time Data Infrastructure

The article chronicles Netflix’s evolution from a failing batch pipeline to a cloud‑native, self‑service streaming platform, detailing four development phases, the technical challenges faced, the stream‑processing patterns introduced, key learnings, and future opportunities for real‑time data and machine‑learning workloads.

FlinkKafkaReal-time Streaming
0 likes · 30 min read
Four Innovation Phases of Netflix’s Trillion‑Scale Real‑Time Data Infrastructure
Architects Research Society
Architects Research Society
Jun 24, 2023 · Big Data

Why Enterprise Data Architects Should Build Distributed Data Meshes Instead of Large Centralized Platforms

The article argues that traditional centralized data warehouses and lakes are increasingly unsustainable for large enterprises and proposes a paradigm shift to a distributed data mesh architecture that emphasizes domain‑owned data products, discoverability, interoperability, and global governance to overcome inherent inefficiencies.

Data MeshDistributed Architecturedata platform
0 likes · 6 min read
Why Enterprise Data Architects Should Build Distributed Data Meshes Instead of Large Centralized Platforms
DeWu Technology
DeWu Technology
Jun 16, 2023 · Big Data

Traffic Replay Platform for Data Platform Testing

The team built an online traffic‑replay platform that captures real user requests, replays them in a synchronized pre‑release environment, automatically compares responses using AAdiff and field‑ignore rules, achieving 86% interface coverage, 30% fewer regression bugs, 98% replay success and halving manual testing effort, while providing a zero‑intrusion, high‑concurrency solution for ongoing smoke, regression, stress and cache validation.

Big Datadata platformtraffic replay
0 likes · 10 min read
Traffic Replay Platform for Data Platform Testing
DataFunSummit
DataFunSummit
Jun 16, 2023 · Big Data

Apache Kyuubi Practices and Service Evolution at iQIYI

This article details iQIYI's implementation of Apache Kyuubi for Spark Thrift Server, covering the evolution from native Spark Thrift to Kyuubi 0.7 and 1.x, multi‑tenant architecture, tag‑based configurations, SQL auditing, lineage collection, service monitoring, small‑file and Z‑order optimizations, and a brief Q&A.

Apache KyuubiSpark SQLdata platform
0 likes · 15 min read
Apache Kyuubi Practices and Service Evolution at iQIYI
DataFunTalk
DataFunTalk
Jun 13, 2023 · Big Data

Building a Big Data Security Center with Apache Ranger: Practices and Technical Insights from NetEase

This article presents NetEase's practical experience of constructing a big‑data security center using Apache Ranger, covering Ranger's core features, a comprehensive security solution, detailed technical analyses, and the outcomes of commercializing the platform across multiple enterprise environments.

Apache RangerBig Data Securityaccess control
0 likes · 30 min read
Building a Big Data Security Center with Apache Ranger: Practices and Technical Insights from NetEase
DevOps
DevOps
May 29, 2023 · Artificial Intelligence

Key Takeaways from Microsoft Build 2023: Azure OpenAI Service, Microsoft Fabric, and the Copilot Stack

The article summarizes Microsoft Build 2023 highlights, detailing Azure OpenAI Service's GA and new plugins, enterprise data integration via vector search, Microsoft Fabric's OneLake data lake, Windows Copilot features, and the Copilot Stack architecture that enables developers to build AI‑powered Copilot applications.

AIAzure OpenAICopilot
0 likes · 6 min read
Key Takeaways from Microsoft Build 2023: Azure OpenAI Service, Microsoft Fabric, and the Copilot Stack
DataFunSummit
DataFunSummit
May 27, 2023 · Big Data

Building and Practicing the Performance Assurance System of YouShu BI

This article presents an in‑depth overview of the YouShu BI product, outlines the high‑concurrency performance challenges faced by enterprise BI, and details the multi‑layer performance architecture—including front‑end, back‑end, data engine, and data source layers—along with smart caching, MPP acceleration, materialized views, and the Data Doctor operations that together ensure low‑latency, reliable analytics for large‑scale users.

BIMPPMaterialized Views
0 likes · 16 min read
Building and Practicing the Performance Assurance System of YouShu BI
Data Thinking Notes
Data Thinking Notes
May 17, 2023 · Big Data

Inside Wing Pay’s Scalable Big Data Platform: Architecture & Governance

This article details how Wing Pay built a comprehensive data development and governance platform, covering company background, business scenarios, goals, challenges, task development workflow, task types, SparkSQL editor features, double‑environment deployment, Airflow scheduling, DataX data bus, resource isolation, compute optimization, data quality monitoring, cloud‑native practices, future outlook, and a Q&A on data permissions and governance.

AirflowBig DataCloud Native
0 likes · 17 min read
Inside Wing Pay’s Scalable Big Data Platform: Architecture & Governance
Bilibili Tech
Bilibili Tech
May 12, 2023 · Big Data

Upgrade of Dependency Model in Bilibili Data Platform

Bilibili’s data platform upgraded its dependency model by shifting from project‑level to task‑level dependencies, adding root and end nodes, using virtual tasks for external data, introducing offset handling, implementing an abstract DependencySubject and asynchronous callbacks, achieving sub‑second latency for tens of thousands of daily tasks while planning automated lineage and richer rule support.

Bilibilidata dependencydata platform
0 likes · 10 min read
Upgrade of Dependency Model in Bilibili Data Platform
Data Thinking Notes
Data Thinking Notes
May 7, 2023 · Big Data

How Financial Institutions Can Master Data‑Driven Transformation in 2024

This article examines two decades of data warehouse evolution in the financial sector, identifies persistent pain points such as platform lag, data quality, and low service efficiency, and proposes a cloud‑native, data‑centric framework—including a unified blueprint, three‑layer architecture, and six core capabilities—to accelerate enterprise‑wide data capability building and drive high‑quality digital growth.

Big DataCloud NativeData Governance
0 likes · 18 min read
How Financial Institutions Can Master Data‑Driven Transformation in 2024
Top Architect
Top Architect
May 4, 2023 · Big Data

Data Middle Platform: General Architecture and Core Components

The article explains the concept, benefits, and detailed modular architecture of a data middle platform, covering data storage, acquisition, processing, governance, security, and operation frameworks, and illustrates how enterprises can build and evolve such platforms to turn data into valuable services.

Big DataData GovernanceData Integration
0 likes · 19 min read
Data Middle Platform: General Architecture and Core Components
DataFunTalk
DataFunTalk
Apr 9, 2023 · Big Data

Building an Agile Business Intelligence Platform at Zhongyuan Bank: Architecture, Practices, and Future Outlook

The article details Zhongyuan Bank's end‑to‑end agile BI platform construction, covering business goals, a step‑by‑step development timeline, core architecture, eight key functionalities, low‑code data processing, real‑time streaming, visualization dashboards, intelligent Q&A, and future directions for platform intelligence and openness.

BIBig DataData Visualization
0 likes · 19 min read
Building an Agile Business Intelligence Platform at Zhongyuan Bank: Architecture, Practices, and Future Outlook
ITPUB
ITPUB
Apr 7, 2023 · Big Data

How WeDataSphere Builds a One‑Stop, Open‑Source Big Data Platform

This article outlines the motivations for building a comprehensive data platform, describes the measurement and tailoring approach, details WeDataSphere’s architecture—including DataSphere Studio and Apache Linkis middleware—and shares the open‑source roadmap and future vision for the platform.

Apache LinkisDataSphere StudioMiddleware
0 likes · 11 min read
How WeDataSphere Builds a One‑Stop, Open‑Source Big Data Platform
DataFunSummit
DataFunSummit
Mar 11, 2023 · Big Data

Insights on Data Platform SaaS Transformation and Customization Strategies

The article examines the opportunities and challenges of turning data platforms into SaaS solutions, compares sales‑driven and product‑driven models, analyzes cost factors and industry gaps, and shares practical approaches such as platform‑plus‑component architecture, real‑world case studies, and product‑management considerations for better meeting B2B customization demands.

Cloud ComputingSaaScustomization
0 likes · 21 min read
Insights on Data Platform SaaS Transformation and Customization Strategies
ShiZhen AI
ShiZhen AI
Mar 1, 2023 · Cloud Native

Why We Chose Kafka for Our Open‑Source Real‑Time Streaming Platform

The article explains how market trends, data‑driven enterprise needs, and internal platform experience led Didi to build Know Streaming—a zero‑intrusion, plugin‑based real‑time streaming solution built on Kafka—to address scalability, operability, and community adoption challenges.

Cloud NativeKafkaKnowStreaming
0 likes · 12 min read
Why We Chose Kafka for Our Open‑Source Real‑Time Streaming Platform
DataFunTalk
DataFunTalk
Feb 27, 2023 · Big Data

Comprehensive Overview of Data Middle Platform Architecture and Its Core Frameworks

This article provides a detailed overview of data middle platform concepts, describing a decoupled six‑subsystem architecture—including storage, collection, processing, governance, security, and operation frameworks—while illustrating typical enterprise implementations, industry‑specific solutions, and best‑practice considerations for building scalable, secure, and value‑driven data platforms.

Big DataData GovernanceData Integration
0 likes · 25 min read
Comprehensive Overview of Data Middle Platform Architecture and Its Core Frameworks
DataFunTalk
DataFunTalk
Feb 26, 2023 · Big Data

Design, Optimization, and Use Cases of Data Lineage in ByteDance's DataLeap Platform

This article presents an in‑depth overview of DataLeap's data lineage capabilities, covering the challenges, multi‑layer model design, implementation with Apache Atlas and JanusGraph, performance optimizations, diverse use cases across asset, development, governance and security domains, and future trends for lineage technology.

Apache AtlasBig DataData Governance
0 likes · 19 min read
Design, Optimization, and Use Cases of Data Lineage in ByteDance's DataLeap Platform
StarRing Big Data Open Lab
StarRing Big Data Open Lab
Feb 17, 2023 · Big Data

Inside Xinghuan Tech’s Next‑Gen Big Data 3.0 Architecture: Unified, Cloud‑Native, Real‑Time

This article details Xinghuan Technology’s evolution from 2013 to the present, describing its self‑developed Big Data 3.0 stack—including a unified data platform, SQL‑centric development, cloud‑native resource scheduling, distributed storage managed by Raft, DAG‑based compute engines, and real‑time stream processing—while highlighting key milestones and design principles that differentiate it from traditional Hadoop‑based solutions.

SQL Optimizerdata platformreal-time processing
0 likes · 19 min read
Inside Xinghuan Tech’s Next‑Gen Big Data 3.0 Architecture: Unified, Cloud‑Native, Real‑Time
DataFunSummit
DataFunSummit
Feb 17, 2023 · Big Data

Data Governance Practices and Platform Construction with Alibaba DataWorks

Alibaba’s DataWorks team shares extensive experiences in building and operating a large‑scale data platform, covering data governance across stages—from data stability and quality to security, cost control, and organizational culture—illustrating how systematic practices and tools drive efficiency, reliability, and value for enterprises.

Big DataData Governancecost optimization
0 likes · 55 min read
Data Governance Practices and Platform Construction with Alibaba DataWorks
Data Thinking Notes
Data Thinking Notes
Feb 14, 2023 · Big Data

How Cloud Music Turned 60k Tables into Valuable Data Assets

This article details Cloud Music's year‑long data assetization journey, covering the background, practical achievements, governance methods, and future roadmap for turning massive data warehouses into high‑value, well‑governed assets that drive cost reduction and business insight.

Big DataData GovernanceData Warehouse
0 likes · 10 min read
How Cloud Music Turned 60k Tables into Valuable Data Assets
Kuaishou Big Data
Kuaishou Big Data
Feb 14, 2023 · Big Data

How OAX Revolutionizes Open Analysis in Kuaishou’s Data Platform

This article introduces OAX (Open Analysis eXpressions), Kuaishou’s unified open‑analysis language, detailing its design background, guiding principles, five‑layer language model, syntax—including data types, compute capabilities and five analysis elements—its access protocol, runtime architecture, optimization steps, and the benefits it brings to the company’s big‑data analytics ecosystem.

OAXOpen AnalysisQuery Language
0 likes · 19 min read
How OAX Revolutionizes Open Analysis in Kuaishou’s Data Platform
Data Thinking Notes
Data Thinking Notes
Feb 6, 2023 · Big Data

How Tencent Tackles Data Governance Challenges with the WeData Platform

This article outlines Tencent's data governance challenges, its internal three‑stage practice, detailed case studies such as Tencent News and PCG cost governance, and introduces the WeData platform's architecture and tools for standardization, quality, security, and metadata management, concluding with a Q&A session.

Big DataData GovernanceTencent
0 likes · 17 min read
How Tencent Tackles Data Governance Challenges with the WeData Platform
DataFunSummit
DataFunSummit
Jan 16, 2023 · Big Data

Building an O2O Industry Data Platform: From Monitoring to Diagnosis

This article shares practical insights on constructing an O2O industry data platform, detailing user classification, business pain points, and a three‑step strategy—monitoring, analysis, and diagnosis—to extract core metrics, implement tailored reporting, conduct operational and pricing analyses, and drive data‑driven product improvements.

AnalysisBusiness IntelligenceData Product
0 likes · 15 min read
Building an O2O Industry Data Platform: From Monitoring to Diagnosis
DataFunTalk
DataFunTalk
Jan 14, 2023 · Big Data

Lean Data Methodology: A Visual Guide to Data‑Driven Digital Transformation

Lean Data Methodology combines lean thinking, design thinking, the Cynefin framework, and agile principles to create a data‑driven digital transformation system that defines value, eliminates waste, and equips enterprises with strategic, product, governance, collaboration, platform, and cultural capabilities for building lean digital enterprises.

Data GovernanceLean Datadata platform
0 likes · 11 min read
Lean Data Methodology: A Visual Guide to Data‑Driven Digital Transformation
DataFunSummit
DataFunSummit
Jan 7, 2023 · Big Data

Redefining the Customer Data Platform (CDP) for New Energy Vehicle Companies

This article explores why the automotive industry's shift to new energy vehicles necessitates a redefinition of the Customer Data Platform (CDP), detailing the changing traffic structure, varied departmental demands, CDP typologies, implementation strategies, and the benefits of a unified, extensible CDP architecture for marketing, sales, and after‑sales.

AutomotiveBig DataCDP
0 likes · 13 min read
Redefining the Customer Data Platform (CDP) for New Energy Vehicle Companies
StarRing Big Data Open Lab
StarRing Big Data Open Lab
Jan 6, 2023 · Big Data

Designing a Unified Enterprise Data Storage and Compute Platform

This article explains how enterprises can build a unified data storage and compute foundation, covering strategic goals, functional and architectural requirements, and the layered design of business support, storage‑compute, and resource management to enable scalable, secure, and high‑performance data platforms.

ComputeEnterprise Architecturedata platform
0 likes · 15 min read
Designing a Unified Enterprise Data Storage and Compute Platform
DataFunTalk
DataFunTalk
Jan 6, 2023 · Big Data

ZhongAn's Hundred‑Billion‑Scale Data Integration Service: Architecture, Business Support, and Evolution

This article presents the architecture and practical experience of ZhongAn's hundred‑billion‑scale data integration service, covering common integration technologies, business support scenarios for offline and real‑time data, technical challenges, evolution from single‑machine to service‑oriented designs, and future directions using Flink and DataX.

Data IntegrationDataXETL
0 likes · 31 min read
ZhongAn's Hundred‑Billion‑Scale Data Integration Service: Architecture, Business Support, and Evolution
Data Thinking Notes
Data Thinking Notes
Jan 5, 2023 · Big Data

Why Data Lakes Are Outshining Traditional Data Warehouses: A Deep Dive

This comprehensive guide explains the evolution from traditional data warehouses to modern data lakes, detailing concepts, architectures, differences, implementation steps, and real‑world case studies, while also comparing major cloud providers' solutions and highlighting how data platforms support digital transformation and analytics.

Big DataData LakeData Warehouse
0 likes · 97 min read
Why Data Lakes Are Outshining Traditional Data Warehouses: A Deep Dive
DataFunSummit
DataFunSummit
Jan 4, 2023 · Big Data

Data Intelligence Expert Interview – Maturity, Trends, and Practices of Data Middle Platforms

The interview gathers insights from data‑platform experts on the maturity stages, technology trends, implementation methodologies, open‑source ecosystems, system architectures, governance, security, and assessment criteria of modern data middle platforms, offering a comprehensive guide for practitioners.

Big DataData GovernanceData Observability
0 likes · 28 min read
Data Intelligence Expert Interview – Maturity, Trends, and Practices of Data Middle Platforms
Data Thinking Notes
Data Thinking Notes
Jan 3, 2023 · Big Data

How a Scalable Data Service Platform Transforms Big Data into APIs

This article outlines the design and implementation of a unified data service platform that standardizes data access, accelerates model processing, provides flexible API construction, and ensures high availability through gateway, caching, and monitoring, ultimately reducing cost and improving efficiency for both C‑end and B‑end applications.

API GatewayBig DataData Modeling
0 likes · 25 min read
How a Scalable Data Service Platform Transforms Big Data into APIs
vivo Internet Technology
vivo Internet Technology
Dec 28, 2022 · Big Data

Vivo Real-Time Computing Platform: Architecture, Practices, and Applications

The Vivo Real‑Time Computing Platform, built on Apache Flink, delivers a one‑stop data construction and governance solution that processes up to 5 PB daily, offering high‑availability submission and control services, robust stability, rich SQL usability, efficient Kubernetes deployment, strong security, and supports real‑time warehouses and short‑video recommendation, while targeting future elastic scaling and lake‑house unification.

Apache FlinkReal-Time ComputingStream Processing
0 likes · 18 min read
Vivo Real-Time Computing Platform: Architecture, Practices, and Applications
High Availability Architecture
High Availability Architecture
Dec 27, 2022 · Big Data

Design and Implementation of a Data Service Middle Platform for Scalable Data SaaS

This article presents a comprehensive overview of a data service middle platform, detailing its background, architectural design, data construction, model definition and acceleration, API creation, query processing, service gateway, common solutions for standardization and cost reduction, as well as achieved results and future plans.

APIBig DataData Service
0 likes · 22 min read
Design and Implementation of a Data Service Middle Platform for Scalable Data SaaS
DevOpsClub
DevOpsClub
Dec 19, 2022 · R&D Management

How ByteDance’s DevMind Platform Transforms R&D Efficiency Measurement

The article details ByteDance’s DevMind platform, describing its origins, the challenges of measuring software development efficiency, the collaborative and value‑driving “flywheel” concepts, the architectural design across data lifecycle and query engine layers, and the principles and future roadmap for scaling R&D performance.

DevMindR&D Efficiencydata platform
0 likes · 29 min read
How ByteDance’s DevMind Platform Transforms R&D Efficiency Measurement
DataFunSummit
DataFunSummit
Dec 13, 2022 · Big Data

Introducing the Star River Big Data Development Platform: Architecture, Core Capabilities, and Future Plans

This article presents an in‑depth overview of 58.com’s self‑built Star River big data platform, covering its evolution across three eras, resource management hierarchy, core technical capabilities such as metadata services, data maps and lineage, governance practices, and the roadmap for further enhancements.

Big DataData Governancearchitecture
0 likes · 14 min read
Introducing the Star River Big Data Development Platform: Architecture, Core Capabilities, and Future Plans
DataFunSummit
DataFunSummit
Dec 10, 2022 · Big Data

Applying Apache Spark in Guanyuan Self-Service Analytics System: Architecture, Challenges, and Solutions

This presentation details how Guanyuan Data leverages Apache Spark within its self‑service analytics platform, covering product features, flexible deployment, resource isolation, performance challenges, architectural solutions, and future cloud‑native enhancements to support thousands of users and massive query workloads.

Apache SparkBig DataSelf‑service analytics
0 likes · 14 min read
Applying Apache Spark in Guanyuan Self-Service Analytics System: Architecture, Challenges, and Solutions
转转QA
转转QA
Dec 8, 2022 · Backend Development

Applying AOP to Reduce Coupling in a Data Construction Platform

This article explains how Aspect‑Oriented Programming (AOP) was introduced into a data construction platform to address high development effort, strong business coupling, and maintenance difficulty by isolating cross‑cutting concerns such as logging, thereby improving modularity, development speed, and code maintainability.

AOPSpringaspect-oriented-programming
0 likes · 5 min read
Applying AOP to Reduce Coupling in a Data Construction Platform
DataFunSummit
DataFunSummit
Dec 7, 2022 · Big Data

Modern Data Governance at NetEase DataFan: Evolution, Challenges, and Solutions

This article details NetEase DataFan's journey in building a full‑stack big‑data platform, explains the design‑first data‑mid‑platform approach, analyzes cost, quality, and security problems encountered, and presents the modern data‑governance framework that integrates development, governance, and consumption into a closed loop.

Big DataCost ManagementData Governance
0 likes · 22 min read
Modern Data Governance at NetEase DataFan: Evolution, Challenges, and Solutions
DataFunTalk
DataFunTalk
Dec 5, 2022 · Big Data

Data Governance Practices at ZTO Express: Challenges, Solutions, and Future Plans

The article details ZTO Express's data governance journey, covering company background, drivers and goals, challenges such as data asset inventory, standardization, quality, and modeling, and outlines their multi‑layered governance framework, practical implementations in data quality, model and metadata, and future plans.

Metadatadata platformlogistics
0 likes · 17 min read
Data Governance Practices at ZTO Express: Challenges, Solutions, and Future Plans
DataFunSummit
DataFunSummit
Dec 1, 2022 · Big Data

City Data Acquisition Platform: Architecture, Core Technologies, and Incremental Synchronization Strategies

This article presents an overview of a smart city unified perception platform, detailing its modular architecture, solutions for multi-source heterogeneity, incremental synchronization strategies, and real-time API data collection, while discussing extensibility and practical implementation considerations.

API integrationBig DataData Ingestion
0 likes · 20 min read
City Data Acquisition Platform: Architecture, Core Technologies, and Incremental Synchronization Strategies
ITPUB
ITPUB
Nov 25, 2022 · Big Data

How Berserker’s Big Data Platform Solved Scheduling, State and Scaling Challenges

This article details the architecture, evolution, and technical solutions of the Berserker big‑data platform—including component design, state‑management problems, release strategies, two‑phase commit, RPC handling, routing, message queuing, containerized execution, dependency model redesign, and future roadmap—demonstrating how the system achieved high availability, low latency, and scalable operations.

DockerKubernetesRaft
0 likes · 19 min read
How Berserker’s Big Data Platform Solved Scheduling, State and Scaling Challenges
Huolala Tech
Huolala Tech
Nov 24, 2022 · Big Data

How Huolala Built Its Own Self-Service Data Analysis Platform from Scratch

This article details Huolala's journey from identifying the need for a fast, secure, and scalable BI solution to designing and implementing a self‑service data analysis platform that integrates diverse data sources, offers intuitive visualisation, and addresses real‑world operational challenges.

BIProduct DevelopmentSelf‑service analytics
0 likes · 13 min read
How Huolala Built Its Own Self-Service Data Analysis Platform from Scratch
DataFunSummit
DataFunSummit
Nov 22, 2022 · Big Data

BI Platform Practice at Xiaomi: Evolution, Architecture, and Future Directions

This article details Xiaomi's multi‑year journey in building a group‑wide Business Intelligence platform, covering its historical evolution, technical challenges in performance, modeling, visualization and permissions, the current four‑layer architecture, and future plans to make the platform more business‑centric and simpler.

BIBig DataData Modeling
0 likes · 15 min read
BI Platform Practice at Xiaomi: Evolution, Architecture, and Future Directions
DataFunTalk
DataFunTalk
Nov 21, 2022 · Big Data

Building a Unified Data Analytics Platform at TCL Using StarRocks

The article describes how TCL leveraged StarRocks to create a unified data analytics platform, detailing the company’s background, OLAP evolution, typical StarRocks use cases such as real‑time dashboards, HR analytics, and email alerts, and outlines future plans for further integration and performance improvements.

Case studyStarRocksTcl
0 likes · 10 min read
Building a Unified Data Analytics Platform at TCL Using StarRocks
DataFunTalk
DataFunTalk
Nov 20, 2022 · Big Data

Evolution and Practices of Modern Data Governance at NetEase DataFun

This article outlines NetEase DataFun's journey in building a full‑stack big data platform, describing the four‑stage development of data governance—from designing a unified data middle‑platform to addressing cost, quality, and security challenges—and presents the principles of modern data governance that integrate development, consumption, and continuous improvement.

data platformdata security
0 likes · 23 min read
Evolution and Practices of Modern Data Governance at NetEase DataFun
Data Thinking Notes
Data Thinking Notes
Nov 10, 2022 · Big Data

Building Kuaishou’s Scalable Metadata Management Platform for Big Data

This article details Kuaishou’s evolution of its metadata management platform—from early Hive‑centric beginnings to a unified 2.0 architecture and a forward‑looking 3.0 vision—highlighting challenges, key technologies, and how metadata drives data production, consumption, governance, and cost optimization across the big‑data middle platform.

Data Governancedata platformmetadata lineage
0 likes · 17 min read
Building Kuaishou’s Scalable Metadata Management Platform for Big Data
政采云技术
政采云技术
Nov 8, 2022 · Industry Insights

How Small Big‑Data Frontend Teams Can Thrive: A Survival Guide

This guide outlines the essential concepts of big data, the roles of a front‑end data team, practical workflow steps, platform architecture, industry benchmarks, and actionable strategies for small teams to improve efficiency, visualization capabilities, and digital operations.

Big DataData VisualizationFrontend
0 likes · 14 min read
How Small Big‑Data Frontend Teams Can Thrive: A Survival Guide