DataOps | BestHub

DataFunSummit

May 22, 2025 · Operations

Automated Fault Detection and Repair System for Grab's Data Pipelines (Hugo) – Architecture, Implementation, and Impact

This article presents Grab's Hugo platform, an automated fault‑detection and self‑healing system for over 4,000 data pipelines that combines multi‑source signal collection, intelligent diagnosis, layered auto‑repair, and a health API to dramatically improve data visibility, reduce manual intervention, and boost operational efficiency across the company.

Big DataDataOpsautomation

0 likes · 12 min read

Automated Fault Detection and Repair System for Grab's Data Pipelines (Hugo) – Architecture, Implementation, and Impact

DataFunSummit

Nov 1, 2024 · Big Data

DataFun Summit Session Overview and E‑book Access Instructions

The article outlines how to obtain the DataFun Summit e‑book by following the public account instructions and provides concise English summaries of twelve technical sessions covering data lineage, integration, AI language models, multimodal content, game AI agents, lake‑warehouse governance, big‑data architecture, and cluster management.

AIBig DataDataOps

0 likes · 5 min read

DataFun Summit Session Overview and E‑book Access Instructions

DataFunTalk

Jul 21, 2024 · Artificial Intelligence

Integrating DataOps with Large Language Models for Text2SQL: Practices, Challenges, and Future Directions

This article presents a comprehensive overview of how DataOps principles combined with large language models such as GPT‑4 enable more agile and intelligent data engineering workflows, focusing on Text2SQL applications, schema‑linking techniques, practical product implementations, and future research challenges.

AIData EngineeringDataOps

0 likes · 23 min read

Integrating DataOps with Large Language Models for Text2SQL: Practices, Challenges, and Future Directions

DataFunTalk

May 13, 2024 · Big Data

Data Integration Maturity Model: From ETL to EtLT

The article examines the evolution of data integration architectures—from traditional ETL through ELT to the emerging EtLT model—highlighting their advantages, disadvantages, industry trends, maturity stages, and practical guidance for enterprises and professionals navigating modern big‑data pipelines.

Big DataDataOpsELT

0 likes · 31 min read

Data Integration Maturity Model: From ETL to EtLT

DataFunSummit

May 13, 2024 · Big Data

Metadata‑Driven Data Governance: Concepts, Architectures, and Practices

This article explains how metadata‑driven data governance addresses the challenges of the digital economy by detailing the era background, limitations of traditional methods, the roles of Data Fabric, Data Mesh and DataOps, and presenting real‑world case studies and future directions.

AIBig DataData Mesh

0 likes · 14 min read

Metadata‑Driven Data Governance: Concepts, Architectures, and Practices

DataFunSummit

Apr 11, 2024 · Big Data

Building Integrated Data Governance and R&D Operations with DataOps: Practices and Insights from China Unicom Digital Technology

This article shares how China Unicom Digital Technology leverages DataOps to build an integrated data governance, research and development, and operations capability, outlining challenges, methodological considerations, a seven-step governance framework, and a multi-center collaborative mechanism to achieve sustainable data-driven value.

Big DataData EngineeringDataOps

0 likes · 15 min read

Building Integrated Data Governance and R&D Operations with DataOps: Practices and Insights from China Unicom Digital Technology

DataFunSummit

Apr 4, 2024 · Big Data

Design Principles and Future Directions of DataOps

This article outlines the core capabilities of data-driven development, the background and architecture of DataOps, its research challenges and focus areas, and explores future directions such as data virtualization, platform governance, and data value assessment, providing a comprehensive overview of DataOps practices.

Big DataData EngineeringDataOps

0 likes · 8 min read

Design Principles and Future Directions of DataOps

DataFunSummit

Apr 1, 2024 · Big Data

DataOps at ByteDance: Challenges, Implementation, and Future Outlook

This article examines ByteDance's DataOps journey, detailing the data‑engineering challenges faced, the concrete solutions and productization through the DataLeap platform, the metrics and best‑practice framework adopted, and the future directions involving AI‑assisted development and open‑source collaboration.

Big DataData EngineeringDataOps

0 likes · 20 min read

DataOps at ByteDance: Challenges, Implementation, and Future Outlook

DataFunSummit

Mar 15, 2024 · Product Management

How to Build a Good Data Platform: Insights from Tencent’s Senior Product Manager

This presentation shares the speaker’s experience and practical methods for creating an effective data platform, covering the transition from technical roles to product management, deep understanding of data workers' needs, Tencent Oura asset‑factory practices, a product‑management methodology, and a Q&A session that addresses governance, performance, and engineering challenges.

Data EngineeringDataOpsdata governance

0 likes · 15 min read

How to Build a Good Data Platform: Insights from Tencent’s Senior Product Manager

DevOps

Jan 17, 2024 · Operations

Agile Data Management: Principles, Practices, and Implementation Guide

This article explains how agile methodologies can be applied to data management, covering the need for agile data practices, core principles, iterative modeling, governance, CI/CD pipelines, tooling, metrics, security, case studies, challenges, and future outlooks in a comprehensive, step‑by‑step guide.

AgileData ModelingDataOps

0 likes · 13 min read

Agile Data Management: Principles, Practices, and Implementation Guide

DataFunTalk

Dec 28, 2023 · Product Management

Building an Effective Data Platform: Insights and Practices from Tencent's Senior Product Manager

Senior Tencent product manager He Zhichao shares his experience and methodology for creating a high‑quality data platform, covering the transition from technical roles to product, understanding data users’ needs, the Euler asset‑factory implementation, product‑manager best practices, and solutions to common data‑engineering challenges.

Data EngineeringDataOpsdata governance

0 likes · 16 min read

Building an Effective Data Platform: Insights and Practices from Tencent's Senior Product Manager

DataFunTalk

Dec 3, 2023 · Big Data

NetEase Data: Practices and Architecture of a Metrics Middle Platform

This article presents NetEase Data's end‑to‑end experience in building a metrics middle platform, covering product evolution, the motivations for a unified metrics layer, core technologies such as a logical semantic model, a custom metric query language, engine‑agnostic execution, and future roadmap plans.

Big DataData AnalyticsDataOps

0 likes · 12 min read

NetEase Data: Practices and Architecture of a Metrics Middle Platform

DataFunSummit

Oct 24, 2023 · Big Data

DataOps & DataFabric in the Era of Large Models

In this presentation, Guo Wei, CEO of Baijiang Open Source and seasoned big‑data expert, explores how large‑model AI reshapes DataOps and DataFabric, detailing efficiency gains, intelligent deployment, and future enterprise architectures for big‑data and AI integration.

Artificial IntelligenceBig DataDataFabric

0 likes · 3 min read

DataOps & DataFabric in the Era of Large Models

DataFunSummit

Oct 20, 2023 · Big Data

Tencent OLA t‑Metric Metric Platform: Headless BI Practices and Architecture

The article introduces Tencent's OLA data‑governance platform and its t‑Metric metric middle‑platform, explains the Headless BI concept, details the configuration‑driven metric production workflow, core capabilities, architecture, unified query service, ecosystem integration, and answers audience questions about real‑time analysis, dimension handling, and trust mechanisms.

Big DataDataOpsdata governance

0 likes · 21 min read

Tencent OLA t‑Metric Metric Platform: Headless BI Practices and Architecture

DataFunTalk

Oct 8, 2023 · Big Data

Full-Process DataOps Practices for Large-Scale Business Data Reporting at Baidu

This article reveals how Baidu implements end‑to‑end DataOps for its commercial data products, covering challenges of massive report generation, the design of a layered data architecture, platform‑wide automation, serverless deployment, risk control, monitoring, and optimization to achieve scalable, reliable data pipelines.

Big DataDataOpsOptimization

0 likes · 13 min read

Full-Process DataOps Practices for Large-Scale Business Data Reporting at Baidu

Bilibili Tech

Sep 15, 2023 · Big Data

Introducing Bilibili's SQLScan: Architecture, Key Technologies, and Production Impact

Bilibili's SQLScan is a static‑code analysis tool that parses Hive, Spark, Presto and Flink SQL via Antlr4, builds a unified AST, applies engine‑specific metadata plugins for rule enforcement, provides field‑lineage and cost‑analysis services, and has processed hundreds of thousands of daily queries, intercepting thousands of problematic statements to improve data quality and operational efficiency.

Big DataBilibiliDataOps

0 likes · 11 min read

Introducing Bilibili's SQLScan: Architecture, Key Technologies, and Production Impact

DataFunSummit

Aug 28, 2023 · Big Data

Building Data Production Pipelines with DataOps: Concepts, Practices, and a Six‑Stage Workflow

This article introduces DataOps, outlines its background and the problems it addresses, describes NetEase’s big‑data product ecosystem, and details a six‑stage data production pipeline—including coding, orchestration, testing, code review, release approval, and deployment – plus insights into two pipeline explorations.

Big DataData EngineeringDataOps

0 likes · 15 min read

Building Data Production Pipelines with DataOps: Concepts, Practices, and a Six‑Stage Workflow

DataFunSummit

Jul 15, 2023 · Big Data

Intelligent and Automated Data Quality Management in Big Data Systems

This article explores the challenges of data quality in mature big‑data environments and presents intelligent, automated approaches—including assertions, automatic detection, rule recommendation, link checking, and collaborative mechanisms—to embed quality checks throughout the data pipeline, improving efficiency and reliability.

Big DataData ObservabilityDataOps

0 likes · 18 min read

Intelligent and Automated Data Quality Management in Big Data Systems

DevOps Cloud Academy

Jun 1, 2023 · Big Data

DataOps 2.0: Integrated Data Development and Governance Practices at NetEase

The article recounts NetEase’s presentation at the inaugural DataOps conference, detailing the evolution from DataOps 1.0 pipeline to a 2.0 integrated data development‑governance model, the challenges faced, practical solutions, and strategic advice for data managers.

Big DataData EngineeringDataOps

0 likes · 11 min read

DataOps 2.0: Integrated Data Development and Governance Practices at NetEase

DataFunSummit

May 13, 2023 · Big Data

Expert Interview on Data Governance: Core Domains, Challenges, and Future Trends

In this interview, three data‑governance experts from Tencent, ByteDance, and Alibaba discuss the fundamental processes, core domains such as metadata, data lineage, metric systems, data quality and security, the main challenges they face, and emerging trends like DataOps, AI‑driven automation, and privacy‑preserving technologies.

Big DataData SecurityDataOps

0 likes · 14 min read

Expert Interview on Data Governance: Core Domains, Challenges, and Future Trends