This article presents Ant Group's recent explorations and practices in real-time data warehousing, detailing its architecture, data quality assurance, stream‑batch integration, and future data lake implementation, while highlighting the use of Flink, ODPS, and Paimon for scalable, low‑latency analytics.
The article traces Dewu’s supply‑chain data platform from slow MySQL reporting through early CDC‑based wide tables to a Flink‑Kafka‑ClickHouse 1.0 design, then to a more scalable Flink‑Kafka‑Hologres 2.0 architecture that solves upsert and compute‑storage separation, while detailing key operational tricks, code‑generation tools, and future plans for lake‑house integration.
The article describes a modular traffic‑control platform for e‑commerce search and recommendation that lets operators quickly adjust strategies for emergencies, cold‑start items, and experiments, replacing costly multi‑team development with a unified operation center, service center, data hub, algorithmic PID controller, real‑time metrics, independent recall chain, and cross‑scene AB testing, while outlining future extensions.
The Xianyu recommendation system, built by backend expert Wan Xiaoyong, evolved from offline scoring to a full‑graph, serverless recall‑ranking pipeline that tackles C2C uncertainties through centralized feature engineering, model compression, staged deployment, flexible experimentation, robust governance, and plans for automated attribution and interpretability.
Chronon is an open‑source framework that centralizes feature definitions to guarantee training‑inference consistency, eliminates complex ETL pipelines, and supports real‑time and batch processing across diverse data sources, cutting feature‑development cycles from months to under a week, as demonstrated by Airbnb’s 40,000‑feature deployment.
The Shenzhen Elasticsearch technical event, co‑hosted by the Elastic Chinese community and Tencent Cloud, presented practical sessions on optimizing the Elastic Stack for search, real‑time analytics, logging, security and APM, featuring compression encoding, MongoDB fusion, ByteDance extensions, cost‑effective log storage, Lucene indexing, cross‑cluster replication, vector engine integration, and large‑scale case studies from Tencent, Tiptop Data and vivo.
The paper presents a configurable real‑time rule engine for live‑streaming product audits that decouples data aggregation from rule execution, uses QLExpress for dynamic conditions, supports Dubbo and HTTP sources, and enables safe gray‑release updates, cutting the rule‑change cycle from weeks to near‑real‑time.
This article explains how to build a dual‑stream real‑time data pipeline for big‑data applications, defines construction standards, and details a three‑step high‑fidelity load‑testing process that ensures stability and high availability during peak promotional periods.
This article describes the four-stage evolution of the Lego client‑side logging system—covering its initial zero‑to‑one architecture, the separation of business and technical logs, real‑time reporting improvements, and the latest architecture redesign that boosts performance, reduces overhead, and provides a safe migration path.
This article demonstrates how to use ActionOMS to capture incremental changes from OceanBase, stream them to Kafka in various formats, and employ Flink to deduplicate and aggregate transaction data into a daily summary, illustrating a complete real-time data pipeline for financial use cases.
This article compares HTTP polling, WebSocket, and Server‑Sent Events (SSE) for large‑screen data fetching, explains their advantages and drawbacks, outlines suitable business scenarios, and provides complete front‑end and back‑end code examples for implementing SSE and WebSocket connections.
This article chronicles the development, design decisions, and technical achievements of Alibaba's Data Replication Center (DRC), a real‑time heterogeneous database synchronization platform that has become a core infrastructure for multi‑active data centers, large‑scale e‑commerce, and cloud services.
This article chronicles Netflix’s evolution from a failing batch pipeline to a cloud‑native, multi‑tenant streaming platform across four phases, detailing the motivations, challenges, strategic bets, and patterns that enabled the company to scale real‑time data processing to trillions of events per day.
This article presents Huolala’s end‑to‑end implementation of Flink CDC for real‑time data capture, detailing the business background, reasons for selecting Flink CDC over Canal, component comparisons, production‑level platform enhancements, data‑lake integration, validation methods, and future directions for unified data ingestion.
Feature engineering, the process of creating input variables for machine learning models, is crucial for banking risk control; this article explains the concepts of features, variables, and metrics, outlines challenges in real‑time feature pipelines, and proposes a practical architecture and best practices for building an efficient, low‑code feature platform.
This article details CVTE’s adoption of stream computing, describing the company background, the challenges of traditional data pipelines, the design of a CDC‑Kafka integration platform, evaluations of PipelineDB, ksqlDB, Materialize and RisingWave, and the overall impact on real‑time analytics and operational efficiency.
This article summarizes JD's real‑time data product practice, covering product overview, low‑code real‑time platform construction, stream‑batch integrated architecture, and the three‑layer operational defense model, while highlighting challenges, evolution, user distribution, and future directions.
The article details Zhihu's data empowerment team's design and implementation of a low‑cost, high‑response real‑time data platform built on Apache Doris, covering real‑time business metrics, algorithm features, and user profiling, and explains the challenges, architectural choices, tooling, performance gains, and future directions.
This article presents Ant Group's recent explorations and practices in real-time data warehousing, covering the system architecture, streaming data quality assurance, flow‑batch integrated applications, and future data lake integration, while sharing technical details and operational insights for large‑scale data processing.
This article describes how NetEase Youdao replaced its Doris‑based real‑time data warehouse with Amoro Mixed Hive, detailing the architectural challenges, the Mixed Hive design, implementation steps, performance optimizations, community contributions, and future roadmap to achieve a unified lakehouse with minute‑level freshness and reduced development and operational costs.