2 views collected around this technical thread.
This article explains JD's complex recommendation system data pipeline—from indexing, sampling, and feature engineering to explainability and real‑time metrics—highlighting challenges such as data consistency, latency, and the use of Flink for massive, low‑latency processing.
This article presents Huolala’s end‑to‑end implementation of Flink CDC for real‑time data capture, detailing the business background, reasons for selecting Flink CDC over Canal, component comparisons, production‑level platform enhancements, data‑lake integration, validation methods, and future directions for unified data ingestion.
Chronon is an open‑source framework that centralizes feature definitions to guarantee training‑inference consistency, eliminates complex ETL pipelines, and supports real‑time and batch processing across diverse data sources, cutting feature‑development cycles from months to under a week, as demonstrated by Airbnb’s 40,000‑feature deployment.
This article compares HTTP polling, WebSocket, and Server‑Sent Events (SSE) for large‑screen data fetching, explains their advantages and drawbacks, outlines suitable business scenarios, and provides complete front‑end and back‑end code examples for implementing SSE and WebSocket connections.
This article demonstrates how to use ActionOMS to capture incremental changes from OceanBase, stream them to Kafka in various formats, and employ Flink to deduplicate and aggregate transaction data into a daily summary, illustrating a complete real-time data pipeline for financial use cases.
This article presents Ant Group's recent explorations and practices in real-time data warehousing, detailing its architecture, data quality assurance, stream‑batch integration, and future data lake implementation, while highlighting the use of Flink, ODPS, and Paimon for scalable, low‑latency analytics.
This article describes the four-stage evolution of the Lego client‑side logging system—covering its initial zero‑to‑one architecture, the separation of business and technical logs, real‑time reporting improvements, and the latest architecture redesign that boosts performance, reduces overhead, and provides a safe migration path.
Feature engineering, the process of creating input variables for machine learning models, is crucial for banking risk control; this article explains the concepts of features, variables, and metrics, outlines challenges in real‑time feature pipelines, and proposes a practical architecture and best practices for building an efficient, low‑code feature platform.
This article presents Ant Group's recent explorations and practices in real-time data warehousing, covering the system architecture, streaming data quality assurance, flow‑batch integrated applications, and future data lake integration, while sharing technical details and operational insights for large‑scale data processing.
This article details CVTE’s adoption of stream computing, describing the company background, the challenges of traditional data pipelines, the design of a CDC‑Kafka integration platform, evaluations of PipelineDB, ksqlDB, Materialize and RisingWave, and the overall impact on real‑time analytics and operational efficiency.
This article describes how NetEase Youdao replaced its Doris‑based real‑time data warehouse with Amoro Mixed Hive, detailing the architectural challenges, the Mixed Hive design, implementation steps, performance optimizations, community contributions, and future roadmap to achieve a unified lakehouse with minute‑level freshness and reduced development and operational costs.
This article presents Ant Group's comprehensive real-time data framework and streaming graph processing engine, detailing its architecture, unified batch‑stream capabilities, and practical applications such as traffic attribution, real‑time OLAP, and user‑behavior intent analysis, while outlining future directions.
This article explains how to build a dual‑stream real‑time data pipeline for big‑data applications, defines construction standards, and details a three‑step high‑fidelity load‑testing process that ensures stability and high availability during peak promotional periods.
Didi’s unified real‑time data stack outlines best‑practice component choices for four key scenarios—metric monitoring, BI analysis, online services, and feature/tag systems—detailing pipelines from source to sink, resource‑usage guidelines, and a one‑stop development platform to build stable, high‑performance streaming solutions.
This article presents a comprehensive case study of Manbang's real‑time data platform, detailing its business background, cloud‑native Flink + Hologres architecture, migration from self‑built clusters, real‑time product features, decision‑making workflows, and future roadmap, highlighting performance and cost benefits.
The article traces Dewu’s supply‑chain data platform from slow MySQL reporting through early CDC‑based wide tables to a Flink‑Kafka‑ClickHouse 1.0 design, then to a more scalable Flink‑Kafka‑Hologres 2.0 architecture that solves upsert and compute‑storage separation, while detailing key operational tricks, code‑generation tools, and future plans for lake‑house integration.
The article describes a modular traffic‑control platform for e‑commerce search and recommendation that lets operators quickly adjust strategies for emergencies, cold‑start items, and experiments, replacing costly multi‑team development with a unified operation center, service center, data hub, algorithmic PID controller, real‑time metrics, independent recall chain, and cross‑scene AB testing, while outlining future extensions.
This article summarizes JD's real‑time data product practice, covering product overview, low‑code real‑time platform construction, stream‑batch integrated architecture, and the three‑layer operational defense model, while highlighting challenges, evolution, user distribution, and future directions.
The article details Zhihu's data empowerment team's design and implementation of a low‑cost, high‑response real‑time data platform built on Apache Doris, covering real‑time business metrics, algorithm features, and user profiling, and explains the challenges, architectural choices, tooling, performance gains, and future directions.
The Xianyu recommendation system, built by backend expert Wan Xiaoyong, evolved from offline scoring to a full‑graph, serverless recall‑ranking pipeline that tackles C2C uncertainties through centralized feature engineering, model compression, staged deployment, flexible experimentation, robust governance, and plans for automated attribution and interpretability.