ByteDance’s StreamShield delivers a three‑layer resiliency framework—engine self‑healing, hybrid replication at the cluster level, and chaos‑tested releases—that enables over 70,000 concurrent Flink jobs on 11 million CPU cores to meet strict SLAs with second‑level startup and robust fault tolerance.
This article explains how Baidu Netdisk transitioned from Spark Streaming to a Flink‑based Tiangong real‑time computing engine, detailing the evolution, reasons for choosing Flink, architecture, configuration examples, business use cases, technical challenges, and future platform plans.
This article details Alibaba's decade‑long evolution of its real‑time computing platform, the massive operational challenges of managing Flink clusters at million‑core scale, and the comprehensive strategies—including SLA metrics, self‑healing services, cloud‑native redesign, and job‑level advisory tools—used to ensure stability, cost efficiency, and performance during peak events like Double‑11.
Alibaba Cloud, together with Zhejiang University, Alipay and Xieyun Technology, received the Zhejiang Province Science and Technology Progress First Prize for their cloud‑native large‑scale cloud‑edge collaborative platform, which addresses edge resource constraints, real‑time computing, and massive node management, and has been widely applied across multiple industries.
Businesses across finance, automotive, logistics, and retail are increasingly adopting real-time computing with Flink and Hologres to meet growing data volume and latency demands, enabling instant analytics, risk monitoring, dynamic recommendations, and efficient operations, while cloud architectures evolve to support massive, low‑latency data streams.
The article reviews how data development accelerated in 2023—with mature offline computing, rapid adoption of real‑time and lake‑warehouse solutions, and a clearer technical layering—while offering practical insights and future directions for professionals entering 2024.
The 7th Tongcheng Travel Big Data Technology Salon in Beijing featured a series of expert talks covering StarRocks architecture evolution, lake‑house solutions with Paimon, Iceberg real‑time upsert, Data+AI for travel recommendation, vector retrieval in AI, JD Logistics real‑time computing governance, and multi‑task hotel ranking modeling, providing deep technical insights and future roadmaps.
This article examines the decade‑long digital transformation of China's express sector, focusing on Shentong's shift from electronic waybills to AI‑driven real‑time automation, data‑centric decision making, and autonomous delivery technologies that boost efficiency and reduce costs.
To replace costly, duplicated Flink jobs, Didi built Observe‑Compute (OBC), a cloud‑native, PromQL‑driven real‑time metric engine with centralized policy management, scalable containerized workers, and zero‑downtime scaling, achieving million‑RMB annual savings while handling 10 M points per second.
Manbang Group shares its practical experiences and insights on real-time computing, multi‑cloud platform architecture, data warehousing with Flink and Holo, real‑time decision and feature platforms, and future plans for scaling these systems to support logistics and recommendation algorithms.
The DataFun Summit 2023 Real‑Time Computing Forum showcases a series of expert talks on Apache Flink, stream‑batch integration, cloud‑native streaming databases, and large‑scale real‑time data warehousing, featuring speakers from Alibaba Cloud, Taobao, Didi, Ant Group and RisingWave.
This article details NetEase Media's migration of most Flink jobs to a self‑built real‑time platform on Kubernetes, covering the benefits of K8s isolation, the chosen native deployment mode, performance‑critical optimizations, monitoring, resource‑recommendation, and future directions for cloud‑native streaming workloads.
The article details Baidu's precise watermark design for its unified streaming‑batch data warehouse, describing how a centralized watermark server and client ensure end‑to‑end data completeness, align real‑time and batch windows with 99.9‑99.99% precision, and support accurate anti‑fraud calculations within the broader big‑data ecosystem.
Zhihu’s real‑time computing platform, initially built as Skytree 1.0 on Kubernetes and later re‑engineered as Mipha 2.0 with Flink SQL, unified metadata management, dynamic jar loading, UDF support, Protobuf format, CDC integration, and extensive operational optimizations, now processes petabyte‑scale data with high reliability.
The Vivo Real‑Time Computing Platform, built on Apache Flink, delivers a one‑stop data construction and governance solution that processes up to 5 PB daily, offering high‑availability submission and control services, robust stability, rich SQL usability, efficient Kubernetes deployment, strong security, and supports real‑time warehouses and short‑video recommendation, while targeting future elastic scaling and lake‑house unification.
NetEase’s advertising engine platform evolved from a monolithic, high‑concurrency system handling over a billion daily requests into a layered, distributed architecture that unifies indexing, billing, user‑tagging, and monitoring services, leverages Elasticsearch and custom extensions for fast retrieval, and plans further upgrades such as a custom retrieval kernel and Go‑based services.
The 2022 Vivo developer conference showcased a series of technical breakthroughs—including a custom wepy‑chameleon compiler for frontend upgrades, low‑code platforms for backend and game development, a real‑time computing platform built on Flink, advanced graph scheduling, cloud‑native container strategies, monitoring enhancements, database automation, and large‑scale messaging middleware—highlighting Vivo's comprehensive push toward efficiency and innovation across its internet services.
This article presents Tencent's practical experience in improving Flink job stability, covering the Oceanus platform, stability challenges, and concrete optimization techniques such as reducing failures, minimizing impact, accelerating recovery, and proactive issue detection, followed by a summary and future outlook.
This article describes the design and implementation of an automated FlinkSQL tuning system that monitors metrics, evaluates task health with rule‑based logic, calculates optimal resource adjustments, and performs fast scaling to reduce cluster waste, lower operational costs, and maintain SLA compliance.
This article details Car Home's AutoStream platform evolution from Storm to Flink‑based versions, covering real‑time application scenarios, strict budget‑controlled resource management, automatic scaling, lake‑house architecture with Iceberg, PyFlink integration, and future plans for resource optimisation and batch‑stream unification.
The talk details Huya’s real‑time computing platform evolution from chaotic early stages to a unified, containerized system, defines core SLA metrics focused on latency compliance, describes capability enhancements such as demand monitoring, task analysis, dynamic scaling, and outlines future goals for usability, stability, openness, and unified stream‑batch processing.
This article details Huya's real‑time computing platform evolution, core SLA definitions focused on latency compliance, capability enhancements such as demand management, task analysis, dynamic resource scaling, and outlines future directions emphasizing usability, stability, openness, and unified batch‑stream processing.
This article details OPPO’s real‑time computing platform, covering its business scope, big‑data architecture built on Flink, Spark and Trino, the end‑to‑end job development lifecycle, SQL IDE features, diagnostic and monitoring mechanisms, link latency tracking, SLA guarantees, practical use cases, and upcoming lakehouse and cloud‑native evolution.
The article reviews highlights from Flink Forward 2021, describing how real‑time computing is spreading across traditional industries, the unstoppable move toward Flink SQL, the emergence of Flink ML, and the vision of a streaming warehouse built on Flink Dynamic Table technology.
This comprehensive guide outlines a step‑by‑step learning roadmap for aspiring big data professionals, covering fundamentals, programming languages, Linux, databases, distributed theory, networking, offline and real‑time computing, data governance, warehouses, toolchains, video/book recommendations, future industry trends, interview tips, and community resources.
This article details JD's adoption of Flink for real‑time computing, covering its evolution from Storm to Flink on Kubernetes, the platform architecture, major optimization techniques such as preview topology, backpressure handling, dynamic rebalance, checkpoint‑as‑savepoint, and outlines future plans including stream‑batch integration, stability improvements, intelligent operations, and AI integration.
Youzan upgraded its real‑time computing engine from Flink 1.10 to 1.13 to meet rising SQL and containerization demands, gaining enhanced SQL syntax, time‑function handling, Window TVF standardization, Hive integration, K8s stability, elastic scaling, richer Kafka and format support, improved metrics and debugging tools, and successfully migrated all custom connectors, UDFs, and SQL jobs to the new Kubernetes‑based platform.
This article details NetEase Yanxuan's real-time computing platform development from 2017 to present, covering its architecture, Flink‑SQL development environment, service‑oriented deployment, resource optimization, cloud‑native migration, comprehensive data governance, and future plans for stream‑batch integration and intelligent job diagnostics.
This article details Bilibili's comprehensive optimization of Flink real‑time computing, covering connector stability improvements, SQL interval‑join enhancements, runtime state and checkpoint refinements, a diagnostic tool, and future directions for high‑throughput streaming workloads.
This article details the design and implementation of 58.com’s real‑time computing platform, covering its architecture, data ingestion, storage, Flink‑based stream processing, SQL extensions, performance optimizations, Storm‑to‑Flink migration tools, the Wstream management console, state handling, monitoring, and future roadmap.
This article examines the evolution of big‑data architectures toward cloud‑native, real‑time processing, and provides a detailed comparison of dimension‑table and result‑table options—including MySQL, Redis, and Alibaba Cloud Tablestore—along with their performance, cost, and scalability characteristics for Flink SQL workloads.
This article explores the evolution of real-time computing in the big data domain, detailing Apache Flink's capabilities, architectural designs, technology selections such as Kafka, Canal, HBase, ClickHouse, and provides practical implementation guides and case studies from Alibaba, Tencent, and other enterprises.
This article presents the evolution, architecture, and key innovations of Bigo's real‑time computing platform—covering its migration from Spark Streaming to Flink, unified platform design, development tools, operational enhancements, and the efficiency gains achieved in business scenarios such as ETL and AB‑testing.
At Youzan, Flink real‑time tasks running on Kubernetes are optimized by daily GC‑log memory analysis and Kafka‑throughput monitoring, which compute recommended heap sizes and parallelism adjustments to eliminate over‑provisioned CPU and memory, automate alerts, and pave the way for fully automated resource tuning.
This article explains the essence of frontend development as view‑reaction to events, compares reactive programming with MVVM and Redux, demonstrates a complete RxJS implementation for a news app, and shows how the same data‑flow concepts extend to real‑time computing with Apache Flink.
This article details the evolution of Beike's commercial strategy algorithm platform, describing its business scenarios, bidding mechanisms, architecture redesign across online, near‑real‑time, and offline layers, model training, vector retrieval, service governance, and the performance and stability improvements achieved.
This article details Jingdong's evolution from Storm to Flink, the architecture of its Kubernetes‑based real‑time computing platform, extensive containerization practices, performance and stability optimizations, and the future plan to unify batch‑stream processing while expanding SQL support and intelligent operations.
Tencent’s game marketing analysis system “EAS” evolved from inefficient TDW HiveSQL jobs and file‑heavy real‑time pipelines to a scalable ClickHouse‑based deduplication service that processes hundreds of thousands of daily activity counts in sub‑second time, offering fast, reliable, and maintainable participant deduplication for massive marketing campaigns.
JD Digits' intelligent anti‑fraud platform leverages machine learning, big‑data processing, graph neural networks and small‑sample knowledge‑graph algorithms to provide millisecond‑level, real‑time protection across 600+ scenarios, while also offering AI‑powered solutions to banks and publishing research at top conferences.
iQIYI’s senior data engineer shares the evolution of its big‑data services from Hadoop to a Flink‑based real‑time computing platform, detailing architecture, monitoring improvements, StreamingSQL capabilities, business use cases like recommendation and deep‑learning data generation, and future plans for unified stream‑batch processing.
This article describes the evolution of a real‑time computing platform from SQL 1.0 built on Spark Structured Streaming to SQL 2.0 powered by Flink‑SQL, covering dynamic tables, continuous queries, dimension‑table joins, cache optimization, DDL extensions, platformization, operational challenges and future roadmap.
Didi has unified its real‑time computing on Apache Flink, creating an enhanced StreamSQL service with extended DDL, built‑in parsers and UDX, supporting thousands of nodes, millions of jobs, and trillions of daily records, while addressing state management, high availability, multi‑language UDFs, and pursuing real‑time ML and data‑warehouse integration.
Ctrip’s flight search service processes two billion daily queries by employing a multi‑level Redis cache, machine‑learning‑driven TTLs, distributed pooling and overload protection, AI‑based anti‑scraping, and robust load‑balancing across three data centers, delivering sub‑second latency, up to three‑fold throughput gains and significant cost reductions.
Senior Didi technology expert Liang Li-yin shares how Didi leverages Apache Flink for large‑scale real‑time computing, covering service architecture, StreamSQL advantages, multi‑cluster management, task control, monitoring, meta‑store integration, challenges, and future plans such as high availability, real‑time ML, and unified batch‑stream processing.
This article details Dada Group's development of the Dada Flink SQL engine, describing its background, architecture, parser design, dimension‑table join strategies, numerous enhancements such as HA support, Kafka keyword handling, metadata integration, Redis and ClickHouse sinks, BINLOG simplification, and future migration plans toward Flink 1.10.
This article introduces Apache Flink's Streaming SQL, explains its theoretical foundations such as the table‑stream relationship and watermark semantics, describes the platform's practical enhancements—including source/sink wrappers, built‑in functions, and native Retract Stream support—and showcases several real‑time computation examples.
The article thoroughly explains Apache Flink’s checkpoint mechanism—including state types, coordinator workflow, exactly‑once versus at‑least‑once semantics, common failure sources such as code exceptions, storage or network issues, and practical configuration tips like interval settings, local recovery and externalized checkpoints.
The article examines how Apache Flink's state management features, including QueryableState and the upcoming Savepoint Processor API, can serve as a lightweight database for real‑time applications, discussing their advantages, limitations, and practical usage scenarios.
This article details Beike's real‑time computing engine, covering its background, streaming platform built on Spark Streaming and Flink, data ingestion via Kafka, metadata handling, SQL‑based task development, monitoring, storage solutions, and future roadmap for resource management and AI‑enhanced monitoring.
This article details the design, implementation, and optimization of a Flink‑based real‑time computing platform at Tongcheng‑Elong, covering the evolution from Storm to Flink, support for FlinkSQL and FlinkStream, metric collection, logging, data lineage, savepoint management, and numerous stability fixes contributed back to the open‑source community.
The article chronicles former Google engineer He Changhua’s return to China to lead Ant Financial’s AI and big‑data initiatives, highlighting the BASIC strategy, the open‑source SQLFlow tool, and the ambitious development of a fully real‑time, AI‑powered big‑data computing platform.
The article reviews the essential infrastructure for autonomous driving, covering massive sensor data storage strategies, the role of metadata, offline and real‑time computing platforms, basic micro‑service components, and various business scenarios, highlighting why robust big‑data handling is critical.
The 2019 QInfrarch technical carnival in Suzhou gathered nearly 160 engineers from Qunar and partner companies to share in‑depth sessions on private cloud databases, real‑time hotel computation, AI‑driven ticket customer service, and OceanBase architecture, concluding with networking, tea breaks, and a lucky‑draw for attendees.
Ant Group's Vice‑CTO announced the open‑source SQLFlow tool that merges SQL simplicity with machine‑learning power, aiming to lower AI adoption barriers, while chief architect He Changhua outlines a real‑time big‑data platform that fuses OLTP, OLAP, and AI for universal data intelligence.
The article presents Alibaba Cloud's real-time computing solution based on Flink and HBase, covering market competition, open‑source ecosystem, containerized architecture on Kubernetes, and typical applications such as online education video analysis, city‑brain traffic management, and fraud detection.
The article presents Renrenche’s end‑to‑end mobile data platform, detailing its overall architecture, real‑time Spark‑based computation engine, Web IDE, metadata management, BI reporting built on ClickHouse, and how data‑driven practices empower both online and offline business operations.
Data Mill is a Spark‑Streaming‑based real‑time computation framework that abstracts tasks as DataFrames, enables SQL‑driven development, and supports DSP business requirements by reducing latency to 15‑30 minutes while providing a scalable architecture, caching strategy, and automated fault handling.
From early self‑built Storm and Spark Streaming clusters to a unified YARN‑based Spark platform and finally a low‑latency Flink system with extended CEP and StreamSQL capabilities, Didi’s real‑time computing platform evolved through three stages, delivering multi‑tenant isolation, rich SQL processing, and dramatically reduced development costs.
This article provides a comprehensive technical overview of Kafka, covering its core concepts, producer and consumer models, architecture, configuration parameters, replication mechanisms, performance optimizations, operational monitoring, tooling scripts, and related product implementations for real-time data processing.
The paper presents a real‑time computation architecture on Alibaba Cloud Blink that scores and ranks non‑timeline feed items within a sliding 72‑hour window, updating rankings every few minutes, using Redis ZSET for fast retrieval, and discusses scaling optimizations such as interval tuning and external join‑and‑rank services.
This article details Alibaba's real‑time computing platform, outlining its operational challenges, the unified automation platform Aquila, proactive fault‑elimination strategies, and ongoing moves toward intelligent, data‑driven management to support massive workloads during events like Double‑11.
The article introduces Meteor, a Storm‑based real‑time computation platform developed by Ctrip Marketing to simplify topology management, automate deployment, and improve resource efficiency for complex marketing scenarios, highlighting its architecture, features, and measurable business impact.
This article explains how Alibaba built a massive real‑time computing platform using Flink and its Blink extensions, detailing the challenges of ultra‑low latency, exactly‑once guarantees, and high throughput, and showing how these technologies powered the record‑breaking Double 11 shopping festival.
He Xiaofeng, JD Mall Infrastructure chief architect, discusses his role in building a real‑time computing platform, applying streaming frameworks, machine learning, and knowledge‑graph techniques to product data mining, improve search accuracy, and outline future research directions.
The article explains how NetEase’s real‑time big‑data stream computing platform, Sloth, handles massive, continuously generated data during China’s Double 11 shopping festival, covering use cases, architectural shifts from batch to incremental processing, technical challenges, and the role of stream‑SQL for easier development.
This article explains how Alibaba’s Blink, built on Apache Flink, transforms batch‑oriented big‑data platforms into a unified, high‑performance real‑time computing engine, detailing its architecture, state management, checkpointing, and successful deployment in e‑commerce, search, recommendation, and online machine‑learning scenarios.
Alibaba’s e‑commerce search engine evolved over three years from offline batch models to a sophisticated AI-driven system that integrates real‑time feature ingestion, online learning, deep and reinforcement learning, enabling dynamic personalization and decision‑making that boosts conversion during high‑traffic events like Double 11.
This article details Alibaba's real‑time computing architecture for the 2016 Double 11 event, covering background, core components such as DRC, TT, Galaxy, OTS, XTool and OneService, and explains optimization techniques, fault‑tolerance strategies, stress‑testing practices, and future upgrade plans to handle massive streaming data workloads.
This article explains how Tencent built a billion‑scale, generalized recommendation system by designing a reusable algorithm library, deploying a low‑latency, highly available real‑time streaming platform (R2), and offering a cloud‑based recommendation engine that simplifies integration for internet businesses.
This article explains Alibaba’s dual‑link real‑time computing framework, detailing its micro‑ and macro‑level pipelines, key components such as Pora, iGraph and SP, online learning architectures, pointwise and pairwise ranking models, bandit‑based strategy optimization, PID‑controlled traffic balancing, and the impressive performance gains achieved during the Double 11 shopping festival.
The article presents Alibaba's real‑time computing platform for search, detailing its micro‑ and macro‑level architectures, online learning frameworks, point‑wise and pair‑wise ranking models, bandit‑based strategy optimization, and PID‑controlled traffic regulation, and reports significant performance gains during the Double‑11 shopping festival.
This article explains the principles of real-time computing, compares it with offline batch processing, and demonstrates a practical solution using Kafka for ingestion, Apache Storm for continuous computation, and various storage options, while also covering streaming concepts and Storm's high‑availability mechanisms.
During a detailed live session, senior Dianping engineer Wang Xinchun explains the company's real‑time computing platform built on Apache Storm, covering use cases such as dashboards, search and recommendation, system architecture, data ingestion tools like Blackhole and Puma, performance tuning, monitoring, and practical best‑practice recommendations.
