BestHub
Sign in
Tagged articles

Flink

1000 articles · Page 6 of 10
58 Tech
58 Tech
May 5, 2022 · Big Data

Low-Code Real-Time Data Warehouse Construction System Using Flink

This article describes a low‑code, Flink‑based real‑time data‑warehouse construction system that abstracts the warehouse building process into ODS, DWD, DWS, and ADS layers, leverages a domain‑specific language and plugin engine to reduce development effort, and details its architecture, DSL design, plugin extensibility, dimension‑table completion, stream merging, aggregation, and storage strategies.

DSLFlinkReal-time Data Warehouse
0 likes · 11 min read
Low-Code Real-Time Data Warehouse Construction System Using Flink
Big Data Technology & Architecture
Big Data Technology & Architecture
May 4, 2022 · Big Data

Apache Hudi 0.11.0 Release Highlights: Multi‑Mode Index, Data Skipping, Async Index, Spark & Flink Integration, and New Utilities

The Apache Hudi 0.11.0 release introduces multi‑mode metadata indexing, enhanced data‑skipping, asynchronous indexing, extensive Spark and Flink integration improvements, new bundle utilities, and expanded metadata synchronization with BigQuery, AWS Glue, and DataHub, while also adding bucket indexing and encryption support.

Apache HudiAsync IndexData Skipping
0 likes · 13 min read
Apache Hudi 0.11.0 Release Highlights: Multi‑Mode Index, Data Skipping, Async Index, Spark & Flink Integration, and New Utilities
Bilibili Tech
Bilibili Tech
May 3, 2022 · Artificial Intelligence

Bilibili AI Collaboration Platform Based on AIFlow: Architecture, Evolution, and Stream‑Batch Fusion

Bilibili built an AI collaboration platform based on AIFlow to simplify real-time machine-learning workflows, evolving through three stages that added event-driven scheduling, UI-driven parameter management, version snapshots, and a stateless client-server design, while enabling stream-batch fusion for feature back-filling; future work targets high availability, Airflow 2.0 compatibility, and richer streaming ML operators.

AIFlowBilibiliFlink
0 likes · 17 min read
Bilibili AI Collaboration Platform Based on AIFlow: Architecture, Evolution, and Stream‑Batch Fusion
HomeTech
HomeTech
Apr 27, 2022 · Big Data

AutoStream Real‑Time Computing Platform: Architecture, Resource Management, Scaling, Lakehouse Integration, and PyFlink Practices

This article details Car Home's AutoStream platform evolution from Storm to Flink‑based versions, covering real‑time application scenarios, strict budget‑controlled resource management, automatic scaling, lake‑house architecture with Iceberg, PyFlink integration, and future plans for resource optimisation and batch‑stream unification.

AutoStreamFlinkLakehouse
0 likes · 15 min read
AutoStream Real‑Time Computing Platform: Architecture, Resource Management, Scaling, Lakehouse Integration, and PyFlink Practices
DataFunSummit
DataFunSummit
Apr 22, 2022 · Big Data

Huya Real-Time Computing SLA Practice: Platform Evolution, Core SLA Definition, Capability Building, and Future Outlook

The talk details Huya’s real‑time computing platform evolution from chaotic early stages to a unified, containerized system, defines core SLA metrics focused on latency compliance, describes capability enhancements such as demand monitoring, task analysis, dynamic scaling, and outlines future goals for usability, stability, openness, and unified stream‑batch processing.

FlinkReal-Time ComputingSLA
0 likes · 12 min read
Huya Real-Time Computing SLA Practice: Platform Evolution, Core SLA Definition, Capability Building, and Future Outlook
ITPUB
ITPUB
Apr 19, 2022 · Big Data

Which Real-Time Data Warehouse Architecture Fits Your Needs? A Deep Dive

This article explains why modern enterprises need real‑time data‑warehouse architectures, breaks down traditional layered warehouse concepts, compares Lambda and Kappa models, evaluates five practical real‑time solutions—including Iceberg‑based lakehouse and MPP databases—provides code snippets, and offers selection guidance with real‑world company examples.

FlinkKappa ArchitectureLambda Architecture
0 likes · 19 min read
Which Real-Time Data Warehouse Architecture Fits Your Needs? A Deep Dive
Big Data Technology & Architecture
Big Data Technology & Architecture
Apr 15, 2022 · Big Data

Configuring Flink SQL Client with Iceberg: Catalogs, DDL, Data Insertion and Query

This guide explains how to set up the Flink SQL client to work with Apache Iceberg, covering Scala version requirements, downloading and deploying Iceberg jars, configuring Hive and HDFS catalogs, creating databases and tables, performing insert and overwrite operations, and querying data in both batch and streaming modes.

CatalogFlinkHadoop
0 likes · 18 min read
Configuring Flink SQL Client with Iceberg: Catalogs, DDL, Data Insertion and Query
DataFunTalk
DataFunTalk
Apr 15, 2022 · Big Data

Huya Real-Time Computing SLA Practices: Platform Evolution, Core SLA Definition, Capability Building, and Future Outlook

This article details Huya's real‑time computing platform evolution, core SLA definitions focused on latency compliance, capability enhancements such as demand management, task analysis, dynamic resource scaling, and outlines future directions emphasizing usability, stability, openness, and unified batch‑stream processing.

FlinkReal-Time ComputingSLA
0 likes · 13 min read
Huya Real-Time Computing SLA Practices: Platform Evolution, Core SLA Definition, Capability Building, and Future Outlook
Shopee Tech Team
Shopee Tech Team
Apr 14, 2022 · Big Data

URL Normalization and Statistical Analysis in MDAP Using Probabilistic and Machine Learning Techniques

MDAP normalizes URLs by automatically learning pattern‑tree rule models using entropy‑based splits, gibberish and numeric detection, and scalable Flink processing, which groups millions of raw URLs into concise patterns for accurate statistical monitoring, dramatically reducing data noise while still facing latency and model‑iteration challenges.

Flinkmachine learningpattern tree
0 likes · 20 min read
URL Normalization and Statistical Analysis in MDAP Using Probabilistic and Machine Learning Techniques
dbaplus Community
dbaplus Community
Apr 13, 2022 · Big Data

How Meituan Built a Scalable Real‑Time Data Warehouse with Flink

This article explains Meituan's real‑time data warehouse architecture, covering typical business scenarios, the evolution of its streaming platform, key design challenges, solutions such as unified data models, SQL‑based development, UDF hosting, operator optimizations, and future plans for incremental processing and unified batch‑stream semantics.

FlinkMeituanReal-time Data
0 likes · 18 min read
How Meituan Built a Scalable Real‑Time Data Warehouse with Flink
Big Data Technology & Architecture
Big Data Technology & Architecture
Apr 11, 2022 · Big Data

Real-Time Data Warehouse Construction: Background, Objectives, Architecture, and Case Studies

This article explains the growing demand for real‑time data warehouses, outlines their objectives and layered architecture, and presents detailed case studies from Didi, Kuaishou, Tencent, Youzan and others, illustrating design choices, implementation challenges, and best practices for building scalable streaming data platforms.

ClickHouseFlinkKafka
0 likes · 48 min read
Real-Time Data Warehouse Construction: Background, Objectives, Architecture, and Case Studies
DataFunSummit
DataFunSummit
Apr 6, 2022 · Big Data

Real-time Dimension Modeling with Flink SQL: Challenges and Solutions

This article presents a JD.com case study on applying Flink SQL for real‑time dimension modeling, detailing two complex streaming scenarios—full‑join of multiple streams and full‑group aggregation—along with the associated challenges of historical data handling, state management, and performance optimization, and proposes component‑based architectural solutions.

FlinkSQLStreaming
0 likes · 14 min read
Real-time Dimension Modeling with Flink SQL: Challenges and Solutions
Big Data Technology & Architecture
Big Data Technology & Architecture
Apr 5, 2022 · Big Data

Using ElasticsearchSink with Apache Flink: Configuration, Retry Strategies, and Failure Handling

This article introduces the ElasticsearchSink for Apache Flink, explains how to add Maven dependencies, implement the sink with configuration and retry settings, details failure handlers, and highlights important considerations such as exception handling and checkpoint requirements for reliable streaming pipelines.

ElasticsearchFlinkRetry
0 likes · 9 min read
Using ElasticsearchSink with Apache Flink: Configuration, Retry Strategies, and Failure Handling
NetEase Yanxuan Technology Product Team
NetEase Yanxuan Technology Product Team
Mar 30, 2022 · Big Data

Data Lake Construction and Practice at NetEase Yanxuan

NetEase Yanxuan replaced its cumbersome data‑warehouse with a flexible Delta‑Lake/Iceberg data lake, creating a unified metadata layer and real‑time ingestion pipelines that cut latency from nightly batches to seconds, slashed compute and storage costs, supported diverse business scenarios and machine‑learning feature engineering, and set the stage for broader future expansion.

Data IntegrationData LakeDelta Lake
0 likes · 16 min read
Data Lake Construction and Practice at NetEase Yanxuan
Efficient Ops
Efficient Ops
Mar 29, 2022 · Big Data

How Tencent Cloud Boosted APM Metric Computation Speed 2‑3× with Flink Optimizations

This article explains how Tencent Cloud's APM metric calculation, which transforms massive Span data into aggregated metrics using Flink, faced performance bottlenecks and was optimized through job splitting, batch merging, and dimension pruning, ultimately achieving a 2‑3× speed increase and cutting resource usage to about 30% of the original.

APMFlinkbig data
0 likes · 10 min read
How Tencent Cloud Boosted APM Metric Computation Speed 2‑3× with Flink Optimizations
58 Tech
58 Tech
Mar 29, 2022 · Big Data

Design and Implementation of the 58 Group Penalty Data Center

This article presents the design, architecture, and implementation of a unified penalty data center for 58 Group, detailing the challenges of heterogeneous data sources, the selection of Flink for real‑time ETL, the use of a DSL and LRU aggregation, and the adoption of MVEL for feature recognition to achieve standardized, high‑performance penalty data processing.

ETLFlinkPenalty System
0 likes · 13 min read
Design and Implementation of the 58 Group Penalty Data Center
Big Data Technology & Architecture
Big Data Technology & Architecture
Mar 28, 2022 · Big Data

Real-time Dimension Modeling with Flink SQL: Problems, Challenges, and Solutions

This article presents JD's real-time dimension modeling case using Flink SQL, detailing two complex streaming scenarios, the difficulties of handling historical data and state management, and a component‑based solution that leverages external KV stores and optimized Flink operators to improve performance and scalability.

FlinkSQLStreaming
0 likes · 13 min read
Real-time Dimension Modeling with Flink SQL: Problems, Challenges, and Solutions
StarRocks
StarRocks
Mar 28, 2022 · Backend Development

Scaling Microservice Tracing with Zipkin and StarRocks: A Practical Guide

This article explains how Sohu Smart Media built a high‑performance tracing system for microservices by integrating Zipkin for data collection with StarRocks for storage and analytics, covering architecture, data models, SQL queries, Flink processing, and real‑world results that boost observability and engineering efficiency.

FlinkMicroservicesObservability
0 likes · 31 min read
Scaling Microservice Tracing with Zipkin and StarRocks: A Practical Guide
Alibaba Cloud Developer
Alibaba Cloud Developer
Mar 24, 2022 · Big Data

How Flink Powers Real‑Time Process Operations in China Construction Bank

This article details how China Construction Bank's fintech subsidiary leveraged Apache Flink to ingest, join, and analyze massive front‑end, request, and response logs in real time, overcoming data silos, latency challenges, and state‑management issues to enable end‑to‑end process visibility and operational optimization.

Flinkbankingprocess mining
0 likes · 17 min read
How Flink Powers Real‑Time Process Operations in China Construction Bank
StarRocks
StarRocks
Mar 23, 2022 · Databases

Accelerating Zepp Health’s Analytics with StarRocks: An OLAP Case Study

Facing inflexible point‑lookup limits and slow query times on HBase, Zepp Health redesigned its massive event‑tracking data pipeline—migrating ingestion through Kafka, Flink, and Hudi to a StarRocks‑based OLAP layer—achieving sub‑100 ms average query latency, 20 % storage savings, and dramatically faster multi‑dimensional analytics.

FlinkHudiKafka
0 likes · 9 min read
Accelerating Zepp Health’s Analytics with StarRocks: An OLAP Case Study
DataFunTalk
DataFunTalk
Mar 23, 2022 · Big Data

Iceberg Data Lake Query Optimization Practices and Governance

This talk by Tencent senior engineer Chen Liang covers Iceberg table format fundamentals, data lake ingestion, query processing, hidden partitioning, time‑travel, major features, optimization techniques such as compaction, bin‑packing, sorting and Z‑ordering, and outlines a future roadmap for improving performance and governance in big‑data environments.

Data LakeFlinkZ-Ordering
0 likes · 12 min read
Iceberg Data Lake Query Optimization Practices and Governance
DeWu Technology
DeWu Technology
Mar 21, 2022 · Big Data

Real-time Customer Service Dashboard: Architecture and Implementation with Flink and ClickHouse

The article describes a real‑time customer‑service dashboard built on Flink for streaming MySQL changes captured via Kafka, which cleans and aggregates ~60 operational metrics before writing them to ClickHouse’s MergeTree/ReplacingMergeTree tables, enabling sub‑second queries and exactly‑once guarantees while separating offline and live pipelines.

ClickHouseData PipelineFlink
0 likes · 18 min read
Real-time Customer Service Dashboard: Architecture and Implementation with Flink and ClickHouse
Alibaba Cloud Developer
Alibaba Cloud Developer
Mar 17, 2022 · Big Data

How AutoStream Scales Real‑Time Data Processing with Flink, Iceberg, and PyFlink

This article details AutoStream's evolution from a Java‑only Storm platform to a Flink‑based, Kubernetes‑native streaming system that integrates budgeting controls, automatic scaling, lakehouse architecture with Iceberg, and PyFlink support, highlighting the technical challenges, solutions, and future roadmap for real‑time analytics.

AutoscalingFlinkLakehouse
0 likes · 23 min read
How AutoStream Scales Real‑Time Data Processing with Flink, Iceberg, and PyFlink
Yiche Technology
Yiche Technology
Mar 9, 2022 · Cloud Native

Design and Implementation of the Yunji Logging System Using Flink and ClickHouse

The article presents the Yunji logging system, a Flink+ClickHouse-based cloud-native platform for real-time ingestion, storage, querying, analysis, and monitoring of massive heterogeneous logs, covering its architecture, configuration center, storage design, processing flow, monitoring features, and future enhancements.

ClickHouseCloud NativeFlink
0 likes · 21 min read
Design and Implementation of the Yunji Logging System Using Flink and ClickHouse
Alibaba Cloud Developer
Alibaba Cloud Developer
Mar 7, 2022 · Big Data

How China Mobile’s Real‑Time Computing Platform Scales Billions of Events with Flink

This article details China Mobile (Suzhou) Software Technology's evolution from Storm to Flink for real‑time computing, its multi‑version engine and log‑retrieval designs, signal‑business data pipeline optimizations, stability practices around ZooKeeper, and future directions in resource scaling and data‑lake integration.

FlinkKafkacloud
0 likes · 12 min read
How China Mobile’s Real‑Time Computing Platform Scales Billions of Events with Flink
dbaplus Community
dbaplus Community
Mar 2, 2022 · Big Data

How Real‑Time Data Warehouses Power Modern Business: Architecture, Cases, and Best Practices

This article explores the growing demand for real‑time data warehouses, compares them with traditional offline warehouses, and presents detailed architectures, layer designs, naming conventions, and case studies from companies like Didi, Kuaishou, Tencent, and Youzan, highlighting challenges, solutions, and performance optimizations.

Big Data ArchitectureFlinkKafka
0 likes · 47 min read
How Real‑Time Data Warehouses Power Modern Business: Architecture, Cases, and Best Practices
DataFunTalk
DataFunTalk
Mar 1, 2022 · Cloud Native

Alibaba Cloud Native Data Lake with Apache Iceberg: Architecture, Challenges, and Solutions

The presentation outlines Alibaba Cloud's native data lake solution built on Apache Iceberg, covering data lake fundamentals, cloud migration challenges, Iceberg's architecture and features, real‑time ingestion with Flink, unified metadata management, security guarantees, and testing practices to ensure reliable, scalable big‑data analytics.

Apache IcebergData LakeFlink
0 likes · 16 min read
Alibaba Cloud Native Data Lake with Apache Iceberg: Architecture, Challenges, and Solutions
dbaplus Community
dbaplus Community
Feb 23, 2022 · Big Data

Inside OPPO’s Real‑Time Computing Platform: Architecture, Practices, and Future Roadmap

This article details OPPO’s real‑time computing platform, covering its business scope, big‑data architecture built on Flink, Spark and Trino, the end‑to‑end job development lifecycle, SQL IDE features, diagnostic and monitoring mechanisms, link latency tracking, SLA guarantees, practical use cases, and upcoming lakehouse and cloud‑native evolution.

Data PipelineFlinkReal-Time Computing
0 likes · 23 min read
Inside OPPO’s Real‑Time Computing Platform: Architecture, Practices, and Future Roadmap
vivo Internet Technology
vivo Internet Technology
Feb 23, 2022 · Big Data

Kafka-based Real-Time Data Warehouse: Architecture and Practice for Search

The article explains how Kafka serves as the core of a real‑time data warehouse for search, detailing its advantages over traditional databases, integration with Flink for low‑latency stream processing, architectural patterns such as Lambda/Kappa, scaling challenges, and comprehensive monitoring using Kafka Eagle.

Apache KafkaData IntegrationFlink
0 likes · 15 min read
Kafka-based Real-Time Data Warehouse: Architecture and Practice for Search
Big Data Technology & Architecture
Big Data Technology & Architecture
Feb 23, 2022 · Big Data

Understanding Mini‑Batch Streaming Aggregation in Flink SQL

This article explains Flink SQL’s streaming aggregation Mini‑Batch feature, covering its purpose, configuration parameters, underlying optimizer rules, operator implementations, watermark handling, buffer processing, and the optional Local‑Global two‑phase aggregation optimization for improving throughput and reducing state overhead in large‑scale data pipelines.

AggregationFlinkMini-Batch
0 likes · 10 min read
Understanding Mini‑Batch Streaming Aggregation in Flink SQL
Volcano Engine Developer Services
Volcano Engine Developer Services
Feb 16, 2022 · Big Data

ByteDance’s Journey to a Unified Data Lake with Flink and Hudi

This article recounts ByteDance’s evolution from batch‑only Flink pipelines to a unified data‑lake integration platform, detailing the three integration modes, challenges with Spark‑based CDC, the decision to adopt Hudi over Iceberg, and how Hudi’s indexing and Merge‑On‑Read formats enable near‑real‑time analytics at massive scale.

CDCFlinkHudi
0 likes · 10 min read
ByteDance’s Journey to a Unified Data Lake with Flink and Hudi
Big Data Technology & Architecture
Big Data Technology & Architecture
Feb 16, 2022 · Big Data

Using Flink CDC to Capture MySQL Changes and Sync Them to ClickHouse

This article introduces Change Data Capture (CDC), compares query‑based and log‑based approaches, explains Debezium and ClickHouse, and provides detailed Flink CDC and Flink SQL CDC examples—including Java source code, custom deserialization schema, ClickHouse sink implementation, and required Maven dependencies—to synchronize MySQL data into ClickHouse in real time.

CDCClickHouseData Streaming
0 likes · 17 min read
Using Flink CDC to Capture MySQL Changes and Sync Them to ClickHouse
Alibaba Cloud Developer
Alibaba Cloud Developer
Feb 14, 2022 · Backend Development

How Kuaishou Boosted Flink SQL Performance with Window Extensions and State Optimizations

Kuaishou dramatically increased Flink SQL adoption, introduced Group Window Aggregate and Window TVF extensions, applied aggregation state reuse and mini‑batch techniques, and enhanced stability through data‑skew mitigation and aggregate‑state compatibility, outlining future plans for streaming and batch SQL improvements.

FlinkSQLState Optimization
0 likes · 19 min read
How Kuaishou Boosted Flink SQL Performance with Window Extensions and State Optimizations
DataFunTalk
DataFunTalk
Feb 12, 2022 · Big Data

NetEase Internal Data Lake Project Arctic: Architecture, Requirements, and Future Roadmap

This article introduces NetEase's internally incubated data lake project Arctic, explains the concept of data lakes, outlines NetEase's specific requirements for a unified streaming‑batch platform, details Arctic's core architecture, storage strategy, data‑merge mechanisms, current achievements, and future development plans.

Apache IcebergArcticData Lake
0 likes · 10 min read
NetEase Internal Data Lake Project Arctic: Architecture, Requirements, and Future Roadmap
DataFunTalk
DataFunTalk
Feb 3, 2022 · Big Data

Improving Data Processing Efficiency at Kuaishou with Apache Hudi

This article explains how Kuashou tackled latency and efficiency problems in large‑scale data pipelines by adopting Apache Hudi, detailing the pain points, reasons for choosing Hudi, its architecture, model design, handling of bursty updates, back‑fill scenarios, and operational safeguards.

Data LakeFlinkHudi
0 likes · 13 min read
Improving Data Processing Efficiency at Kuaishou with Apache Hudi
DataFunSummit
DataFunSummit
Jan 30, 2022 · Big Data

Real‑time Data Warehouse at Meituan: Architecture, Challenges, and Solutions

This article presents Meituan's real‑time data warehouse platform, describing typical streaming use cases, the evolution of its architecture from Storm and Spark Streaming to Flink, the challenges of development, operations and data quality, and the engineering solutions—including unified SQL, web IDE, UDF hosting, pipeline testing, and operator performance optimizations—implemented to support large‑scale, low‑latency analytics.

FlinkPlatform ArchitectureReal-time Data
0 likes · 17 min read
Real‑time Data Warehouse at Meituan: Architecture, Challenges, and Solutions
Baidu Geek Talk
Baidu Geek Talk
Jan 26, 2022 · Big Data

How a Real‑Time CDP Solves Data Silos: Architecture, Tech Choices & Lessons

This article examines the design and implementation of a tenant‑level real‑time Customer Data Platform, detailing CDP fundamentals, business and technical challenges, key architectural components, technology selections such as graph databases, stream processing, storage engines, and the operational practices that enable high‑throughput, low‑latency data integration and analytics.

CDPData IntegrationFlink
0 likes · 42 min read
How a Real‑Time CDP Solves Data Silos: Architecture, Tech Choices & Lessons
HomeTech
HomeTech
Jan 26, 2022 · Operations

Design and Practice of Autohome's Performance Testing Platform PTS

The article details the architecture, key components, testing types, and operational results of Autohome's PTS platform, which uses Docker Swarm, gRPC, JMeter, Flume‑Kafka, and Flink to conduct large‑scale distributed load testing for the 818 event and outlines future improvements toward Kubernetes and direct Kafka logging.

Docker SwarmFlinkJMeter
0 likes · 8 min read
Design and Practice of Autohome's Performance Testing Platform PTS
Architecture Digest
Architecture Digest
Jan 21, 2022 · Big Data

Building a Real-Time Data Warehouse with Flink: Architecture, Core Concepts, and Practical Implementation

This article explains how to build a unified stream‑batch real‑time data warehouse using FlinkSQL, covering prerequisite knowledge, five core concepts, two implementation approaches, a comparison of traditional versus real‑time architectures, and a comprehensive hands‑on example, illustrated with diagrams.

Batch ProcessingFlinkReal-time Data Warehouse
0 likes · 6 min read
Building a Real-Time Data Warehouse with Flink: Architecture, Core Concepts, and Practical Implementation
StarRocks
StarRocks
Jan 12, 2022 · Big Data

How Flink + StarRocks Deliver Lightning‑Fast Real‑Time Data Warehousing

This article explains the evolution, challenges, and technical solutions for building an end‑to‑end real‑time data warehouse by combining Apache Flink's stream processing with StarRocks' ultra‑fast OLAP engine, covering architecture, data models, integration methods, best‑practice cases, and future roadmap.

FlinkOLAPReal-time Data Warehouse
0 likes · 21 min read
How Flink + StarRocks Deliver Lightning‑Fast Real‑Time Data Warehousing
DataFunTalk
DataFunTalk
Jan 10, 2022 · Big Data

Real‑Time Data Warehouse at Meituan: Architecture, Challenges, and Solutions

The talk by Tang Chuxi of Meituan explains typical real‑time data scenarios, the challenges faced when building a streaming data warehouse, and the design, development, operation, and performance‑optimisation solutions implemented on a Flink‑based platform to support massive, low‑latency business applications.

FlinkMeituanReal-time Data
0 likes · 17 min read
Real‑Time Data Warehouse at Meituan: Architecture, Challenges, and Solutions
Big Data Technology & Architecture
Big Data Technology & Architecture
Jan 10, 2022 · Big Data

Key Takeaways from Flink Forward 2021: Real‑Time Computing, Flink SQL, ML, and Streaming Warehouse

The article reviews highlights from Flink Forward 2021, describing how real‑time computing is spreading across traditional industries, the unstoppable move toward Flink SQL, the emergence of Flink ML, and the vision of a streaming warehouse built on Flink Dynamic Table technology.

FlinkReal-Time ComputingStreaming Warehouse
0 likes · 8 min read
Key Takeaways from Flink Forward 2021: Real‑Time Computing, Flink SQL, ML, and Streaming Warehouse
dbaplus Community
dbaplus Community
Jan 5, 2022 · Big Data

How ByteDance Optimized Flink SQL for Real‑World Streaming at Scale

This article details ByteDance's practical experience with Apache Flink, covering SQL extensions, a visual SQL platform, performance tweaks such as window mini‑batching and custom windows, join and checkpoint recovery improvements, stream‑batch integration experiments, and future roadmap plans.

Batch IntegrationCheckpointFlink
0 likes · 16 min read
How ByteDance Optimized Flink SQL for Real‑World Streaming at Scale
DataFunTalk
DataFunTalk
Jan 1, 2022 · Big Data

JD's Flink Journey: Evolution, Optimizations, and Future Directions

This article details JD's adoption of Flink for real‑time computing, covering its evolution from Storm to Flink on Kubernetes, the platform architecture, major optimization techniques such as preview topology, backpressure handling, dynamic rebalance, checkpoint‑as‑savepoint, and outlines future plans including stream‑batch integration, stability improvements, intelligent operations, and AI integration.

FlinkJDKubernetes
0 likes · 10 min read
JD's Flink Journey: Evolution, Optimizations, and Future Directions
Big Data Technology & Architecture
Big Data Technology & Architecture
Dec 31, 2021 · Big Data

Apache SeaTunnel Joins the Apache Incubator: Overview, Features, and Real‑World Use Cases

SeaTunnel, the China‑originated data‑integration platform built on Spark and Flink, has been accepted into the Apache Incubator, and this article introduces its history, architecture, plugin ecosystem, deployment requirements, and numerous enterprise deployments across batch and streaming big‑data scenarios.

ApacheData IntegrationETL
0 likes · 7 min read
Apache SeaTunnel Joins the Apache Incubator: Overview, Features, and Real‑World Use Cases
Tencent Cloud Developer
Tencent Cloud Developer
Dec 28, 2021 · Industry Insights

How Flink and ClickHouse Combine to Build High‑Performance Real‑Time Data Warehouses

This article analyzes the challenges of massive data query efficiency, explains how Flink's stream processing and ClickHouse's OLAP engine complement each other, and presents a layered real‑time data‑warehouse architecture with practical guidance on data ingestion, write strategies, quality assurance, and evolving batch‑stream integration patterns.

ClickHouseFlinkOLAP
0 likes · 19 min read
How Flink and ClickHouse Combine to Build High‑Performance Real‑Time Data Warehouses
Big Data Technology & Architecture
Big Data Technology & Architecture
Dec 22, 2021 · Big Data

Using Flink CDC to Capture MySQL Changes and Sink Them into ClickHouse

This article explains Change Data Capture (CDC), compares query‑based and log‑based approaches, introduces Debezium and ClickHouse, and provides step‑by‑step Flink CDC and Flink SQL CDC examples—including Java source, deserialization, sink code and required Maven dependencies—to stream MySQL binlog changes into ClickHouse for real‑time analytics.

CDCClickHouseData Streaming
0 likes · 14 min read
Using Flink CDC to Capture MySQL Changes and Sink Them into ClickHouse
HelloTech
HelloTech
Dec 20, 2021 · Big Data

Building an ElasticSearch-based Search Platform for Ride-Hailing: Architecture, Data Synchronization, and Performance Optimization

Hello Mobility unified its fragmented ElasticSearch clusters into a single, real‑time search platform—leveraging Kafka‑driven CDC, Flink stream processing, custom ES plugins, and extensive performance tuning—to deliver scalable matching, recommendation and voice services, ultimately raising completed orders by 49.8 % and driver acceptance by 37 %.

FlinkSearch Platformbig data
0 likes · 19 min read
Building an ElasticSearch-based Search Platform for Ride-Hailing: Architecture, Data Synchronization, and Performance Optimization
DataFunTalk
DataFunTalk
Dec 19, 2021 · Big Data

OPPO Real-Time Computing Platform Architecture and Practices

This article details OPPO’s real-time computing platform architecture, covering its background, open‑source and self‑developed components, job lifecycle, SQL IDE, diagnostic and monitoring mechanisms, SLA guarantees, practical applications such as real‑time warehousing and dashboards, and future plans for lakehouse integration and cloud‑native deployment.

FlinkPlatform Architecturecloud-native
0 likes · 20 min read
OPPO Real-Time Computing Platform Architecture and Practices
Youzan Coder
Youzan Coder
Dec 17, 2021 · Big Data

Upgrading Real-Time Computing Engine from Flink 1.10 to 1.13: Practices and Challenges

Youzan upgraded its real‑time computing engine from Flink 1.10 to 1.13 to meet rising SQL and containerization demands, gaining enhanced SQL syntax, time‑function handling, Window TVF standardization, Hive integration, K8s stability, elastic scaling, richer Kafka and format support, improved metrics and debugging tools, and successfully migrated all custom connectors, UDFs, and SQL jobs to the new Kubernetes‑based platform.

FlinkK8sReal-Time Computing
0 likes · 22 min read
Upgrading Real-Time Computing Engine from Flink 1.10 to 1.13: Practices and Challenges
Beike Product & Technology
Beike Product & Technology
Dec 17, 2021 · Operations

Practices for Monitoring, Resource Optimization, and Containerization of Large-Scale Flink Jobs at Beike

This article describes Beike's real‑time computing team's end‑to‑end practices for collecting and storing Flink metrics, building visual monitoring dashboards, implementing multi‑level alerting, analyzing logs, estimating CPU and memory resources, and deploying Flink on Kubernetes with containerization and storage separation to improve stability, resource utilization, and operational efficiency.

FlinkKubernetesResource Optimization
0 likes · 25 min read
Practices for Monitoring, Resource Optimization, and Containerization of Large-Scale Flink Jobs at Beike
dbaplus Community
dbaplus Community
Dec 13, 2021 · Backend Development

ElasticSearch Powers Ride‑Matching at Haro Mobility: Architecture & Lessons

This article details how Haro Mobility built a search‑driven ride‑matching platform using ElasticSearch and Flink, covering the business background, architectural evolution, data‑sync challenges, performance tuning, stability measures, and the resulting improvements in order completion and user engagement.

FlinkPerformance OptimizationRide Matching
0 likes · 21 min read
ElasticSearch Powers Ride‑Matching at Haro Mobility: Architecture & Lessons
HelloTech
HelloTech
Dec 13, 2021 · Big Data

Smart Matching Engine for Ride-Sharing: Technical Implementation and Algorithms

The Smart Matching Engine for Haolo’s ride‑sharing service ingests driver and passenger orders via Kafka‑Flink pipelines into Elasticsearch, then applies multi‑stage matching—nearby search, itinerary‑based filtering using ETA, angle, distance, route‑similarity and shared‑mileage calculations—and finally ranks results with evolving pre‑sorting and algorithmic models, including PMML and deep‑learning, to optimize driver‑passenger pairing.

ElasticsearchFlinkKafka
0 likes · 9 min read
Smart Matching Engine for Ride-Sharing: Technical Implementation and Algorithms
DataFunTalk
DataFunTalk
Dec 10, 2021 · Big Data

Building and Evolving NetEase Yanxuan Real-Time Computing Platform: Architecture, SQLization, Serviceization, and Data Governance

This article details NetEase Yanxuan's real-time computing platform development from 2017 to present, covering its architecture, Flink‑SQL development environment, service‑oriented deployment, resource optimization, cloud‑native migration, comprehensive data governance, and future plans for stream‑batch integration and intelligent job diagnostics.

Cloud NativeFlinkReal-Time Computing
0 likes · 14 min read
Building and Evolving NetEase Yanxuan Real-Time Computing Platform: Architecture, SQLization, Serviceization, and Data Governance
DataFunSummit
DataFunSummit
Dec 10, 2021 · Big Data

Real‑Time Platform Construction at NetEase Yanxuan: Architecture, SQL‑Based Streaming, Serviceization, and Data Governance

This article details NetEase Yanxuan's evolution of a real‑time data platform from 2017 to present, covering background, current scale, layered architecture, Flink‑SQL development IDE, service‑oriented task execution, resource‑optimizing deployment modes, cloud‑native migration, comprehensive data governance, and future batch‑stream integration plans.

Cloud NativeFlinkbig data
0 likes · 15 min read
Real‑Time Platform Construction at NetEase Yanxuan: Architecture, SQL‑Based Streaming, Serviceization, and Data Governance
Youzan Coder
Youzan Coder
Dec 8, 2021 · Big Data

How to Build a Real‑Time Data Quality Monitoring System with Flink

This article outlines a comprehensive approach to monitoring and ensuring the accuracy and timeliness of real‑time data streams, detailing background challenges, solution design, implementation steps using Flink and automated testing, alert handling procedures, and future improvement plans.

Data QualityFlinkalerting
0 likes · 10 min read
How to Build a Real‑Time Data Quality Monitoring System with Flink
HomeTech
HomeTech
Dec 7, 2021 · Big Data

Flink Task Auto-scaling Design and Implementation

This article presents the design and implementation of Flink task auto‑scaling, covering background, manual and automatic scaling mechanisms, architecture with RescaleCoordinator, persistence via Zookeeper and HDFS, scaling policies for parallelism, CPU and memory, and future plans for fine‑grained and time‑based resource adjustments.

FlinkHDFSZooKeeper
0 likes · 4 min read
Flink Task Auto-scaling Design and Implementation
Alibaba Cloud Developer
Alibaba Cloud Developer
Nov 30, 2021 · Big Data

Scaling Real‑Time Data Warehousing for Double‑11: Flink + Hologres in Action

During the 2021 Double‑11 shopping festival, logistics provider DiSiFang upgraded its real‑time data warehouse with Flink and Hologres, enabling multi‑billion‑row joins, cutting costs by 50%, and delivering stable, low‑latency analytics that powered high‑frequency dashboards and improved overall delivery speed.

Cloud ComputingFlinkHologres
0 likes · 13 min read
Scaling Real‑Time Data Warehousing for Double‑11: Flink + Hologres in Action
StarRocks
StarRocks
Nov 26, 2021 · Big Data

How Autohome Achieved Sub‑Second Real‑Time Analytics with StarRocks

Autohome replaced Flink and Kylin with StarRocks to power sub‑second real‑time OLAP analytics, detailing data sources, pain points, benchmark comparisons against Apache Kylin, ClickHouse, Presto, Spark, and Doris, integration with Flink‑connector, broker‑load scripts, monitoring setup, and lessons learned from large‑scale deployments.

FlinkOLAPStarRocks
0 likes · 12 min read
How Autohome Achieved Sub‑Second Real‑Time Analytics with StarRocks
HomeTech
HomeTech
Nov 24, 2021 · Databases

Real‑Time Data Analysis at AutoHome: Evaluation and Adoption of StarRocks

This article describes AutoHome's real‑time data analysis architecture, the challenges of existing OLAP solutions, the reasons for choosing StarRocks, detailed performance comparisons with Kylin, ClickHouse, Doris, Presto and Spark, and the practical integration of StarRocks with Flink, broker‑load scripts, and monitoring tools.

FlinkOLAPReal-time Analytics
0 likes · 9 min read
Real‑Time Data Analysis at AutoHome: Evaluation and Adoption of StarRocks
Big Data Technology Architecture
Big Data Technology Architecture
Nov 23, 2021 · Big Data

Step-by-Step Guide to Setting Up Flink CDC with MySQL, Hudi, and Hive Integration on a Hadoop Cluster

This comprehensive tutorial walks through configuring a Hadoop‑based environment (Flink 1.13.1, Scala 2.11, CDH 6.2.0, Hive 2.1.1, Hudi 0.10), compiling Hudi, setting up Flink and MySQL binlog, creating CDC source and Hudi sink tables, running Flink jobs, and synchronizing the results to Hive partitions for query via Hive and Presto.

CDCFlinkHive
0 likes · 15 min read
Step-by-Step Guide to Setting Up Flink CDC with MySQL, Hudi, and Hive Integration on a Hadoop Cluster
Big Data Technology & Architecture
Big Data Technology & Architecture
Nov 22, 2021 · Big Data

Comprehensive Big Data Learning Path and Resource Guide

This article presents a detailed learning roadmap for aspiring big‑data experts, covering foundational programming languages, data structures, Linux basics, databases, distributed system theory, and essential frameworks such as Hadoop, Spark, Flink, Kafka, and provides curated B‑site video links and reference materials.

FlinkHadoopKafka
0 likes · 9 min read
Comprehensive Big Data Learning Path and Resource Guide
Alibaba Cloud Developer
Alibaba Cloud Developer
Nov 22, 2021 · Big Data

Achieving Exactly-Once Writes from Flink to ClickHouse: Architecture and Performance

This article explains how Flink and ClickHouse can be combined to build a real-time data warehouse with end-to-end Exactly-Once guarantees, detailing the underlying write mechanisms, transaction state machine, connector implementation, and performance test results, while also outlining future enhancements for distributed transactions.

ClickHouseFlinkperformance testing
0 likes · 15 min read
Achieving Exactly-Once Writes from Flink to ClickHouse: Architecture and Performance
Big Data Technology & Architecture
Big Data Technology & Architecture
Nov 20, 2021 · Big Data

Comprehensive Overview of Apache Flink Concepts, Mechanisms, and Interview Questions

This article provides an extensive technical guide to Apache Flink, covering its exactly‑once consumption guarantees, checkpoint and two‑phase commit mechanisms, differences from Spark, state backends, watermark handling, time semantics, window joins, CEP, backpressure, architecture layers, deployment, resource management, and common operational issues.

CEPCheckpointFlink
0 likes · 77 min read
Comprehensive Overview of Apache Flink Concepts, Mechanisms, and Interview Questions
Tencent Cloud Developer
Tencent Cloud Developer
Nov 19, 2021 · Artificial Intelligence

End‑to‑End Breast Cancer Prediction Solution Using Decision Tree on Tencent Cloud AI Platform

This guide details an end‑to‑end breast‑cancer prediction pipeline on Tencent Cloud, covering offline decision‑tree training with TI‑ONE, model packaging as a PMML service, real‑time feature generation via Oceanus and CKafka, and live inference stored in ClickHouse, all within a secure VPC.

AIFlinkTencent Cloud
0 likes · 19 min read
End‑to‑End Breast Cancer Prediction Solution Using Decision Tree on Tencent Cloud AI Platform
Big Data Technology & Architecture
Big Data Technology & Architecture
Nov 16, 2021 · Big Data

Flink Checkpoint, Backpressure, and Memory Tuning Guide

This article provides a comprehensive guide on optimizing Flink checkpoints, diagnosing and alleviating backpressure, and fine‑tuning memory configurations—including process, heap, off‑heap, managed, and network memory—to improve job stability and performance in large‑scale streaming applications.

CheckpointFlinkMemory Tuning
0 likes · 25 min read
Flink Checkpoint, Backpressure, and Memory Tuning Guide
Big Data Technology & Architecture
Big Data Technology & Architecture
Nov 8, 2021 · Big Data

Why Choose Apache Iceberg? Tencent’s Optimizations and Real‑World Practices

This article examines the strengths and weaknesses of Apache Iceberg, explains why Tencent selected it over alternatives, details Tencent’s own enhancements and integration with Flink, Spark, and other engines, and shares multiple real‑world implementations for building enterprise‑grade real‑time data lakes.

Apache IcebergData LakeFlink
0 likes · 17 min read
Why Choose Apache Iceberg? Tencent’s Optimizations and Real‑World Practices
Big Data Technology & Architecture
Big Data Technology & Architecture
Nov 8, 2021 · Big Data

Understanding Flink CDC 2.0: Core Design, Snapshot & Incremental Reading, and Code Walkthrough

This article introduces Flink CDC 2.0, explains its distributed full‑load and incremental reading mechanisms, details the slice partitioning, snapshot correction, and binlog handling logic, and provides a complete Java example that demonstrates how to configure Flink SQL, MySQL source, and Kafka sink.

CDCData IntegrationDebezium
0 likes · 29 min read
Understanding Flink CDC 2.0: Core Design, Snapshot & Incremental Reading, and Code Walkthrough
DataFunTalk
DataFunTalk
Nov 6, 2021 · Artificial Intelligence

Elastic Federated Learning Solution (EFLS): Project Overview, Architecture, and Technical Implementation

The article introduces Alibaba's Elastic Federated Learning Solution (EFLS), describing its business motivations, core functionalities, system architecture, sample‑set intersection, federated training pipeline, novel algorithms, product console, and future roadmap for privacy‑preserving advertising in large‑scale sparse scenarios.

AdvertisingFlinkKubernetes
0 likes · 18 min read
Elastic Federated Learning Solution (EFLS): Project Overview, Architecture, and Technical Implementation
HomeTech
HomeTech
Nov 3, 2021 · Big Data

Real‑time Materialized View Practices with Apache Flink: System Analysis, Algorithm Design, and Implementation

This article presents Car Home's experience building a real‑time materialized view system on Apache Flink, detailing system analysis, problem decomposition, a global‑version‑based CDC algorithm, its implementation as a Flink connector, practical deployment results, and remaining challenges such as clock dependency and state size.

CDCFlinkMaterialized View
0 likes · 17 min read
Real‑time Materialized View Practices with Apache Flink: System Analysis, Algorithm Design, and Implementation
Big Data Technology & Architecture
Big Data Technology & Architecture
Oct 29, 2021 · Big Data

Dimension Table Join Strategies in Apache Flink: Preload, Distributed Cache, Hot Storage, Broadcast, and Temporal Table Function

The article explains various dimension‑table join approaches in Apache Flink, including preloading tables into memory, using distributed cache, leveraging hot storage with async I/O, broadcasting state, and temporal table function joins, and compares their trade‑offs for different data volumes and update frequencies.

Dimension TableFlinkJOIN
0 likes · 10 min read
Dimension Table Join Strategies in Apache Flink: Preload, Distributed Cache, Hot Storage, Broadcast, and Temporal Table Function
Alimama Tech
Alimama Tech
Oct 27, 2021 · Artificial Intelligence

Elastic Federated Learning Solution (EFLS): Architecture, Core Functions, and Technical Details

The Elastic Federated Learning Solution (EFLS) is Alibaba’s open‑source platform that enables privacy‑preserving vertical and horizontal federated learning for large‑scale sparse advertising, offering data‑intersection, high‑performance C++ training, a visual console, novel aggregation algorithms, and a roadmap toward multi‑party scaling and advanced encryption.

AdvertisingElastic Federated LearningFlink
0 likes · 16 min read
Elastic Federated Learning Solution (EFLS): Architecture, Core Functions, and Technical Details
Big Data Technology & Architecture
Big Data Technology & Architecture
Oct 26, 2021 · Big Data

Practical Experience Building a Real‑Time Clickstream Data Warehouse with Flink and ClickHouse

This article shares practical insights on designing and operating a real‑time clickstream data warehouse using Flink for streaming processing and ClickHouse for near‑real‑time OLAP, covering dimensional modeling, layered architecture, Flink‑ClickHouse sink implementation, and data rebalancing strategies.

ClickHouseFlinkReal-time Analytics
0 likes · 10 min read
Practical Experience Building a Real‑Time Clickstream Data Warehouse with Flink and ClickHouse