BestHub
Sign in
Tagged articles

Flink

979 articles · Page 1 of 10
Ray's Galactic Tech
Ray's Galactic Tech
Aug 7, 2026 · Databases

OLAP Engine Selection for Billion‑Row Analytics: Doris, ClickHouse, StarRocks

This guide compares Doris, ClickHouse, and StarRocks for billion‑row real‑time analytics, outlining their strengths and weaknesses across write throughput, update handling, join performance, concurrency, cost, and operational complexity, and provides concrete modeling, ingestion, query governance, and fault‑tolerance recommendations to help teams choose the most suitable OLAP platform.

ClickHouseDorisFlink
0 likes · 32 min read
OLAP Engine Selection for Billion‑Row Analytics: Doris, ClickHouse, StarRocks
Alibaba Cloud Big Data AI Platform
Alibaba Cloud Big Data AI Platform
Aug 1, 2026 · Big Data

Designing a Differentiated DSP Near‑Real‑Time Data Warehouse with Flink, DLF Paimon, and EMR Serverless StarRocks

The DSP advertising data pipeline was re‑architected by splitting three distinct links—NRR, BT, and CT—and assigning DLF Paimon for massive low‑frequency data, EMR Serverless StarRocks primary tables for high‑value low‑latency queries, and unified service‑layer merging, achieving 2‑minute freshness, sub‑5 ms point queries, ~60% storage cost reduction, and improved fault isolation.

DLF PaimonDSPEMR Serverless
0 likes · 12 min read
Designing a Differentiated DSP Near‑Real‑Time Data Warehouse with Flink, DLF Paimon, and EMR Serverless StarRocks
YiSu Grain
YiSu Grain
Jul 28, 2026 · Big Data

Day 39: Big Data Architecture – Distributed Storage, Batch vs Stream Processing, and Compute‑Storage Separation

This lesson explains why a single database cannot scale for massive e‑commerce data, introduces the three core pillars of distributed storage—sharding, replication, and horizontal scaling—covers batch and stream processing differences with Spark, Hive, Flink and Storm, compares compute‑storage integration versus separation, and shows how Kafka, HDFS, Spark and Flink fit together in a real‑time data platform.

Batch ProcessingBig DataCompute-Storage Separation
0 likes · 25 min read
Day 39: Big Data Architecture – Distributed Storage, Batch vs Stream Processing, and Compute‑Storage Separation
Smart Sea Tide
Smart Sea Tide
Jul 27, 2026 · Big Data

Comprehensive Overview of Big Data Open‑Source Frameworks

This article provides a detailed, structured survey of the most widely used open‑source big‑data technologies—including Hadoop ecosystems, storage systems, processing engines, query tools, data ingestion, exchange, messaging, scheduling, governance, visualization, mining, and cloud platforms—highlighting each project's origins, core features, typical use cases, and notable strengths or limitations to aid technology selection and system design.

Big DataFlinkHBase
0 likes · 59 min read
Comprehensive Overview of Big Data Open‑Source Frameworks
DataFunTalk
DataFunTalk
Jul 14, 2026 · Big Data

How Xiaohongshu Re‑engineered Its Data Architecture for the Big AI Data Era

Xiaohongshu transformed its data platform from a simple ClickHouse‑based stack to a Lambda‑enhanced architecture and finally to a Lakehouse with incremental compute, cutting architecture complexity, resource and development costs by two‑thirds while delivering second‑level analytics on petabyte‑scale data.

Big DataClickHouseFlink
0 likes · 22 min read
How Xiaohongshu Re‑engineered Its Data Architecture for the Big AI Data Era
dbaplus Community
dbaplus Community
Jul 13, 2026 · Interview Experience

How to Process a 10 GB CSV File with Only 512 MB RAM?

The article walks through an interview scenario where a candidate must handle a 10 GB CSV using just 512 MB of memory, critiques naive answers, and presents a step‑by‑step streaming, chunked, and checkpoint‑based solution with optional Kafka/Flink considerations.

CSVFlinkInterview
0 likes · 8 min read
How to Process a 10 GB CSV File with Only 512 MB RAM?
Niu Liu
Niu Liu
Jul 13, 2026 · Artificial Intelligence

Building a Real‑Time Recommendation Engine with Flink: A Complete Example Project

The article walks through constructing a full‑stack real‑time recommendation system—from user‑behavior collection via Kafka, through Flink streaming jobs for hot‑list, user and item profiling, to storage in Redis, HBase and Elasticsearch, and finally a React/Ant Design console that visualizes the pipeline and enables debugging.

ElasticsearchFlinkHBase
0 likes · 12 min read
Building a Real‑Time Recommendation Engine with Flink: A Complete Example Project
DataFunTalk
DataFunTalk
Jul 9, 2026 · Artificial Intelligence

How Flink Is Rebuilding Itself for AI Agents

At Flink Forward Asia 2026, experts argued that real‑time computing is undergoing a fundamental identity shift: Flink is evolving from a batch‑stream engine into the core infrastructure for AI agents, driven by data gravity, agentic streaming, GPU acceleration, and a unified data lake.

AI AgentAgentic StreamingApache Paimon
0 likes · 13 min read
How Flink Is Rebuilding Itself for AI Agents
DataFunTalk
DataFunTalk
Jul 6, 2026 · Big Data

How Xiaohongshu Evolved Its Data Architecture for the Big AI Data Era

Xiaohongshu transformed its data platform from a simple ClickHouse‑based ad‑hoc analysis system to a Lambda‑style architecture and finally to a lakehouse with incremental compute, cutting architecture complexity, resource and development costs by one‑third while delivering second‑level queries over petabyte‑scale data.

Big DataClickHouseFlink
0 likes · 23 min read
How Xiaohongshu Evolved Its Data Architecture for the Big AI Data Era
dbaplus Community
dbaplus Community
Jul 2, 2026 · Fundamentals

Production Hit by Silent Data Corruption: JDK 25 G1GC Bug Explained

A rare silent data‑corruption bug in JDK 25’s G1GC caused Parquet and ORC files written by Spark and Flink to become unreadable, prompting a multi‑stage investigation that traced the issue to an optional evacuation flaw affecting JNI‑pinned objects, which was later back‑ported and fixed in the OpenJDK community.

AI debuggingFlinkG1GC
0 likes · 20 min read
Production Hit by Silent Data Corruption: JDK 25 G1GC Bug Explained
DataFunSummit
DataFunSummit
Jul 1, 2026 · Artificial Intelligence

How Bailei Knowledge Base Uses Flink and DLF (Paimon) to Build an Enterprise‑Scale Full‑Modal RAG System

Bailei Knowledge Base delivers an enterprise‑grade, full‑modal Retrieval‑Augmented Generation solution covering documents, tables, images and audio‑video, powered by Flink's high‑throughput streaming for billions of daily document indexes and DLF/Paimon’s three‑layer reliable backup, achieving sub‑200 ms latency and 99.99% availability.

DLFEnterprise AIFlink
0 likes · 26 min read
How Bailei Knowledge Base Uses Flink and DLF (Paimon) to Build an Enterprise‑Scale Full‑Modal RAG System
DataFunTalk
DataFunTalk
Jun 30, 2026 · Big Data

How Xiaohongshu Evolved Its Data Architecture for the Big AI Data Era

Xiaohongshu, with over 3.5 billion monthly users and daily logs in the trillions, migrated 500 PB of data to Alibaba Cloud and iterated its data platform through four architecture generations—ClickHouse‑based ad‑hoc, Lambda, Lakehouse, and a unified incremental compute model—cutting resource, development, and storage costs to one‑third while delivering sub‑10‑second query latency at petabyte scale.

Big DataClickHouseFlink
0 likes · 22 min read
How Xiaohongshu Evolved Its Data Architecture for the Big AI Data Era
Niu Liu
Niu Liu
Jun 29, 2026 · Big Data

Designing a Real‑Time Data Warehouse Backend with Flink and Paimon for SMBs

The article walks through the architecture and core workflow of a lightweight real‑time data warehouse built on Flink and Paimon, covering service‑layer refactoring, CDC task lifecycle, dual Flink submission modes, metadata synchronization, and layered security measures for small‑to‑medium enterprises.

Backend ArchitectureCDCFlink
0 likes · 11 min read
Designing a Real‑Time Data Warehouse Backend with Flink and Paimon for SMBs
DataFunTalk
DataFunTalk
Jun 24, 2026 · Big Data

How Xiaohongshu Re‑engineered Its Data Architecture for the Big AI Data Era

Xiaohongshu, with over 350 million monthly users and daily logs in the billions, migrated its data platform from AWS to Alibaba Cloud and iterated four times—from a ClickHouse‑based ad‑hoc layer to a Lambda architecture and finally a Lakehouse with incremental compute—cutting architecture complexity, resource cost and development effort each to about one‑third while delivering second‑level analytics on trillion‑scale data.

Big DataClickHouseFlink
0 likes · 22 min read
How Xiaohongshu Re‑engineered Its Data Architecture for the Big AI Data Era
DataFunTalk
DataFunTalk
Jun 20, 2026 · Big Data

How Xiaohongshu Evolved Its Data Architecture for the Big AI Data Era

The article details Xiaohongshu's step‑by‑step migration from a simple ClickHouse‑based analytics stack to a Lambda‑style 2.0 architecture and finally to a Lakehouse‑based 3.0 design, highlighting concrete performance numbers, cost reductions, and the definition of a generic incremental‑compute model (SPOT) that underpins the evolution.

Big DataClickHouseFlink
0 likes · 22 min read
How Xiaohongshu Evolved Its Data Architecture for the Big AI Data Era
Niu Liu
Niu Liu
Jun 19, 2026 · Big Data

Building a Real‑Time Risk Control Engine with Flink 2.2.1, CEP, and Aviator

This article details a real‑time risk control system for e‑commerce and finance built on Apache Flink 2.2.1 and CEP, featuring a dynamic Aviator rule engine, three Kafka event streams, multi‑channel output to Redis, MySQL and Kafka, a Spring Boot‑React management UI, and step‑by‑step deployment instructions.

AviatorCEPFlink
0 likes · 11 min read
Building a Real‑Time Risk Control Engine with Flink 2.2.1, CEP, and Aviator
DataFunTalk
DataFunTalk
Jun 11, 2026 · Artificial Intelligence

How Qichacha Leverages Large Language Models for Field‑Level Data Lineage

This article details Qichacha's use of large language models to extract field‑level data lineage from heterogeneous, non‑standard code and ETL assets, describing the motivation, architectural blueprint, practical challenges such as cost, accuracy and hallucination, and the resulting improvements in impact analysis, metric tracing, and sensitive‑data governance.

Big DataFlinkLLM
0 likes · 11 min read
How Qichacha Leverages Large Language Models for Field‑Level Data Lineage
DataFunSummit
DataFunSummit
Jun 7, 2026 · Artificial Intelligence

How Qichacha Uses Large Language Models for Field‑Level Data Lineage

This article details Qichacha's technical journey of applying large language models to resolve field‑level data lineage challenges in a complex, multi‑source data environment, describing the motivation, architecture, practical implementation, engineering trade‑offs, and measurable outcomes.

AIBig DataFlink
0 likes · 11 min read
How Qichacha Uses Large Language Models for Field‑Level Data Lineage
DataFunTalk
DataFunTalk
May 28, 2026 · Big Data

How Xiaohongshu Evolved Its Data Architecture for the Big AI Data Era

Xiaohongshu transformed its data platform from a simple ClickHouse‑based ad‑hoc analysis to a Lambda‑style architecture and finally to a lakehouse with generic incremental compute, cutting architecture complexity, resource and development costs by one‑third while delivering second‑level queries over trillions of rows.

Big DataClickHouseFlink
0 likes · 21 min read
How Xiaohongshu Evolved Its Data Architecture for the Big AI Data Era
DataFunTalk
DataFunTalk
May 22, 2026 · Big Data

How Xiaohongshu Cut Data Architecture Complexity and Cost by One‑Third in the Big AI Data Era

The article details Xiaohongshu's evolution from a simple ClickHouse‑based analytics layer to a Lambda‑enabled 2.0 stack and finally a Lakehouse‑based 3.0 architecture, showing how each iteration reduced infrastructure complexity, resource consumption and development effort by roughly one‑third while supporting trillions of daily events and AI‑driven use cases.

Big DataClickHouseFlink
0 likes · 21 min read
How Xiaohongshu Cut Data Architecture Complexity and Cost by One‑Third in the Big AI Data Era
DataFunTalk
DataFunTalk
May 11, 2026 · Big Data

How Xiaohongshu Re‑engineered Its Data Architecture for the Big AI Data Era

Xiaohongshu transformed its data platform from a simple ClickHouse‑based ad‑hoc analysis to a Lambda‑style architecture and finally to a lakehouse built on Iceberg, StarRocks, Flink and Spark, cutting architecture complexity, resource and development costs by two‑thirds while supporting trillions of daily events with sub‑second query latency.

Big DataClickHouseFlink
0 likes · 22 min read
How Xiaohongshu Re‑engineered Its Data Architecture for the Big AI Data Era
DataFunTalk
DataFunTalk
May 6, 2026 · Big Data

How Xiaohongshu Evolved Its Data Architecture for the Big AI Data Era

The article details Xiaohongshu's four‑stage data‑platform evolution—from a simple ClickHouse ad‑hoc setup to a Lambda‑based 2.0 design and finally a lakehouse‑driven 3.0 architecture—highlighting the adoption of general incremental compute, cost‑reduction to one‑third, performance gains of up to ten‑fold, and the SPOT standards that guide the new system.

Big DataClickHouseFlink
0 likes · 21 min read
How Xiaohongshu Evolved Its Data Architecture for the Big AI Data Era
DataFunTalk
DataFunTalk
Apr 29, 2026 · Big Data

How Xiaohongshu Revamped Its Data Architecture for the Big AI Data Era

Xiaohongshu transformed its data platform from a simple ClickHouse‑based analytics stack to a unified lakehouse with generic incremental compute, cutting architecture complexity, resource cost, and development effort by roughly one‑third while supporting petabyte‑scale, sub‑second queries across its 350 million‑user app.

Big DataClickHouseFlink
0 likes · 22 min read
How Xiaohongshu Revamped Its Data Architecture for the Big AI Data Era
Lao Guo's Learning Space
Lao Guo's Learning Space
Apr 29, 2026 · Big Data

Designing a Full-Stack Credit Data System: From Ingestion to Real-Time Decision

The article dissects a credit data system architecture, detailing six logical layers—from multi-source data collection and feature engineering (including graph features and feature stores) to model training, real‑time stream processing, decision engine integration, and privacy‑preserving computation—while explaining the trade‑offs, tools, and performance targets needed for accurate, low‑latency risk assessment.

Credit ScoringData PipelineFeature Store
0 likes · 16 min read
Designing a Full-Stack Credit Data System: From Ingestion to Real-Time Decision
Alibaba Cloud Big Data AI Platform
Alibaba Cloud Big Data AI Platform
Apr 27, 2026 · Information Security

Real-Time Agentic Risk Detection with Flink, Fluss, and Large Language Models

The article presents a Flink‑Fluss‑LLM architecture that captures full‑link agent events via a non‑intrusive hook, combines semantic AI inference with deterministic CEP rules, and delivers millisecond‑level alerts for malicious user detection, tool result poisoning, and chain‑attack risk mitigation.

AI FunctionAgent SecurityFlink
0 likes · 41 min read
Real-Time Agentic Risk Detection with Flink, Fluss, and Large Language Models
Big Data Technology & Architecture
Big Data Technology & Architecture
Apr 24, 2026 · Artificial Intelligence

A Deep Dive into Flink Agents: Architecture, Roadmap, and Upcoming Features

The article explains Flink Agents' current 0.3 preview, detailing its layered architecture—from Agent definition to execution plan and runtime operators—while outlining the roadmap for Skills integration, Mem0 long‑term memory, durable execution, and observability enhancements aimed at production readiness.

AI agentsAgentPlanFlink
0 likes · 7 min read
A Deep Dive into Flink Agents: Architecture, Roadmap, and Upcoming Features
Lobster Programming
Lobster Programming
Apr 8, 2026 · Big Data

How to Implement Real‑Time API Traffic Counting at Scale

This article compares three practical approaches—direct database storage, a Flink‑Kafka‑Redis‑Grafana pipeline, and an ELK stack—to achieve real‑time API request counting for high‑concurrency scenarios, outlining their architectures, advantages, and trade‑offs.

API analyticsELKFlink
0 likes · 6 min read
How to Implement Real‑Time API Traffic Counting at Scale
Alibaba Cloud Observability
Alibaba Cloud Observability
Apr 6, 2026 · Cloud Native

How Alibaba Cloud Built Real‑Time OpenAPI Monitoring with Flink + SLS

This article details the design and implementation of a cloud‑native, real‑time monitoring system for Alibaba Cloud OpenAPI, covering background challenges, a Flink‑SLS architecture, multi‑region data processing, checkpoint and state‑backend tuning, source‑side predicate pushdown, visualization with Grafana, and production results.

Big DataCloud NativeFlink
0 likes · 21 min read
How Alibaba Cloud Built Real‑Time OpenAPI Monitoring with Flink + SLS
Big Data Tech Team
Big Data Tech Team
Apr 1, 2026 · Big Data

Why Your 2026 Big Data Resume Is Being Ignored and How to Fix It

In the 2026 spring hiring season, many big‑data job seekers see their resumes disappear because they still focus on offline batch processing, while employers now demand real‑time streaming, AI‑driven data pipelines, and cloud‑native deployment skills such as Flink, vector databases, and Kubernetes.

AI integrationBig DataCloud Native
0 likes · 7 min read
Why Your 2026 Big Data Resume Is Being Ignored and How to Fix It
Cloud Architecture
Cloud Architecture
Mar 29, 2026 · Backend Development

Smart Parking Guidance Using Kafka, Flink, and Spring Boot

This article presents a senior architect’s end‑to‑end design of a smart parking guidance system, detailing how high‑frequency sensor streams are ingested via Spring Boot gateways, processed with Kafka and Flink for stateful de‑duplication, debouncing, and aggregation, and finally served through Redis, PostgreSQL and Spring Boot APIs for real‑time vehicle routing.

FlinkSmart ParkingSpring Boot
0 likes · 35 min read
Smart Parking Guidance Using Kafka, Flink, and Spring Boot
TechVision Expert Circle
TechVision Expert Circle
Mar 17, 2026 · Industry Insights

How Leading CTOs Build an End-to-End Data Pipeline from Collection to AI‑Driven Insight

The article outlines a full‑stack data architecture for 2026, covering edge‑preprocessed collection, unified observability with OpenTelemetry, CDC‑based real‑time sync, a Kafka‑Flink‑Iceberg processing stack, Data Mesh governance, and AI‑Agent powered insight, while offering practical adoption paths and pitfalls to avoid.

AI AgentCDCData Mesh
0 likes · 14 min read
How Leading CTOs Build an End-to-End Data Pipeline from Collection to AI‑Driven Insight
Architect-Kip
Architect-Kip
Mar 2, 2026 · Big Data

How to Build a Scalable Tiered Archive & Query System for MySQL Data

This article presents a comprehensive design for a layered storage and unified scheduling platform that archives MySQL historical data, reduces storage costs, ensures high‑performance queries, and enables efficient data analysis through tiered hot, warm, and cold storage using big‑data technologies.

Data ArchivingDorisFlink
0 likes · 13 min read
How to Build a Scalable Tiered Archive & Query System for MySQL Data
DataFunSummit
DataFunSummit
Mar 1, 2026 · Big Data

How Ant Group’s Flex Engine Supercharges Flink with Vectorization

This article details Ant Group’s Flex vectorized engine built on Velox, covering the current state of vectorization, Flex’s architecture (Flink + Velox), core feature development, correctness guarantees, large‑scale deployment results, and future directions for full‑link vectorization and broader hardware support.

Big DataFlexFlink
0 likes · 18 min read
How Ant Group’s Flex Engine Supercharges Flink with Vectorization
ITPUB
ITPUB
Feb 13, 2026 · Big Data

Real‑Time Sync of New MySQL Tables to Doris Using Flink CDC

This article explains how to extend a Flink CDC job that already syncs an entire MySQL database to Doris so that newly created tables are automatically created in Doris in real time, using the CdcTools utility, side‑output streams, and asynchronous I/O.

CDCCdcToolsDoris
0 likes · 9 min read
Real‑Time Sync of New MySQL Tables to Doris Using Flink CDC
DeWu Technology
DeWu Technology
Feb 9, 2026 · Big Data

How to Build a Production‑Ready Flink ClickHouse Sink with Dynamic Sharding, Batch‑by‑Size, and Robust Retry

This article presents a production‑grade Flink ClickHouse sink that solves common pain points such as lack of size‑based batching, static table schemas, and distributed‑table latency by introducing data‑size batching, dynamic table routing, local‑table writes, load‑balanced node discovery, back‑pressure queues, dual‑trigger flush, and recursive retry with node exclusion, all integrated with Flink checkpoint semantics for at‑least‑once guarantees.

CheckpointClickHouseDynamic Sharding
0 likes · 25 min read
How to Build a Production‑Ready Flink ClickHouse Sink with Dynamic Sharding, Batch‑by‑Size, and Robust Retry
ITPUB
ITPUB
Feb 9, 2026 · Databases

ClickHouse vs Doris vs Redis: Real‑World Query Performance Test with Flink

Using a 600k‑record IP range dataset, we built identical tables in ClickHouse and Doris, and a Redis skip‑list store, then ran three Flink‑Kafka streaming jobs to compare query latency across the three databases under varying traffic rates, revealing Redis as fastest, ClickHouse second, Doris slowest.

ClickHouseDatabase PerformanceDoris
0 likes · 8 min read
ClickHouse vs Doris vs Redis: Real‑World Query Performance Test with Flink
Alibaba Cloud Big Data AI Platform
Alibaba Cloud Big Data AI Platform
Feb 2, 2026 · Big Data

How We Built a Scalable Lakehouse Architecture with StarRocks, Paimon, and Flink

This article details the evolution of a data warehouse at RenliJia from a MaxCompute‑centric setup to a modern lakehouse using StarRocks, Paimon, Flink, and Fluss, describing design goals, technical evaluations, implementation steps for offline, OLAP, and real‑time workloads, and the challenges and future plans that emerged.

Big DataData WarehouseFlink
0 likes · 25 min read
How We Built a Scalable Lakehouse Architecture with StarRocks, Paimon, and Flink
ITPUB
ITPUB
Jan 22, 2026 · Backend Development

Sync New MySQL Tables to Doris in Real‑Time with Flink CDC and CdcTools

This article explains how to use Flink CDC together with the CdcTools utility to automatically capture newly created MySQL tables and synchronize both their schema and data to a Doris database in real time, covering the required code, side‑output handling, async execution, and a special delete‑sign field.

Async IOCDCDoris
0 likes · 10 min read
Sync New MySQL Tables to Doris in Real‑Time with Flink CDC and CdcTools
Data Integration and Governance
Data Integration and Governance
Jan 12, 2026 · Big Data

Choosing the Right Real‑Time Data Architecture: Tech Options and Performance Trade‑offs

The article explains that real‑time data processing requires not only speed but also accuracy, stability, and long‑term maintainability, compares Lambda, Kappa, and unified batch‑stream architectures, discusses lakehouse storage, and provides practical guidance on selecting the most suitable approach for different project needs.

FlinkKappa ArchitectureLakehouse
0 likes · 9 min read
Choosing the Right Real‑Time Data Architecture: Tech Options and Performance Trade‑offs
Java Baker
Java Baker
Dec 22, 2025 · Big Data

Mastering Offline and Real-Time Data Warehouses: A Backend Engineer’s Guide

Backend developers need to understand both offline and real-time data warehouses; this guide explains data collection, layering, partitioning, typical use cases, archiving strategies, and how to build a real-time warehouse with Flink, covering practical steps, examples, and key considerations for efficient data processing.

Data WarehouseFlinkOffline
0 likes · 8 min read
Mastering Offline and Real-Time Data Warehouses: A Backend Engineer’s Guide
dbaplus Community
dbaplus Community
Dec 8, 2025 · Databases

Which Database Wins IP Range Lookups? ClickHouse vs Doris vs Redis Benchmarks

This article presents a systematic benchmark comparing ClickHouse, Doris, and Redis for IP‑range dimension lookups using Flink‑Kafka pipelines, detailing test design, result table schema, query interfaces, and performance results across varying data rates, concluding that Redis offers the fastest and most stable query latency.

ClickHouseDatabase BenchmarkDoris
0 likes · 7 min read
Which Database Wins IP Range Lookups? ClickHouse vs Doris vs Redis Benchmarks
Ctrip Technology
Ctrip Technology
Nov 20, 2025 · Big Data

How Ctrip Achieved Minute‑Level Real‑Time Analytics with Flink CDC & Apache Paimon

Ctrip transformed its traditional T+1 offline warehouse into a near‑real‑time lakehouse by integrating Flink CDC with Apache Paimon, designing a two‑stage CDC ingestion, optimizing performance, implementing dynamic updates, and deploying the solution across multiple business scenarios, achieving minute‑level latency, reduced costs, and faster data‑driven decisions.

CDCData EngineeringFlink
0 likes · 27 min read
How Ctrip Achieved Minute‑Level Real‑Time Analytics with Flink CDC & Apache Paimon
Big Data Technology & Architecture
Big Data Technology & Architecture
Nov 17, 2025 · Big Data

Flink 2025 Updates: Disaggregated State, AI Agents, and SQL Enhancements

The 2025 Flink release introduces a disaggregated state management architecture for cloud‑native elasticity, AI‑driven Flink Agents with LLM, Memory and Tool support, Delta Join and VARIANT type for semi‑structured data, adaptive batch execution, incremental checkpoints, high‑speed network optimizations, and new SQL and Process Table Functions, reshaping real‑time analytics.

Disaggregated StateFlinkReal-Time Analytics
0 likes · 8 min read
Flink 2025 Updates: Disaggregated State, AI Agents, and SQL Enhancements
vivo Internet Technology
vivo Internet Technology
Nov 12, 2025 · Big Data

How Vivo Solved Real‑Time Feature Concatenation with RocksDB and Flink

This article explains the evolution of Vivo's real‑time recommendation feature‑concatenation architecture, compares hour‑level, Redis‑streaming and RocksDB state‑backend solutions, and details the memory, performance, startup and HDFS RPC problems encountered along with the concrete fixes applied.

FlinkRocksDBfeature concatenation
0 likes · 21 min read
How Vivo Solved Real‑Time Feature Concatenation with RocksDB and Flink
Instant Consumer Technology Team
Instant Consumer Technology Team
Nov 10, 2025 · Big Data

Fixing Multi‑Version, Multi‑Cluster and HA with Apache Kyuubi for Spark/Flink

Apache Kyuubi, an enterprise‑grade multi‑tenant data gateway, replaces Livy and Flink SQL Gateway to support multiple engine versions, cross‑cluster elastic scheduling, high‑availability batch jobs, and traffic control, dramatically reducing deployment complexity, improving resource utilization, and accelerating release cycles for large‑scale Spark and Flink workloads.

Apache KyuubiBig DataData Gateway
0 likes · 18 min read
Fixing Multi‑Version, Multi‑Cluster and HA with Apache Kyuubi for Spark/Flink
Smart Sea Tide
Smart Sea Tide
Nov 4, 2025 · Big Data

Implementing an Integrated Data Lake and Lakehouse Architecture with Apache Iceberg and Flink

The article explains the concepts of data lakes and lakehouses, compares them with traditional data warehouses, outlines the reliability, performance, and security challenges of data lakes, and then details a practical lakehouse implementation using Apache Iceberg, Flink SQL, CDC pipelines, and supporting tools such as Hive Metastore and Trino.

Apache IcebergCDCData Warehouse
0 likes · 18 min read
Implementing an Integrated Data Lake and Lakehouse Architecture with Apache Iceberg and Flink
Smart Sea Tide
Smart Sea Tide
Oct 29, 2025 · Big Data

Understanding Flink’s Network Flow Control and Backpressure Mechanisms

This article explains the fundamentals of network flow control, compares TCP‑based and credit‑based backpressure in Flink, walks through buffer pool interactions, and shows why dynamic backpressure cannot fully replace static rate limiting in real‑world streaming pipelines.

Buffer ManagementCredit-basedFlink
0 likes · 15 min read
Understanding Flink’s Network Flow Control and Backpressure Mechanisms
Alibaba Cloud Big Data AI Platform
Alibaba Cloud Big Data AI Platform
Oct 24, 2025 · Big Data

How Leapmotor Scaled to 1M Cars with a Real‑Time Flink Data Platform

Leapmotor’s rapid growth to one million production cars drove a shift from daily batch data to minute‑level real‑time analytics, prompting the adoption of Flink as the core engine of a multi‑layered big‑data platform that handles massive IoT signals, supports fault diagnosis, and integrates batch and streaming workloads on the cloud.

AutomotiveBig DataFlink
0 likes · 13 min read
How Leapmotor Scaled to 1M Cars with a Real‑Time Flink Data Platform
Alibaba Cloud Big Data AI Platform
Alibaba Cloud Big Data AI Platform
Oct 22, 2025 · Big Data

Li Auto’s Trillion‑Row Real‑Time Car‑Network Analytics Using Hologres + Flink

Li Auto’s data team tackled the explosion of vehicle‑telemetry data—over a trillion rows and millions of signals per second—by redesigning their data foundation with Alibaba Cloud’s Hologres and Flink, achieving sub‑second latency, elastic scaling, high availability, and significant cost reductions across real‑time and offline workloads.

Car TelemetryFlinkHologres
0 likes · 16 min read
Li Auto’s Trillion‑Row Real‑Time Car‑Network Analytics Using Hologres + Flink
StarRocks
StarRocks
Oct 14, 2025 · Big Data

How Ctrip Scaled UBT Analytics by Migrating from ClickHouse to StarRocks

Ctrip's User Behavior Tracking (UBT) system, handling 30 TB of daily data, moved from ClickHouse to StarRocks' compute‑storage separated architecture, cutting average query latency from 1.4 seconds to 203 ms, halving storage, reducing nodes from 50 to 40, and boosting write throughput to 3 million rows per second.

Big DataClickHouseData Migration
0 likes · 15 min read
How Ctrip Scaled UBT Analytics by Migrating from ClickHouse to StarRocks
Smart Sea Tide
Smart Sea Tide
Oct 13, 2025 · Big Data

How Zhihu Evolved Its Real-Time Data Warehouse Architecture

The article details Zhihu's step‑by‑step evolution from a Spark‑Streaming‑based real‑time data warehouse (v1.0) to a Flink‑Streaming SQL‑driven platform (v2.0), explaining design choices, Lambda architecture implementation, scalability challenges, and future automation plans.

DruidFlinkHBase
0 likes · 22 min read
How Zhihu Evolved Its Real-Time Data Warehouse Architecture
StarRocks
StarRocks
Sep 23, 2025 · Databases

How Zepto Scaled Real‑Time Brand Analytics with StarRocks: From Postgres MVP to Sub‑Second Queries

Zepto transformed its brand‑analytics platform from a Postgres MVP into a production‑grade, sub‑second real‑time analytics solution by adopting StarRocks, redesigning its data pipeline with Databricks, Kafka, and Flink, and choosing a storage‑compute architecture that supports massive joins and rapid insights.

Data PipelineDatabricksFlink
0 likes · 14 min read
How Zepto Scaled Real‑Time Brand Analytics with StarRocks: From Postgres MVP to Sub‑Second Queries
DataFunSummit
DataFunSummit
Sep 21, 2025 · Big Data

Breaking the CPU Wall: BIGO’s Gluten Engine Accelerates Spark and Flink

When big‑data workloads hit the CPU wall, BIGO’s adoption of the open‑source Gluten project delivers native‑engine execution for Spark and a roadmap for Flink, achieving up to 30% end‑to‑end speedup, 50% memory savings, and a scalable, cost‑effective data processing platform.

Big DataFlinkGluten
0 likes · 16 min read
Breaking the CPU Wall: BIGO’s Gluten Engine Accelerates Spark and Flink
360 Zhihui Cloud Developer
360 Zhihui Cloud Developer
Sep 11, 2025 · Big Data

How Paimon Transforms Membership Data Warehousing: From Legacy Lambda to Real‑Time Lakehouse

This article examines the challenges of a legacy Lambda‑based membership data warehouse, introduces Apache Paimon’s lakehouse architecture and its key features, and showcases three real‑world implementations—partial‑update order wide tables, Bitmap‑based UV counting, and branch‑based data correction—while discussing benefits, remaining challenges, and future directions.

Big DataData WarehouseFlink
0 likes · 29 min read
How Paimon Transforms Membership Data Warehousing: From Legacy Lambda to Real‑Time Lakehouse
High Availability Architecture
High Availability Architecture
Sep 10, 2025 · Big Data

How Ctrip Business Travel Built a Near‑Real‑Time Lakehouse with Flink CDC & Paimon

This article details Ctrip Business Travel’s implementation of a near‑real‑time data warehouse using Flink CDC and the Paimon lakehouse engine, covering order wide‑table construction, ticket refund alerts, ad attribution, batch‑stream integration, and practical lessons on Partial Update, Aggregation, and Tag‑based incremental processing.

==AggregationBatch-Stream Integration
0 likes · 17 min read
How Ctrip Business Travel Built a Near‑Real‑Time Lakehouse with Flink CDC & Paimon
Alibaba Cloud Big Data AI Platform
Alibaba Cloud Big Data AI Platform
Sep 8, 2025 · Big Data

How Ele.me Revolutionized Real‑Time Data Warehousing with Flink‑Paimon Lakehouse

In this detailed case study, Alibaba’s Ele.me team explains how they evolved from siloed, chimney‑style real‑time warehouses to a unified Flink‑Paimon lakehouse, highlighting the three development stages, technology evaluations, the Alake platform’s one‑stop capabilities, production results, and future directions such as Fluss and AI integration.

AlakeFlinkLakehouse
0 likes · 17 min read
How Ele.me Revolutionized Real‑Time Data Warehousing with Flink‑Paimon Lakehouse
Ctrip Technology
Ctrip Technology
Sep 2, 2025 · Big Data

How Ctrip Built a Near‑Real‑Time Lakehouse with Flink & Paimon

This article details Ctrip Business Travel’s implementation of a near‑real‑time data warehouse and lakehouse using Flink CDC and Apache Paimon, covering order wide‑table construction, automated ticket reminders, ad attribution, batch‑stream integration, and lessons on Partial Update, Aggregation, and Tag‑based incremental processing.

Batch-Stream IntegrationFlinkLakehouse
0 likes · 17 min read
How Ctrip Built a Near‑Real‑Time Lakehouse with Flink & Paimon
Baidu Geek Talk
Baidu Geek Talk
Sep 1, 2025 · Big Data

How Baidu Netdisk Built a High‑Performance Real‑Time Engine with Flink

This article explains how Baidu Netdisk transitioned from Spark Streaming to a Flink‑based Tiangong real‑time computing engine, detailing the evolution, reasons for choosing Flink, architecture, configuration examples, business use cases, technical challenges, and future platform plans.

Baidu NetdiskBig DataFlink
0 likes · 16 min read
How Baidu Netdisk Built a High‑Performance Real‑Time Engine with Flink
Alibaba Cloud Big Data AI Platform
Alibaba Cloud Big Data AI Platform
Aug 31, 2025 · Big Data

Disaggregated Flink State AI Anomaly Detection, Slow‑Query Ranking (VLDB 2025)

At VLDB 2025, three Alibaba Cloud papers were accepted: one introduces a disaggregated state‑management architecture for Flink 2.0 that separates storage from compute, another presents a cross‑contrastive learning framework for unsupervised Flink anomaly detection, and the third proposes a multimodal ranking system for identifying root causes of slow queries in cloud databases.

Cross Contrastive LearningDisaggregated State ManagementFlink
0 likes · 10 min read
Disaggregated Flink State AI Anomaly Detection, Slow‑Query Ranking (VLDB 2025)
php Courses
php Courses
Aug 29, 2025 · Operations

How to Build a Real‑Time PHP Log Event Pipeline for Instant Insights

Learn how to transform PHP logs into real‑time, structured events by implementing a log event pipeline that includes JSON logging, lightweight collectors like Filebeat, streaming platforms such as Kafka or Flink, enrichment, and visualization with Grafana, enabling instant monitoring, alerting, and data‑driven decisions.

FlinkGrafanaLog Processing
0 likes · 7 min read
How to Build a Real‑Time PHP Log Event Pipeline for Instant Insights
Alibaba Cloud Big Data AI Platform
Alibaba Cloud Big Data AI Platform
Aug 21, 2025 · Big Data

How Hypergryph Built a High‑Performance Real‑Time Analytics Platform with StarRocks

This case study details how Hypergryph leveraged Alibaba Cloud EMR Serverless StarRocks, Flink, and Kafka to replace a ClickHouse data warehouse with a high‑performance, elastic, and easy‑to‑operate real‑time analytics platform that dramatically improved query speed, stability, operational efficiency, and cost for their gaming business.

Cloud ComputingData PipelineFlink
0 likes · 8 min read
How Hypergryph Built a High‑Performance Real‑Time Analytics Platform with StarRocks
StarRocks
StarRocks
Aug 19, 2025 · Big Data

How Joydata Scaled to 150 Billion Daily Events with StarRocks: A Data Architecture Journey

Facing daily data growth from millions to 150 billion records, Joydata‑U transformed its analytics platform through three architectural stages—Hadoop, Hadoop + Trino, and finally StarRocks—introducing resource isolation, Flat JSON acceleration, and Bitmap indexing to cut query latency by up to seven times and achieve sub‑2‑minute data freshness across BI, ad‑tech, game analytics, and CRM workloads.

Bitmap IndexFlat JSONFlink
0 likes · 12 min read
How Joydata Scaled to 150 Billion Daily Events with StarRocks: A Data Architecture Journey
Alibaba Cloud Big Data AI Platform
Alibaba Cloud Big Data AI Platform
Aug 7, 2025 · Big Data

How Flink ML Transforms Intelligent Operations: Real‑Time Anomaly Detection, Forecasting & Log Clustering

This article explains how Alibaba Cloud’s big‑data platform leverages Flink ML to build an intelligent‑operations service that tackles stability, cost and efficiency challenges through time‑series anomaly detection, forecasting and streaming log‑clustering, dramatically reducing latency, complexity and operational overhead.

FlinkIntelligent OperationsLog Clustering
0 likes · 25 min read
How Flink ML Transforms Intelligent Operations: Real‑Time Anomaly Detection, Forecasting & Log Clustering
58 Tech
58 Tech
Aug 7, 2025 · Big Data

Transform Real‑Time Data Warehousing with Paimon: From Flink ROW_NUMBER to Streaming Lakehouse

This article details how a real‑time data warehouse built on Flink, Kafka, HBase and MySQL was redesigned using Paimon to eliminate costly deduplication, handle out‑of‑order events, enable streaming reads, simplify aggregation, replace multiple lookup sources, and achieve faster, more reliable batch repairs, resulting in major resource and operational gains.

Data WarehouseFlinkLakehouse
0 likes · 24 min read
Transform Real‑Time Data Warehousing with Paimon: From Flink ROW_NUMBER to Streaming Lakehouse
iQIYI Technical Product Team
iQIYI Technical Product Team
Aug 7, 2025 · Big Data

Building a Low‑Latency, High‑Capacity Real‑Time Data Platform for Finance

Facing growing data demands in finance, we replaced two legacy synchronization pipelines with a unified, low‑latency architecture using BabelX Real‑Time, Flink CDC, Iceberg v2 and Paimon, achieving minute‑level data freshness, ten‑to‑thirty‑fold query speedups, reduced storage costs, and streamlined schema management across multiple business units.

Big DataFlinkIceberg
0 likes · 12 min read
Building a Low‑Latency, High‑Capacity Real‑Time Data Platform for Finance
Big Data Technology & Architecture
Big Data Technology & Architecture
Jul 29, 2025 · Big Data

What Interviewers Really Ask About Flink, Data Consistency, and Warehouse Design

An interviewee recounts a challenging first interview that focused on Flink resource configuration, late data handling, and offline data warehouse design, and shares practical advice on attitude, thorough preparation, emphasizing real project storytelling, and post‑interview review to continuously improve performance.

Data ConsistencyData WarehouseFlink
0 likes · 4 min read
What Interviewers Really Ask About Flink, Data Consistency, and Warehouse Design
Alibaba Cloud Big Data AI Platform
Alibaba Cloud Big Data AI Platform
Jul 25, 2025 · Big Data

Cross-Contrastive Learning Cuts Flink Anomaly Detection Errors by 12%

The paper “Noise Matters: Cross Contrastive Learning for Flink Anomaly Detection”, accepted at VLDB 2025, introduces a novel cross‑contrastive method that leverages attention‑based representations and a boundary‑aware loss to detect Flink‑specific hotspot anomalies, achieving a 12.1% F1 improvement over state‑of‑the‑art techniques.

Big DataCross-Contrastive LearningFlink
0 likes · 6 min read
Cross-Contrastive Learning Cuts Flink Anomaly Detection Errors by 12%
Big Data Tech Team
Big Data Tech Team
Jul 23, 2025 · Big Data

From Beginner to Data Warehouse Architect: A Complete Roadmap

This guide walks you through every essential topic—from data warehouse architecture and layering, through ETL, OLAP, Hadoop, and Flink, to visualization tools, learning paths, recommended resources, and the management skills needed to become a proficient data warehouse architect.

Data WarehouseETLFlink
0 likes · 9 min read
From Beginner to Data Warehouse Architect: A Complete Roadmap
Big Data Technology & Architecture
Big Data Technology & Architecture
Jul 16, 2025 · Big Data

Master Flink Optimizations: TTL, Mini‑Batch, Two‑Phase Aggregation, Lookup Join & More

This article reviews the most effective Flink optimization techniques since 2022, including operator‑level TTL, mini‑batch processing, two‑phase aggregation, multi‑dimensional DISTINCT with FILTER, lookup join caching strategies, and TopN implementations, each rated with recommendation stars for production use.

Big DataFlinkLookup Join
0 likes · 8 min read
Master Flink Optimizations: TTL, Mini‑Batch, Two‑Phase Aggregation, Lookup Join & More
DataFunSummit
DataFunSummit
Jul 12, 2025 · Big Data

How Fluss Unifies Stream and Lake to Power AI Data Pipelines

In the era of rapid AI growth, Fluss offers a unified lake‑stream architecture that tackles data quality, timeliness, scale, and multimodal challenges by tightly integrating Flink streaming with a high‑performance data lake, enabling seamless real‑time and batch analytics for AI workloads.

AIFlinkFluss
0 likes · 12 min read
How Fluss Unifies Stream and Lake to Power AI Data Pipelines
StarRocks
StarRocks
Jul 9, 2025 · Big Data

How Shopee Built a Near‑Real‑Time Data Warehouse with Paimon and StarRocks

Shopee combined the Paimon data lake with StarRocks and Flink to create a quasi‑real‑time warehouse, enabling fast task diagnostics and a high‑performance financial reconciliation system while dramatically reducing storage costs and latency through innovative ODS, snapshot, and branch table techniques.

FlinkPaimonReal-time Data Warehouse
0 likes · 13 min read
How Shopee Built a Near‑Real‑Time Data Warehouse with Paimon and StarRocks
Big Data Technology & Architecture
Big Data Technology & Architecture
Jul 8, 2025 · Big Data

Flink’s AI Agents and Disaggregated State: Transforming Big Data

The article reviews key topics from the FFA2025 Singapore conference, highlighting Flink’s new AI‑focused Agents framework, the breakthrough Flink 2.0 disaggregated state architecture, emerging lake storage solutions like Paimon, and the Fluss streaming table store, illustrating how big‑data platforms are evolving for AI workloads.

AI agentsBig DataDisaggregated State
0 likes · 6 min read
Flink’s AI Agents and Disaggregated State: Transforming Big Data
StarRocks
StarRocks
Jul 1, 2025 · Big Data

How StarRocks Boosted Suixingfu’s Real‑Time Data Platform: 3× Faster Queries & 10× Faster Analytics

Suixingfu rebuilt its payment data pipeline by replacing a fragmented Lambda stack with a unified Porter CDC + StarRocks + Elasticsearch architecture, achieving three‑fold query speed, ten‑fold analytics efficiency, 20% storage reduction, and sub‑second data‑capture latency across high‑concurrency, ad‑hoc, and batch workloads.

CDCData WarehouseFlink
0 likes · 14 min read
How StarRocks Boosted Suixingfu’s Real‑Time Data Platform: 3× Faster Queries & 10× Faster Analytics
JD Retail Technology
JD Retail Technology
Jun 10, 2025 · Artificial Intelligence

How JD Builds a Scalable AI‑Powered Recommendation Data System with Flink

This article explains JD's complex recommendation system data pipeline—from indexing, sampling, and feature engineering to explainability and real‑time metrics—highlighting challenges such as data consistency, latency, and the use of Flink for massive, low‑latency processing.

ExplainabilityFlinkReal‑time Data
0 likes · 23 min read
How JD Builds a Scalable AI‑Powered Recommendation Data System with Flink
Alibaba Cloud Big Data AI Platform
Alibaba Cloud Big Data AI Platform
May 21, 2025 · Big Data

How Alibaba’s A+ Traffic Analysis Achieved Sub‑Second Log Queries with StarRocks & Paimon

This article details how Alibaba's A+ traffic analysis platform tackled trillion‑row log ingestion and high‑concurrency queries by redesigning storage with Paimon, leveraging Flink for real‑time ingestion, and using StarRocks for fast lake analytics, ultimately reducing query latency from minutes to seconds.

FlinkLog AnalyticsPaimon
0 likes · 15 min read
How Alibaba’s A+ Traffic Analysis Achieved Sub‑Second Log Queries with StarRocks & Paimon
Big Data Technology & Architecture
Big Data Technology & Architecture
May 21, 2025 · Big Data

Interview Experience: Flink Task Resource Allocation, Issues, and Monitoring

This article shares an interviewee's experience discussing core Flink interview questions, including typical resource allocation for large online tasks, common problems such as data, performance, stability, and resource issues, and the monitoring practices for clusters and tasks, while also containing a brief self‑promotion.

Big DataFlinkInterview
0 likes · 7 min read
Interview Experience: Flink Task Resource Allocation, Issues, and Monitoring
Xiaohongshu Tech REDtech
Xiaohongshu Tech REDtech
May 19, 2025 · Industry Insights

How Xiaohongshu Built a Minute‑Level Near‑Real‑Time Data Warehouse with Incremental Computing

Facing billions of daily logs and the need for minute‑level experiment metrics, Xiaohongshu partnered with Yunqi Tech to design a generic incremental‑compute solution that delivers near‑real‑time data warehousing with lower cost, higher accuracy, simplified pipelines, and improved query performance.

Big DataFlinkIceberg
0 likes · 24 min read
How Xiaohongshu Built a Minute‑Level Near‑Real‑Time Data Warehouse with Incremental Computing
Selected Java Interview Questions
Selected Java Interview Questions
May 15, 2025 · Backend Development

Six Common Approaches to Synchronize MySQL Data to Elasticsearch

This article reviews six mainstream solutions for keeping MySQL and Elasticsearch in sync—including synchronous double‑write, asynchronous MQ‑based double‑write, Logstash polling, Canal binlog listening, DataX batch migration, and Flink stream processing—detailing their scenarios, advantages, drawbacks, and practical code examples to guide optimal technical selection.

CanalData synchronizationElasticsearch
0 likes · 8 min read
Six Common Approaches to Synchronize MySQL Data to Elasticsearch
Huolala Tech
Huolala Tech
May 14, 2025 · Big Data

How Lalamove Scaled Real‑Time Data Warehousing with Flink and Paimon

Lalamove’s international logistics platform transformed its real‑time data warehouse by leveraging Apache Flink and the Paimon lakehouse, addressing challenges of multi‑region data centers, time‑zone diversity, frequent upstream changes, and high costs, while improving scalability, latency, and operational efficiency across global markets.

Big DataFlinkPaimon
0 likes · 13 min read
How Lalamove Scaled Real‑Time Data Warehousing with Flink and Paimon