Skip to content
Tagged articles

streaming

500 articles · Page 3 of 5
MaGe Linux Operations
MaGe Linux Operations
Jan 21, 2024 · Big Data

Master Kafka: Core Concepts, Metrics, and Troubleshooting Guide

This article explains Kafka's fundamental components, version evolution, key monitoring metrics for producers, brokers, consumers and Zookeeper, and provides step‑by‑step troubleshooting methods for common issues such as slow topic throughput and message backlog.

KafkaMessage Queuebig data
0 likes · 8 min read
Master Kafka: Core Concepts, Metrics, and Troubleshooting Guide
Rare Earth Juejin Tech Community
Rare Earth Juejin Tech Community
Jan 13, 2024 · Big Data

What Is Kafka? Overview, Architecture, Features, Deployment, and Sample Code

Kafka, an Apache‑developed distributed publish/subscribe messaging system, provides reliable, high‑throughput real‑time data streaming with producers, consumers, brokers, streams, and connectors, and the article explains its core concepts, architecture, advantages, deployment methods, use cases, and includes Java code examples for producers and consumers.

JavaKafkaMessage Queue
0 likes · 8 min read
What Is Kafka? Overview, Architecture, Features, Deployment, and Sample Code
FunTester
FunTester
Jan 5, 2024 · Big Data

An Overview of Apache Kafka and Kafka Streams Technical Features

This article introduces Apache Kafka as a high‑throughput, scalable, fault‑tolerant distributed streaming platform, explains why it is chosen for real‑time data pipelines, and details key Kafka Streams concepts such as stream processing, interactive queries, stateful processing, windowing, serialization, and testing.

Apache KafkaKafka Streamsbig data
0 likes · 13 min read
An Overview of Apache Kafka and Kafka Streams Technical Features
Sohu Tech Products
Sohu Tech Products
Dec 27, 2023 · Big Data

Practical Implementation of Data Integration with Flink on Kubernetes at Li Auto

Li Auto built a cloud‑native data‑integration platform by deploying Flink on Kubernetes, unifying batch and streaming workloads with a storage layer (JuiceFS + BOS) and Flink Operator, enabling simple source‑sink pipelines, elastic scaling, automated checkpointing, and centralized monitoring while addressing earlier fragmentation and resource inefficiencies.

Cloud NativeFlinkKubernetes
0 likes · 11 min read
Practical Implementation of Data Integration with Flink on Kubernetes at Li Auto
ITPUB
ITPUB
Dec 24, 2023 · Backend Development

Why Kafka Is the Backbone of Modern Messaging, Streaming, and Data Pipelines

This article explains how Kafka serves as a high‑throughput, durable messaging system, a reliable storage layer, a log‑aggregation hub, a stream‑processing engine, and a core component for CDC, system migration, monitoring, and event‑sourcing architectures.

CDCKafkaevent-sourcing
0 likes · 9 min read
Why Kafka Is the Backbone of Modern Messaging, Streaming, and Data Pipelines
DataFunTalk
DataFunTalk
Dec 15, 2023 · Big Data

Flink Forward Asia 2023: New Flink Releases, Apache Paimon, and Flink CDC 3.0

The Flink Forward Asia 2023 conference showcased major updates to Apache Flink (versions 1.17 and 1.18), introduced the Apache Paimon lakehouse project, announced Flink CDC 3.0, and highlighted community growth, cloud‑native deployments, and real‑time data‑warehouse use cases across industry leaders.

Apache FlinkApache PaimonFlink CDC
0 likes · 17 min read
Flink Forward Asia 2023: New Flink Releases, Apache Paimon, and Flink CDC 3.0
ITPUB
ITPUB
Dec 14, 2023 · Big Data

How to Build a Python‑Hadoop Word Count on a Single‑Node Cluster

This step‑by‑step guide shows how to install and configure a single‑node Hadoop 3.2.0 environment on CentOS 7, set up Python 3.7, write MapReduce mapper and reducer scripts in Python, and run a word‑count job using Hadoop streaming, illustrating core Hadoop concepts and their relevance today.

HadoopMapReducePython
0 likes · 21 min read
How to Build a Python‑Hadoop Word Count on a Single‑Node Cluster
DataFunTalk
DataFunTalk
Dec 8, 2023 · Big Data

Zhihu Bridge Platform: Architecture, Capabilities, and Future Trends of Content Operations

This article presents a comprehensive overview of Zhihu's Bridge platform, detailing its content‑operation architecture—including content pool, management, analysis, monitoring, and intervention modules—explaining the underlying streaming and batch technologies such as Flink, Doris, and Elasticsearch, and outlining future automation and AI‑driven workflow directions.

AIZhihuarchitecture
0 likes · 17 min read
Zhihu Bridge Platform: Architecture, Capabilities, and Future Trends of Content Operations
ITPUB
ITPUB
Dec 2, 2023 · Backend Development

Why Did My Flink Kafka Job Lose Data? Uncovering Misconfigured Bootstrap Servers

A Flink job that reads from Kafka and writes to Elasticsearch was losing data because the bootstrap.servers list mixed production and pre‑release clusters, causing random server selection, partition discovery failures, and offset mismatches, which were resolved by correcting the server configuration.

Bootstrap ServersFlinkKafka
0 likes · 8 min read
Why Did My Flink Kafka Job Lose Data? Uncovering Misconfigured Bootstrap Servers
JavaEdge
JavaEdge
Nov 24, 2023 · Backend Development

Why Kafka Is the Ultimate Backbone for Modern Backend Systems

This article explores how Kafka serves as a versatile backbone for messaging, durable storage, log aggregation, monitoring, commit logs, recommendation pipelines, stream processing, CDC, system migration, and event sourcing, highlighting its performance, reliability, and practical deployment patterns.

BackendData PipelineKafka
0 likes · 10 min read
Why Kafka Is the Ultimate Backbone for Modern Backend Systems
Alibaba Cloud Big Data AI Platform
Alibaba Cloud Big Data AI Platform
Nov 23, 2023 · Big Data

Why Apache Paimon Is Revolutionizing Streaming Lakehouse Architecture with Flink

The article traces the shift from traditional Hive‑based warehouses to modern lakehouse architectures, explains the advantages of lake formats, introduces Apache Paimon as a streaming‑first data lake integrated with Flink, presents performance benchmarks showing its superiority over Hudi, and demonstrates a real‑time streaming lakehouse workflow.

Apache PaimonFlinkbig data
0 likes · 15 min read
Why Apache Paimon Is Revolutionizing Streaming Lakehouse Architecture with Flink
Alibaba Cloud Big Data AI Platform
Alibaba Cloud Big Data AI Platform
Nov 22, 2023 · Big Data

Real-Time Data Integration with Flink CDC: Core Tech and Alibaba Cloud Solutions

This article, based on a presentation by Flink CDC and Apache Flink community leaders, explores CDC real‑time integration challenges, delves into Flink CDC’s core technologies such as incremental snapshot and lock‑free processing, and demonstrates Alibaba Cloud’s enterprise‑grade solutions for end‑to‑end real‑time data pipelines.

Alibaba CloudFlink CDCReal-Time Data Integration
0 likes · 21 min read
Real-Time Data Integration with Flink CDC: Core Tech and Alibaba Cloud Solutions
Big Data Technology Architecture
Big Data Technology Architecture
Nov 14, 2023 · Big Data

Open Source Big Data Platform 3.0: Streaming Lakehouse, Serverless Architecture, and AI Integration

The talk outlines the evolution of Alibaba Cloud's open‑source big data platform from Hadoop‑based EMR to a 3.0 architecture featuring a streaming lakehouse, full serverless compute and storage, AI‑driven operations, and upcoming vector search services, highlighting technical motivations, challenges, and product releases.

big datacloud-nativelakehouse
0 likes · 14 min read
Open Source Big Data Platform 3.0: Streaming Lakehouse, Serverless Architecture, and AI Integration
DataFunSummit
DataFunSummit
Nov 9, 2023 · Big Data

Spark 3.4 New Features Overview: Community Updates, SQL Enhancements, PySpark, Streaming, and AI Ecosystem

This article presents a comprehensive overview of Spark 3.4, covering community growth statistics, major SQL improvements such as default column values and timestamp handling, new PySpark and streaming capabilities, and the emerging AI ecosystem that integrates natural‑language interfaces and Spark AI services.

DatabricksPySparkstreaming
0 likes · 14 min read
Spark 3.4 New Features Overview: Community Updates, SQL Enhancements, PySpark, Streaming, and AI Ecosystem
macrozheng
macrozheng
Nov 9, 2023 · Big Data

7 Real-World Kafka Use Cases Every Engineer Should Know

This article explains Kafka's core components and features, then details seven practical scenarios—including log processing, recommendation streams, monitoring, CDC, system migration, event sourcing, and message queuing—showing how Kafka powers modern distributed systems.

KafkaMessage Queuebig data
0 likes · 12 min read
7 Real-World Kafka Use Cases Every Engineer Should Know
ITPUB
ITPUB
Nov 7, 2023 · Big Data

7 Real-World Kafka Use Cases That Power Modern Distributed Systems

This article introduces Apache Kafka’s core components and key features, then details seven practical use cases—including log processing, recommendation streams, monitoring, CDC, system migration, event sourcing, and message queuing—illustrated with diagrams and step‑by‑step workflows for distributed systems.

KafkaMessage Queuebig data
0 likes · 10 min read
7 Real-World Kafka Use Cases That Power Modern Distributed Systems
HelloTech
HelloTech
Oct 31, 2023 · Big Data

Investigation of Data Loss in a Flink Kafka Consumer Caused by Mixed Kafka Cluster Configuration

The data loss in a Flink‑Kafka job was caused by a mis‑configured bootstrap.servers list that mixed production and pre‑release Kafka clusters, leading different subtasks to connect to different clusters, resulting in inconsistent partition discovery and offset fetching, which omitted several partitions until the list was corrected.

Cluster ConfigurationElasticsearchFlink
0 likes · 8 min read
Investigation of Data Loss in a Flink Kafka Consumer Caused by Mixed Kafka Cluster Configuration
Top Architect
Top Architect
Sep 25, 2023 · Backend Development

RabbitMQ vs Kafka: Detailed Comparison and When to Use Each

This article provides an in‑depth technical comparison of RabbitMQ and Apache Kafka, covering their core architectural differences, message ordering, routing, timing, retention, fault handling, scalability, consumer complexity, and offers guidance on selecting the appropriate platform for various backend scenarios.

KafkaMessage QueueRabbitMQ
0 likes · 18 min read
RabbitMQ vs Kafka: Detailed Comparison and When to Use Each
JD Cloud Developers
JD Cloud Developers
Sep 18, 2023 · Backend Development

Mastering Rust gRPC Streaming with Tonic: Build Server & Client

This guide walks through creating a Rust project that uses the Tonic library to implement gRPC streaming, covering project setup, protobuf definitions, server and client code, testing with grpcurl, and enabling the reflection API for service introspection.

BackendRustTonic
0 likes · 15 min read
Mastering Rust gRPC Streaming with Tonic: Build Server & Client
21CTO
21CTO
Sep 8, 2023 · Big Data

Why Real-Time Data Processing Is the Next Frontier for Data Engineers

Real-time data processing transforms traditional batch pipelines by delivering fresh, low‑latency data to millions of concurrent users, leveraging event‑driven architectures, streaming engines, and real‑time databases, with use cases ranging from fraud detection to personalized e‑commerce and operational dashboards, and includes reference architectures and tool recommendations.

Data Engineeringarchitecturebig data
0 likes · 16 min read
Why Real-Time Data Processing Is the Next Frontier for Data Engineers
StarRocks
StarRocks
Sep 6, 2023 · Big Data

How Paimon + StarRocks Revolutionize Lakehouse Analytics

This article reviews traditional Lambda and Kappa data‑warehouse architectures, then details four Paimon‑StarRocks lakehouse solutions—including a data‑lake center, accelerated query with materialized views, hot‑cold data separation, and the JNI connector—while also outlining StarRocks’ future roadmap for lakehouse analytics.

Materialized ViewPaimonStarRocks
0 likes · 11 min read
How Paimon + StarRocks Revolutionize Lakehouse Analytics
Data Thinking Notes
Data Thinking Notes
Aug 27, 2023 · Big Data

How ByteDance’s LAS Team Unified Real‑Time and Offline Warehousing with a Lakehouse Solution

This article analyzes the shortcomings of mainstream Lambda‑style data warehouse architectures, introduces Hudi‑based lakehouse design principles, details the three‑layer unified storage architecture, data distribution, model and read/write mechanisms, and showcases real‑time streaming, multidimensional analysis, and stream‑batch reuse scenarios along with future roadmap plans.

Hudilakehousestreaming
0 likes · 14 min read
How ByteDance’s LAS Team Unified Real‑Time and Offline Warehousing with a Lakehouse Solution
Big Data Technology & Architecture
Big Data Technology & Architecture
Aug 21, 2023 · Big Data

Key Features and Benefits of Lakehouse Frameworks Hudi, Iceberg, and Paimon

This note outlines how Hudi, Iceberg, and Paimon provide unified batch‑stream storage, UPSERT support, time‑travel capabilities, and lower development costs, enabling a streaming‑warehouse architecture that offers near‑real‑time latency, consistent semantics, persisted intermediate results, and easier historical data repair.

HudiIcebergPaimon
0 likes · 5 min read
Key Features and Benefits of Lakehouse Frameworks Hudi, Iceberg, and Paimon
Bitu Technology
Bitu Technology
Aug 9, 2023 · Product Management

Tubi July 2023 Highlights: New CEO, Top FAST Service Ranking, Content Personalization Strategy, Data‑Driven DNA, Emmy Nomination

In July 2023 Tubi announced the appointment of Anjali Sud as CEO, reinforced its position as the highest‑rated FAST service in the U.S., detailed its personalized content strategy, highlighted its data‑driven technology and ad‑tech approach, and celebrated its first Emmy nomination and industry recognitions.

CEOFASTawards
0 likes · 8 min read
Tubi July 2023 Highlights: New CEO, Top FAST Service Ranking, Content Personalization Strategy, Data‑Driven DNA, Emmy Nomination
Java Architecture Diary
Java Architecture Diary
Jul 11, 2023 · Big Data

Redpanda vs Apache Kafka with KRaft: Why Redpanda Is Up to 10× Faster

This article presents a detailed benchmark comparing Redpanda 23.1 and Apache Kafka 3.4.0 (with and without KRaft) across multiple AWS instance types, showing how Redpanda consistently delivers higher throughput and dramatically lower end‑to‑end latency, often outperforming Kafka by 4‑20× even with extra hardware.

Apache KafkaKRaftRedpanda
0 likes · 12 min read
Redpanda vs Apache Kafka with KRaft: Why Redpanda Is Up to 10× Faster
Big Data Technology & Architecture
Big Data Technology & Architecture
Jul 4, 2023 · Big Data

Building a Real‑Time Streaming Data Warehouse with Paimon on Kubernetes for Supply‑Chain Logistics

This article presents a step‑by‑step guide on how the logistics provider Haicheng Bangda implemented a streaming data warehouse using Paimon, Flink CDC, and Kubernetes, covering business background, architecture choices, environment setup, SQL examples, troubleshooting tips, and future roadmap for their digital transformation.

CDCFlinkKubernetes
0 likes · 27 min read
Building a Real‑Time Streaming Data Warehouse with Paimon on Kubernetes for Supply‑Chain Logistics
Sanyou's Java Diary
Sanyou's Java Diary
Jun 26, 2023 · Big Data

Master Kafka Interview Questions: Architecture, Partitioning, and Reliability Explained

This article provides a comprehensive overview of Kafka, covering its core architecture, message queue models, communication process, partition selection, consumer groups, rebalancing strategies, partition assignment algorithms, reliability guarantees, replica synchronization, and reasons for removing Zookeeper in newer versions.

Consumer GroupKafkaPartitioning
0 likes · 20 min read
Master Kafka Interview Questions: Architecture, Partitioning, and Reliability Explained
FunTester
FunTester
Jun 19, 2023 · Big Data

Kafka Architecture and Core Concepts: Brokers, Producers, Consumers, Topics, Partitions, Replicas, and Reliability

This article provides a comprehensive overview of Kafka's architecture and fundamental concepts, covering its overall structure, key components such as brokers, producers, consumers, topics, partitions, replicas, leader‑follower synchronization, offset handling, message storage at both logical and physical layers, as well as producer and consumer workflows, partition assignment strategies, rebalancing, log management, zero‑copy I/O, and reliability mechanisms.

Distributed SystemsKafkaconsumer groups
0 likes · 22 min read
Kafka Architecture and Core Concepts: Brokers, Producers, Consumers, Topics, Partitions, Replicas, and Reliability
phodal
phodal
Jun 18, 2023 · Artificial Intelligence

How to Build Language‑First APIs: 5 LLM‑Powered Architectural Patterns

The article outlines five practical patterns—natural‑language DSL, streaming DSL, DSL‑guided generation, explicit retry, and dynamic proxying—that enable developers to treat large‑language‑model interactions as first‑class APIs, improving efficiency, accuracy, and user experience across diverse scenarios.

DSLLLMdynamic-proxy
0 likes · 10 min read
How to Build Language‑First APIs: 5 LLM‑Powered Architectural Patterns
Architects Research Society
Architects Research Society
Jun 4, 2023 · Big Data

Understanding Transactions in Apache Kafka

This article explains the design, semantics, and practical usage of Apache Kafka's transaction API, covering why transactions are needed for exactly‑once processing, the underlying atomic multi‑partition writes, zombie fencing, consumer guarantees, Java API details, performance considerations, and operational best practices.

Apache KafkaDistributed SystemsJava
0 likes · 19 min read
Understanding Transactions in Apache Kafka
DataFunSummit
DataFunSummit
May 28, 2023 · Big Data

Apache Hudi: Capabilities, Architecture, Use Cases, and Future Outlook

This article introduces Apache Hudi as a next‑generation streaming data‑lake platform, explains its core concepts, architecture, and table types, and showcases real‑world use cases at Tencent such as CDC ingestion, minute‑level real‑time warehousing, streaming analytics, multi‑stream joins, ad attribution, and stream‑to‑batch processing, while also outlining future directions.

Apache HudiCDCFlink
0 likes · 16 min read
Apache Hudi: Capabilities, Architecture, Use Cases, and Future Outlook
StarRocks
StarRocks
May 26, 2023 · Big Data

How SeaTunnel’s StarRocks Connector Enables High‑Performance Data Sync

This article explains SeaTunnel’s architecture and its StarRocks connector, detailing source and sink features such as field projection, predicate push‑down, parallel reading, state recovery, data type mapping, Stream Load writes, CDC support, configuration examples, and future roadmap for exactly‑once semantics.

ConnectorSeaTunnelStarRocks
0 likes · 16 min read
How SeaTunnel’s StarRocks Connector Enables High‑Performance Data Sync
Baidu Geek Talk
Baidu Geek Talk
May 15, 2023 · Industry Insights

How AI‑Driven Perceptual Encoding Cuts Video Bandwidth by Up to 60% While Boosting Quality

This article examines the technical background, core AI‑assisted perceptual encoding methods, practical implementations, and performance results of Baidu's intelligent video cloud, showing how content‑aware preprocessing, ROI‑based bitrate allocation, and AI‑enhanced super‑resolution can dramatically reduce bandwidth consumption while improving user experience.

AIBaidubandwidth optimization
0 likes · 21 min read
How AI‑Driven Perceptual Encoding Cuts Video Bandwidth by Up to 60% While Boosting Quality
DataFunTalk
DataFunTalk
May 5, 2023 · Big Data

NetEase Cloud Music Real-Time Data Warehouse Architecture and Low-Code Platform Practices

This article presents NetEase Cloud Music's real-time data warehouse architecture, covering its streaming and batch scenarios, layered design (ODS, CDM, ADS), technology stack choices, consistency mechanisms, the FastX low-code platform, and future development plans, offering a comprehensive technical overview for data engineers and architects.

ClickHouseFlinkKafka
0 likes · 18 min read
NetEase Cloud Music Real-Time Data Warehouse Architecture and Low-Code Platform Practices
Bilibili Tech
Bilibili Tech
Apr 28, 2023 · Cloud Native

Remote StateBackend for Flink: Design, Optimizations, and Cloud‑Native Migration

To enable Bilibili’s cloud‑native migration, the team built a RemoteStateBackend that moves Flink’s keyed state to the Taishan KV store, using deterministic KeyGroup placement, per‑shard snapshots, asynchronous write batching, off‑heap caching with Bloom‑filter filtering, and a fixed‑size memory model, which together reduce checkpoint overhead, improve disk utilization, and accelerate rescaling for more than one hundred production jobs.

CloudNativeFlinkPerformanceOptimization
0 likes · 18 min read
Remote StateBackend for Flink: Design, Optimizations, and Cloud‑Native Migration
Architects Research Society
Architects Research Society
Apr 25, 2023 · Big Data

Understanding Transactions in Apache Kafka: Semantics, API, and Practical Guidance

This article explains the purpose, semantics, and design of Apache Kafka's transaction API, describes how exactly‑once processing is achieved in stream‑processing applications, outlines the Java client usage, and discusses the internal components, performance considerations, and best‑practice tips for developers.

Distributed SystemsJavaKafka
0 likes · 16 min read
Understanding Transactions in Apache Kafka: Semantics, API, and Practical Guidance
IT Architects Alliance
IT Architects Alliance
Mar 28, 2023 · Big Data

Kafka Core Concepts, Architecture, Performance Optimization, and Production Deployment Guide

This comprehensive guide explains Kafka's core value as a message queue, its fundamental concepts, cluster architecture, high‑performance data handling, resource planning for large‑scale deployments, operational tools, consumer‑group mechanics, offset management, rebalance strategies, and custom partitioner implementation.

consumer groupsdeploymentdistributed-systems
0 likes · 29 min read
Kafka Core Concepts, Architecture, Performance Optimization, and Production Deployment Guide
Qunar Tech Salon
Qunar Tech Salon
Mar 21, 2023 · Frontend Development

Performance Analysis of React CSR, SSR, and React 18 Streaming SSR

This article examines how different React rendering strategies—client‑side rendering, server‑side rendering, and the new React 18 Streaming SSR—affect key web performance metrics such as TTI, FCP, and first‑paint, and demonstrates substantial latency reductions achieved through streaming and selective hydration.

SSRTTIWeb Performance
0 likes · 11 min read
Performance Analysis of React CSR, SSR, and React 18 Streaming SSR
Architect
Architect
Mar 17, 2023 · Backend Development

Million‑Scale Data Export with JPA and MyBatis in Spring Boot

This article explains how to export tens of millions of rows from MySQL using Spring Boot by streaming data with JPA or MyBatis, avoiding OutOfMemoryError, switching to CSV format, and provides complete code examples, performance comparison, and tips for generating test data.

CSVDataExportJPA
0 likes · 12 min read
Million‑Scale Data Export with JPA and MyBatis in Spring Boot
DataFunTalk
DataFunTalk
Mar 9, 2023 · Big Data

Real‑Time Data Platform Architecture and Cloud‑Native Flink Migration at Manbang

This article presents a comprehensive case study of Manbang's real‑time data platform, detailing its business background, cloud‑native Flink + Hologres architecture, migration from self‑built clusters, real‑time product features, decision‑making workflows, and future roadmap, highlighting performance and cost benefits.

Flinkcloud-nativelogistics
0 likes · 16 min read
Real‑Time Data Platform Architecture and Cloud‑Native Flink Migration at Manbang
Alibaba Cloud Big Data AI Platform
Alibaba Cloud Big Data AI Platform
Mar 1, 2023 · Big Data

How We Built a Scalable Real‑Time Data Architecture for a Complex Supply Chain

This article describes the challenges of a highly complex supply‑chain system, the evolution from early MySQL‑based reporting to a modern real‑time data platform using Flink, Kafka, ClickHouse, Hologres and other cloud services, and the tools and lessons learned to achieve low‑latency, high‑throughput analytics.

ClickHouseFlinkKafka
0 likes · 11 min read
How We Built a Scalable Real‑Time Data Architecture for a Complex Supply Chain
DataFunSummit
DataFunSummit
Feb 28, 2023 · Big Data

Iceberg Technology Overview and Its Application at Xiaomi: Practices, Stream‑Batch Integration, and Future Plans

This article introduces the Iceberg table format, explains its core architecture and advantages such as transactionality, implicit partitioning and row‑level updates, details Xiaomi's practical deployments—including CDC pipelines, partition strategies, compaction services, and stream‑batch integration—and outlines future development directions.

FlinkIcebergSpark
0 likes · 20 min read
Iceberg Technology Overview and Its Application at Xiaomi: Practices, Stream‑Batch Integration, and Future Plans
DeWu Technology
DeWu Technology
Feb 24, 2023 · Big Data

Real-Time Data Architecture Evolution for a Complex Supply Chain

The article traces Dewu’s supply‑chain data platform from slow MySQL reporting through early CDC‑based wide tables to a Flink‑Kafka‑ClickHouse 1.0 design, then to a more scalable Flink‑Kafka‑Hologres 2.0 architecture that solves upsert and compute‑storage separation, while detailing key operational tricks, code‑generation tools, and future plans for lake‑house integration.

ClickHouseFlinkHologres
0 likes · 10 min read
Real-Time Data Architecture Evolution for a Complex Supply Chain
Alimama Tech
Alimama Tech
Feb 15, 2023 · Big Data

Dolphin: Alibaba's Hyper‑Converged Multi‑Modal Big Data Engine Overview

Dolphin, Alibaba’s hyper‑converged multi‑modal big‑data engine, unifies OLAP, AI, streaming, and batch workloads on a decoupled compute‑storage MPP foundation, offering a Dolphin SQL layer, advanced bitmap/GroupTable/AFile indexes, intelligent materialization, and one‑write‑multiple‑read storage that cuts costs over 70% while delivering sub‑millisecond queries on trillion‑row datasets.

AIIndexingOLAP
0 likes · 14 min read
Dolphin: Alibaba's Hyper‑Converged Multi‑Modal Big Data Engine Overview
Architecture Digest
Architecture Digest
Feb 9, 2023 · Big Data

Understanding Kafka Messages, Topics, Partitions, and Consumers

This article explains Kafka's core concepts—including messages as byte arrays, optional keys for partition control, topic and partition organization, producer and consumer roles, offsets, consumer groups, and broker clusters—providing a concise technical overview for developers learning Kafka.

ConsumerKafkaMessage
0 likes · 6 min read
Understanding Kafka Messages, Topics, Partitions, and Consumers
Bilibili Tech
Bilibili Tech
Jan 31, 2023 · Big Data

Design and Optimization of Real-Time Data Quality Control (DQC) Platform on Bilibili's Big Data System

Bilibili redesigned its real-time data-quality control platform by replacing per-rule Flink jobs with a unified, dynamically-configured architecture that classifies Kafka topics, aggregates via InfluxDB full-table and continuous queries, mitigates data inflation, adds a high-performance proxy, and implements robust monitoring and recovery to ensure scalable, reliable data quality for its big-data services.

DQCFlinkInfluxDB
0 likes · 22 min read
Design and Optimization of Real-Time Data Quality Control (DQC) Platform on Bilibili's Big Data System
ITPUB
ITPUB
Jan 22, 2023 · Big Data

How Flink Table Store Powers Real‑Time Financial Data Warehousing

This article details a banking‑focused real‑time data‑warehouse solution that leverages Flink Table Store to handle both incremental fact‑table updates and full‑table dimension calculations, compares three traditional approaches, and explains data ingestion, query modes, export options, and future streaming‑warehouse directions.

ELTFlinkTable Store
0 likes · 20 min read
How Flink Table Store Powers Real‑Time Financial Data Warehousing
DataFunSummit
DataFunSummit
Jan 8, 2023 · Big Data

Streaming‑Batch Integrated Real‑time Multi‑dimensional Analysis

This article presents a comprehensive overview of evolving big‑data architectures—from classic offline warehouses to Lambda and Kappa models—and details a streaming‑batch integrated solution that addresses latency, data freshness, and multi‑table join challenges to achieve minute‑level real‑time multi‑dimensional analytics.

Kappa architectureLambda architecturebatch processing
0 likes · 18 min read
Streaming‑Batch Integrated Real‑time Multi‑dimensional Analysis
Big Data Technology & Architecture
Big Data Technology & Architecture
Jan 3, 2023 · Big Data

Migrating Hive SQL Jobs to Flink Using the SQL Gateway

This article explains how to use Apache Flink 1.16's SQL Gateway to migrate Hive SQL tasks to Flink, covering the underlying Hive‑on‑Flink architecture, dialect compatibility, streaming and batch demos, configuration details, and practical tips for developers and platform engineers.

FlinkHiveSQL Gateway
0 likes · 19 min read
Migrating Hive SQL Jobs to Flink Using the SQL Gateway
Big Data Technology & Architecture
Big Data Technology & Architecture
Dec 15, 2022 · Big Data

Migrating Hive SQL to Flink SQL: Motivation, Challenges, Practice, Demo, and Future Plans

This technical article presents a comprehensive overview of migrating Hive SQL to Flink SQL, covering the motivations behind the migration, key challenges such as compatibility, stability and performance, practical implementation steps, a detailed demo, future development directions, and a Q&A session addressing common concerns.

FlinkHiveSQL migration
0 likes · 13 min read
Migrating Hive SQL to Flink SQL: Motivation, Challenges, Practice, Demo, and Future Plans
Architect's Tech Stack
Architect's Tech Stack
Dec 8, 2022 · Backend Development

Implementing Streaming Reads with MyBatis for Large-Scale Java Report Export

This article explains how to overcome export failures caused by large data volumes in a legacy Java system by switching from default full-result JDBC reads to a forward‑only streaming approach using MyBatis, detailing environment setup, configuration changes, and complete code examples for controller, service, DAO, and mapper layers.

JDBCJavaMyBatis
0 likes · 5 min read
Implementing Streaming Reads with MyBatis for Large-Scale Java Report Export
Bilibili Tech
Bilibili Tech
Nov 29, 2022 · Big Data

How Bilibili Supercharged Flink: Checkpoint, HA, and Runtime Optimizations

This article details Bilibili's extensive enhancements to Flink's runtime—including checkpoint recoverability, operator ID stability, state processor extensions, hybrid high‑availability, regional checkpointing, and load‑based channel selection—to improve scalability, reliability, and operational efficiency of large‑scale streaming jobs.

FlinkHigh AvailabilityRuntime Optimization
0 likes · 32 min read
How Bilibili Supercharged Flink: Checkpoint, HA, and Runtime Optimizations
Big Data Technology & Architecture
Big Data Technology & Architecture
Nov 28, 2022 · Big Data

Comprehensive Guide to Big Data Interview Topics: Log Collection, Data Synchronization, Offline Development, Real‑time Technology, Data Services, and Data Mining

This article provides an extensive overview of big‑data interview subjects, covering browser and mobile log collection methods, data synchronization techniques (batch, real‑time, sharding), offline data development platforms, streaming architectures, data service evolution, performance optimization, and data‑mining layers and applications.

Log Collectionbig datadata mining
0 likes · 17 min read
Comprehensive Guide to Big Data Interview Topics: Log Collection, Data Synchronization, Offline Development, Real‑time Technology, Data Services, and Data Mining
ITPUB
ITPUB
Nov 18, 2022 · Big Data

How Xiaomi Uses Iceberg for Real‑Time Streaming and Batch Data Lakes

This article introduces Iceberg’s table‑format fundamentals, details Xiaomi’s large‑scale deployment of Iceberg for CDC and log ingestion, explores their streaming‑batch integration experiments, outlines future roadmap items, and provides a comprehensive Q&A covering practical challenges and solutions.

FlinkIcebergSpark
0 likes · 23 min read
How Xiaomi Uses Iceberg for Real‑Time Streaming and Batch Data Lakes
DataFunTalk
DataFunTalk
Nov 6, 2022 · Big Data

BitSail: ByteDance’s Open‑Source Unified Data Integration Engine – Architecture, Evolution, and Capabilities

BitSail, an open‑source data integration engine from ByteDance, provides a unified solution for batch, streaming, full‑load, and incremental data synchronization across heterogeneous sources, detailing its background, technical evolution, architecture, low‑cost co‑building features, compatibility strategies, and future roadmap.

CDCFlinkdata integration
0 likes · 18 min read
BitSail: ByteDance’s Open‑Source Unified Data Integration Engine – Architecture, Evolution, and Capabilities
DataFunSummit
DataFunSummit
Nov 4, 2022 · Big Data

Real-Time Data Lake Practice at ByteDance: Architecture, Challenges, and Solutions

ByteDance’s data platform team explains their real‑time data lake implementation, covering its evolving definition, six core capabilities, challenges such as data management, concurrent updates, performance and log ingestion, and detailed case studies of multi‑stage deployment, indexing, metadata services, and future roadmap.

HudiIndexingReal-time Data Lake
0 likes · 32 min read
Real-Time Data Lake Practice at ByteDance: Architecture, Challenges, and Solutions
21CTO
21CTO
Nov 1, 2022 · Cloud Computing

How Netflix Engineers Seamless Streaming with Cloud‑Based Encoding and CDN

Netflix delivers billions of hours of video by compressing and transcoding raw movies into multiple formats, splitting them into small chunks processed in parallel on AWS, storing them in S3, and distributing them via its custom Open Connect CDN to ensure low‑latency, high‑quality playback worldwide.

AWSCDNNetflix
0 likes · 10 min read
How Netflix Engineers Seamless Streaming with Cloud‑Based Encoding and CDN
Mike Chen's Internet Architecture
Mike Chen's Internet Architecture
Oct 27, 2022 · Backend Development

Understanding Kafka: Architecture, Principles, Features, and Use Cases

This article explains Kafka's distributed publish‑subscribe architecture, detailing its core components, underlying mechanisms with Zookeeper coordination, key features such as high throughput and fault tolerance, and common application scenarios like log collection, user activity tracking, and stream processing.

Backend ArchitectureDistributed MessagingKafka
0 likes · 5 min read
Understanding Kafka: Architecture, Principles, Features, and Use Cases
DataFunTalk
DataFunTalk
Oct 19, 2022 · Big Data

Understanding Flink Table Store: Design, Usage, and Roadmap

Flink Table Store, an Apache Flink subproject, provides a unified stream‑batch storage layer with SQL‑based table APIs, addressing real‑time and offline data needs, detailing its design goals, usage patterns, architectural layers, implementation choices, and upcoming roadmap.

FlinkLSM-treeStorage
0 likes · 14 min read
Understanding Flink Table Store: Design, Usage, and Roadmap
ITPUB
ITPUB
Oct 15, 2022 · Big Data

Flink & Apache Hudi: Design, Practices, and Roadmap for Streaming Data Lakes

This talk introduces the evolution of data lakes, outlines Apache Hudi’s core features, details the Flink‑Hudi integration architecture—including write pipelines, small‑file handling, and read strategies—covers real‑world use cases such as near‑real‑time DB ingestion, OLAP, and ETL, and previews upcoming Hudi roadmap items.

Apache HudiFlinkbig data
0 likes · 21 min read
Flink & Apache Hudi: Design, Practices, and Roadmap for Streaming Data Lakes
Java High-Performance Architecture
Java High-Performance Architecture
Oct 11, 2022 · Operations

How Meituan Optimized Kafka for Massive Scale: Reducing Latency and Managing Clusters

This article details Meituan's real‑world challenges with a 15,000‑node Kafka deployment and explains the application‑layer and system‑layer optimizations—such as disk balancing, migration pipeline acceleration, fetcher isolation, RAID acceleration, cgroup isolation, and an SSD‑based cache—that together dramatically cut read/write latency and simplify large‑scale cluster management.

MeituanOptimizationcluster management
0 likes · 23 min read
How Meituan Optimized Kafka for Massive Scale: Reducing Latency and Managing Clusters
DataFunTalk
DataFunTalk
Oct 4, 2022 · Big Data

Near‑Real‑Time Data Lake Practices in TikTok E‑commerce Data Warehouse

The presentation by TikTok e‑commerce data‑warehouse engineer Ma Wenyuan explains data‑lake characteristics, near‑real‑time architecture, and practical e‑commerce use cases, highlighting Apache Hudi features, hybrid batch‑stream processing, and future challenges for scaling and integration.

HudiNear Real-TimeTikTok
0 likes · 13 min read
Near‑Real‑Time Data Lake Practices in TikTok E‑commerce Data Warehouse
DataFunTalk
DataFunTalk
Sep 17, 2022 · Big Data

Real-Time Data Warehouse Practices with Hudi at ByteDance

This presentation details ByteDance's real‑time data‑warehouse implementations using Apache Hudi, covering scenario classifications, challenges of traditional offline warehouses, practical solutions for ingestion, upsert, validation, indexing, query optimization, and future plans for extensible indexing and unified batch‑stream processing.

HudiOptimizationdata lake
0 likes · 16 min read
Real-Time Data Warehouse Practices with Hudi at ByteDance
Top Architect
Top Architect
Sep 17, 2022 · Big Data

Meituan's Kafka Architecture: Challenges and Optimizations at Massive Scale

This article details how Meituan's Kafka platform, serving over 15,000 machines and handling petabytes of daily traffic, faces read/write latency, slow nodes, and large‑scale cluster management challenges, and describes a series of application‑layer, system‑layer, and operational optimizations—including disk balancing, migration pipelines, fetcher isolation, consumer async, SSD caching, isolation strategies, full‑link monitoring, lifecycle management, and TOR disaster recovery—to improve performance and reliability.

KafkaMeituandistributed-systems
0 likes · 22 min read
Meituan's Kafka Architecture: Challenges and Optimizations at Massive Scale
DataFunTalk
DataFunTalk
Sep 11, 2022 · Big Data

Flink Table Store v0.2: Application Scenarios, Core Features, and Future Roadmap

This article introduces Flink Table Store v0.2, explains its four primary application scenarios—offline warehouse acceleration, partial update, pre‑aggregation rollup, and real‑time warehouse enhancement—details the core lake‑storage architecture, bucket management, append‑only mode, and outlines the project’s future roadmap and trade‑off considerations.

FlinkLake StorageSQL
0 likes · 16 min read
Flink Table Store v0.2: Application Scenarios, Core Features, and Future Roadmap
DataFunTalk
DataFunTalk
Aug 25, 2022 · Big Data

Applying OpenMLDB for Efficient AI Toolchain and Data‑Driven Architecture at Akulaku

This article presents Akulaku’s practical experience with OpenMLDB, describing the company’s data‑driven requirements, the design of a unified stream‑batch architecture, implementation details across offline, online and RocksDB modes, and future recommendations for high‑performance, scenario‑agnostic big‑data processing.

AIOpenMLDBbatch processing
0 likes · 17 min read
Applying OpenMLDB for Efficient AI Toolchain and Data‑Driven Architecture at Akulaku
Hulu Beijing
Hulu Beijing
Aug 19, 2022 · Artificial Intelligence

Disney’s M5 Model: Multi‑Modal, Multi‑Interest, Multi‑Scenario Boost for Streaming Recommendations

Disney’s Content Discovery team introduces M5, a multi‑modal, multi‑interest, multi‑scenario recall model that enhances VOD and live streaming recommendations by leveraging rich metadata, user behavior, and contextual features, outperforming baseline methods with significant hit‑ratio gains across Hulu and Disney+.

Deep LearningM5 modelMulti-modal
0 likes · 22 min read
Disney’s M5 Model: Multi‑Modal, Multi‑Interest, Multi‑Scenario Boost for Streaming Recommendations
ITPUB
ITPUB
Aug 13, 2022 · Big Data

How Alibaba Uses Flink to Power Massive Real‑Time Risk Control

This article explains how Alibaba leverages Flink to handle over 40 billion events per second across all business units, detailing risk‑control concepts, rule types, architectural stages, resource tuning, dynamic CEP, shared computing, and the FY23 roadmap for large‑scale streaming risk management.

AlibabaCEPFlink
0 likes · 16 min read
How Alibaba Uses Flink to Power Massive Real‑Time Risk Control
Wukong Talks Architecture
Wukong Talks Architecture
Aug 9, 2022 · Big Data

Kafka Basics: 15 Key Questions and In‑Depth Answers

This comprehensive guide covers Kafka’s core concepts, architecture, Zookeeper role, producer sending modes, partitioning strategies, replica types, message deletion, performance optimizations, consumer models, offset management, and best‑practice recommendations for scaling and ensuring ordered delivery in distributed streaming systems.

PartitioningZooKeeperbig-data
0 likes · 31 min read
Kafka Basics: 15 Key Questions and In‑Depth Answers
IT Architects Alliance
IT Architects Alliance
Aug 3, 2022 · Big Data

Understanding Kafka Architecture: Topics, Partitions, Replication, Log Segmentation, Zero‑Copy, and Zookeeper Integration

This article explains Kafka's core concepts—including topics, partitions and replicas, log segment storage, leader‑follower mechanics, consumer groups, network threading model, zero‑copy I/O, and the essential role of Zookeeper for broker, topic, consumer, and offset management—providing a comprehensive overview for developers and architects.

KafkaZero-CopyZooKeeper
0 likes · 10 min read
Understanding Kafka Architecture: Topics, Partitions, Replication, Log Segmentation, Zero‑Copy, and Zookeeper Integration
HomeTech
HomeTech
Jul 20, 2022 · Big Data

Design and Implementation of a Real-Time Advertising Data Warehouse Using Flink and StarRocks

This article presents a comprehensive case study of building a real‑time advertising data warehouse at Auto Home, detailing the evaluation of streaming engines and storage solutions, the layered architecture design, implementation steps with Flink and StarRocks, monitoring practices, encountered issues, and future roadmap, demonstrating how second‑level data freshness and high accuracy were achieved.

FlinkStarRocksadvertising analytics
0 likes · 10 min read
Design and Implementation of a Real-Time Advertising Data Warehouse Using Flink and StarRocks
Open Source Linux
Open Source Linux
Jul 19, 2022 · Backend Development

Master Kafka Basics: Visual Guide to Topics, Partitions, and Architecture

This article visually explains Kafka's core concepts—including producers, consumers, topics, partitions, consumer groups, and cluster architecture—so readers can clearly understand how messages flow, are stored, and remain fault‑tolerant within a distributed streaming system.

BackendKafkaMessage Queue
0 likes · 6 min read
Master Kafka Basics: Visual Guide to Topics, Partitions, and Architecture