BestHub
Sign in
Tagged articles

Streaming

586 articles · Page 4 of 6
Big Data Technology & Architecture
Big Data Technology & Architecture
Feb 23, 2022 · Big Data

Understanding Mini‑Batch Streaming Aggregation in Flink SQL

This article explains Flink SQL’s streaming aggregation Mini‑Batch feature, covering its purpose, configuration parameters, underlying optimizer rules, operator implementations, watermark handling, buffer processing, and the optional Local‑Global two‑phase aggregation optimization for improving throughput and reducing state overhead in large‑scale data pipelines.

AggregationBig DataFlink
0 likes · 10 min read
Understanding Mini‑Batch Streaming Aggregation in Flink SQL
Volcano Engine Developer Services
Volcano Engine Developer Services
Feb 16, 2022 · Big Data

ByteDance’s Journey to a Unified Data Lake with Flink and Hudi

This article recounts ByteDance’s evolution from batch‑only Flink pipelines to a unified data‑lake integration platform, detailing the three integration modes, challenges with Spark‑based CDC, the decision to adopt Hudi over Iceberg, and how Hudi’s indexing and Merge‑On‑Read formats enable near‑real‑time analytics at massive scale.

CDCFlinkHudi
0 likes · 10 min read
ByteDance’s Journey to a Unified Data Lake with Flink and Hudi
Architect
Architect
Feb 12, 2022 · Big Data

In-Depth Overview of Apache Kafka Architecture and Core Concepts

This article provides a comprehensive introduction to Apache Kafka, covering its distributed streaming platform features, message queue patterns, topic and partition design, broker and cluster roles, producer and consumer mechanics, partition assignment strategies, data storage, reliability guarantees, and performance optimizations such as zero‑copy and batch processing.

ConsumerMessage QueueProducer
0 likes · 23 min read
In-Depth Overview of Apache Kafka Architecture and Core Concepts
Su San Talks Tech
Su San Talks Tech
Jan 27, 2022 · Big Data

Why Kafka 2.8 Is Dropping Zookeeper and What It Means for You

This article explains how Kafka 2.8 removes its dependency on Zookeeper, describes the roles of brokers, topics, partitions, and the controller in the Zookeeper‑based architecture, and outlines the KIP‑500 upgrade that replaces Zookeeper with a quorum‑based KRaft controller to improve scalability and operational simplicity.

KIP-500KRaftStreaming
0 likes · 9 min read
Why Kafka 2.8 Is Dropping Zookeeper and What It Means for You
Baidu Geek Talk
Baidu Geek Talk
Jan 26, 2022 · Big Data

How a Real‑Time CDP Solves Data Silos: Architecture, Tech Choices & Lessons

This article examines the design and implementation of a tenant‑level real‑time Customer Data Platform, detailing CDP fundamentals, business and technical challenges, key architectural components, technology selections such as graph databases, stream processing, storage engines, and the operational practices that enable high‑throughput, low‑latency data integration and analytics.

CDPData IntegrationFlink
0 likes · 42 min read
How a Real‑Time CDP Solves Data Silos: Architecture, Tech Choices & Lessons
dbaplus Community
dbaplus Community
Jan 5, 2022 · Big Data

How ByteDance Optimized Flink SQL for Real‑World Streaming at Scale

This article details ByteDance's practical experience with Apache Flink, covering SQL extensions, a visual SQL platform, performance tweaks such as window mini‑batching and custom windows, join and checkpoint recovery improvements, stream‑batch integration experiments, and future roadmap plans.

Batch IntegrationCheckpointFlink
0 likes · 16 min read
How ByteDance Optimized Flink SQL for Real‑World Streaming at Scale
DataFunTalk
DataFunTalk
Jan 1, 2022 · Big Data

JD's Flink Journey: Evolution, Optimizations, and Future Directions

This article details JD's adoption of Flink for real‑time computing, covering its evolution from Storm to Flink on Kubernetes, the platform architecture, major optimization techniques such as preview topology, backpressure handling, dynamic rebalance, checkpoint‑as‑savepoint, and outlines future plans including stream‑batch integration, stability improvements, intelligent operations, and AI integration.

Big DataFlinkJD
0 likes · 10 min read
JD's Flink Journey: Evolution, Optimizations, and Future Directions
Tencent Cloud Developer
Tencent Cloud Developer
Dec 28, 2021 · Industry Insights

How Flink and ClickHouse Combine to Build High‑Performance Real‑Time Data Warehouses

This article analyzes the challenges of massive data query efficiency, explains how Flink's stream processing and ClickHouse's OLAP engine complement each other, and presents a layered real‑time data‑warehouse architecture with practical guidance on data ingestion, write strategies, quality assurance, and evolving batch‑stream integration patterns.

Big DataClickHouseFlink
0 likes · 19 min read
How Flink and ClickHouse Combine to Build High‑Performance Real‑Time Data Warehouses
Su San Talks Tech
Su San Talks Tech
Dec 28, 2021 · Big Data

What Makes Kafka the Backbone of Real‑Time Big Data Processing?

This article provides a comprehensive overview of Apache Kafka, covering its distributed architecture, key advantages and drawbacks, the role of ZooKeeper, message delivery semantics, partitioning strategies, storage mechanisms, and performance optimizations such as zero‑copy and batch processing, all essential for high‑throughput real‑time data pipelines.

Big DataDistributed MessagingStreaming
0 likes · 23 min read
What Makes Kafka the Backbone of Real‑Time Big Data Processing?
Youku Technology
Youku Technology
Dec 10, 2021 · Mobile Development

Overview and Architecture Design of Youku Playback SDK Kernel with Performance Optimization

The Youku Playback SDK kernel provides a cross‑platform, high‑reliability framework that decouples data acquisition, decoding, and rendering into independent AVSource, AVDecoder, and AVRender modules, enabling efficient thread usage, configurable builds for partners, adaptive buffering, health monitoring, and comprehensive error handling for optimal playback performance.

DRMMedia KernelSDK Architecture
0 likes · 10 min read
Overview and Architecture Design of Youku Playback SDK Kernel with Performance Optimization
Java Architect Essentials
Java Architect Essentials
Dec 7, 2021 · Big Data

Apache Kafka 3.0 Release Highlights and New Features

The article provides a comprehensive overview of Apache Kafka 3.0, detailing its core APIs, two main use‑cases, major feature additions, deprecations, KRaft consensus improvements, enhanced producer guarantees, and numerous KIP‑driven changes across the broker, client, Connect, Streams, and MirrorMaker components.

Apache KafkaEvent StreamingKIP
0 likes · 14 min read
Apache Kafka 3.0 Release Highlights and New Features
Programmer DD
Programmer DD
Nov 27, 2021 · Operations

How Netflix’s Open Connect CDN Powers Seamless Streaming Worldwide

Netflix’s Open Connect CDN, a proprietary content‑delivery network built over a decade, strategically places millions of server copies close to ISPs, uses multiple bitrate replicas, and dynamically shifts content to flash storage, ensuring high‑quality streaming even during peak demand and network outages.

CDNInfrastructureNetflix
0 likes · 12 min read
How Netflix’s Open Connect CDN Powers Seamless Streaming Worldwide
Tongcheng Travel Technology Center
Tongcheng Travel Technology Center
Nov 19, 2021 · Big Data

Real‑Time Data Warehouse Practices with Apache Kudu: Architecture, Partitioning, and Platformization

This article reviews the challenges of building a real‑time data warehouse, compares Lambda and Kappa architectures, introduces Apache Kudu’s master‑tablet design, storage model and partition strategies, and shares practical experiences and future directions for a Kudu‑based streaming analytics platform.

Apache KuduBig DataKappa Architecture
0 likes · 8 min read
Real‑Time Data Warehouse Practices with Apache Kudu: Architecture, Partitioning, and Platformization
Douyu Streaming
Douyu Streaming
Nov 12, 2021 · Fundamentals

How FLV and RTP Interact in Douyu’s Low‑Latency WebRTC Streaming

This article explains the end‑to‑end workflow of Douyu’s fast live streaming system, detailing how FLV tags are converted to RTP packets and back, covering WebRTC’s SDP/ICE/DTLS handshake, FLV and RTP header structures, payload formats for audio (OPUS) and video (H.264), and the server‑side processing pipeline.

FLVRTPStreaming
0 likes · 19 min read
How FLV and RTP Interact in Douyu’s Low‑Latency WebRTC Streaming
Big Data Technology & Architecture
Big Data Technology & Architecture
Nov 8, 2021 · Big Data

Understanding Flink CDC 2.0: Core Design, Snapshot & Incremental Reading, and Code Walkthrough

This article introduces Flink CDC 2.0, explains its distributed full‑load and incremental reading mechanisms, details the slice partitioning, snapshot correction, and binlog handling logic, and provides a complete Java example that demonstrates how to configure Flink SQL, MySQL source, and Kafka sink.

Big DataCDCData Integration
0 likes · 29 min read
Understanding Flink CDC 2.0: Core Design, Snapshot & Incremental Reading, and Code Walkthrough
Full-Stack Internet Architecture
Full-Stack Internet Architecture
Oct 29, 2021 · Cloud Native

RocketMQ 5.0 Overview: A Cloud‑Native Messaging, Event and Stream Fusion Platform

This article reviews the evolution of RocketMQ from its early MetaQ roots through the 4.x releases, explains the motivations behind RocketMQ 5.0, and details its cloud‑native architecture, lightweight SDK, storage‑compute separation, POP consumption model, elastic scaling, and the upcoming RocketMQ Streams framework.

Message QueueRocketMQStreaming
0 likes · 18 min read
RocketMQ 5.0 Overview: A Cloud‑Native Messaging, Event and Stream Fusion Platform
Big Data Technology & Architecture
Big Data Technology & Architecture
Oct 29, 2021 · Big Data

Dimension Table Join Strategies in Apache Flink: Preload, Distributed Cache, Hot Storage, Broadcast, and Temporal Table Function

The article explains various dimension‑table join approaches in Apache Flink, including preloading tables into memory, using distributed cache, leveraging hot storage with async I/O, broadcasting state, and temporal table function joins, and compares their trade‑offs for different data volumes and update frequencies.

Dimension TableFlinkJOIN
0 likes · 10 min read
Dimension Table Join Strategies in Apache Flink: Preload, Distributed Cache, Hot Storage, Broadcast, and Temporal Table Function
Big Data Technology & Architecture
Big Data Technology & Architecture
Oct 26, 2021 · Big Data

Practical Experience Building a Real‑Time Clickstream Data Warehouse with Flink and ClickHouse

This article shares practical insights on designing and operating a real‑time clickstream data warehouse using Flink for streaming processing and ClickHouse for near‑real‑time OLAP, covering dimensional modeling, layered architecture, Flink‑ClickHouse sink implementation, and data rebalancing strategies.

ClickHouseFlinkReal-Time Analytics
0 likes · 10 min read
Practical Experience Building a Real‑Time Clickstream Data Warehouse with Flink and ClickHouse
Big Data Technology & Architecture
Big Data Technology & Architecture
Oct 21, 2021 · Big Data

Comparative Overview of Open‑Source CDC Solutions: Debezium, Flink CDC, and Canal

This article provides a detailed comparison of three popular open‑source change data capture tools—Debezium, Flink CDC, and Canal—covering their underlying principles, architecture, deployment options, performance characteristics, and suitability for real‑time data synchronization in big‑data environments.

CDCCanalChange Data Capture
0 likes · 21 min read
Comparative Overview of Open‑Source CDC Solutions: Debezium, Flink CDC, and Canal
Tencent Cloud Developer
Tencent Cloud Developer
Oct 19, 2021 · Backend Development

Comprehensive Guide to gRPC Communication with Go and PHP: Protobuf, Streaming, TLS, and Timeout

This comprehensive guide walks you through creating a gRPC user service in Go and PHP, from defining protobuf messages and generating code, implementing server and client stubs, adding client, server, and bidirectional streaming, securing communication with TLS certificates, and managing request deadlines with timeout controls.

GoPHPStreaming
0 likes · 33 min read
Comprehensive Guide to gRPC Communication with Go and PHP: Protobuf, Streaming, TLS, and Timeout
Big Data Technology & Architecture
Big Data Technology & Architecture
Oct 12, 2021 · Big Data

Data Lake Evolution and a Practical Flink + Iceberg Implementation Guide

This article explores the evolution of data lakes, compares major cloud providers' lake architectures, introduces the emerging lakehouse concept, and provides a step‑by‑step Flink‑Iceberg implementation—including dependencies, catalog setup, table creation, checkpointing, and Kafka ingestion—demonstrating practical big‑data streaming solutions.

FlinkIcebergSQL
0 likes · 14 min read
Data Lake Evolution and a Practical Flink + Iceberg Implementation Guide
Big Data Technology Architecture
Big Data Technology Architecture
Oct 9, 2021 · Big Data

Apache Kafka 3.0 Release Highlights and New Features

Apache Kafka 3.0 introduces major enhancements including KRaft consensus, deprecation of Java 8 and Scala 2.12 support, stronger producer guarantees, updated APIs, improved Kafka Connect, MirrorMaker 2 flexibility, and numerous KIP-driven feature upgrades, marking a significant step forward for the distributed streaming platform.

KafkaKafka 3.0Streaming
0 likes · 13 min read
Apache Kafka 3.0 Release Highlights and New Features
Big Data Technology & Architecture
Big Data Technology & Architecture
Oct 9, 2021 · Big Data

Apache Flink 1.7–1.14 Release Highlights and Feature Evolution

This article provides a comprehensive overview of Apache Flink's major releases from version 1.7 to 1.14, detailing new APIs, state management improvements, Kubernetes integration, SQL and Table API enhancements, checkpointing advances, and performance optimizations that together illustrate the platform's evolution for both streaming and batch processing workloads.

Apache FlinkBatch ProcessingCheckpoint
0 likes · 78 min read
Apache Flink 1.7–1.14 Release Highlights and Feature Evolution
21CTO
21CTO
Oct 6, 2021 · Big Data

Building a Real-Time TB-Scale Bill Query System with Kafka, Kudu, and Presto

This article details the design and implementation of a real‑time, TB‑scale bill‑detail query platform that leverages Kafka for streaming, Debezium and Confluent Platform for change capture, Kudu for low‑latency storage, and Presto/Kylin for fast OLAP queries, while outlining deployment, integration, and future enhancements.

KafkaKuduStreaming
0 likes · 19 min read
Building a Real-Time TB-Scale Bill Query System with Kafka, Kudu, and Presto
Douyu Streaming
Douyu Streaming
Sep 27, 2021 · Game Development

Douyu’s Live2D Virtual Avatar Plugin: Unity Architecture & Key Tech

This article details Douyu's virtual avatar tool built on Unity and Live2D, covering project background, core features, layered architecture, key technologies such as Unity rendering, FairyGUI UI, TCP-based IPC, model dressing and recoloring, face‑data processing, and future development plans.

Game DevelopmentIPCLive2D
0 likes · 13 min read
Douyu’s Live2D Virtual Avatar Plugin: Unity Architecture & Key Tech
Cloud Native Technology Community
Cloud Native Technology Community
Sep 26, 2021 · Big Data

Apache Kafka 3.0.0 Release Summary: New Features, Improvements, Bugs, Tasks, and Tests

Apache Kafka 3.0.0, released on September 21, 2021, introduces major changes such as deprecating Java 8 and Scala 2.12, adding Raft‑based metadata quorum, stronger producer delivery guarantees, removal of old message formats, numerous performance optimizations, extensive bug fixes, and a large set of new and updated JIRA issues across features, improvements, bugs, tasks, tests, and subtasks.

ApacheBig DataKafka3.0
0 likes · 37 min read
Apache Kafka 3.0.0 Release Summary: New Features, Improvements, Bugs, Tasks, and Tests
Programmer DD
Programmer DD
Sep 26, 2021 · Big Data

What’s New in Apache Kafka 3.0? Key Features and Improvements Explained

Apache Kafka 3.0.0 introduces a host of enhancements—including deprecated Java 8/Scala 2.12 support, Raft metadata snapshots, stronger producer guarantees, MirrorMaker 2 upgrades, and Kafka Streams improvements—while continuing to serve real‑time data pipelines and streaming applications.

Apache KafkaBig DataData pipelines
0 likes · 3 min read
What’s New in Apache Kafka 3.0? Key Features and Improvements Explained
IT Architects Alliance
IT Architects Alliance
Sep 25, 2021 · Big Data

Apache Kafka 3.0.0 Release: New Features, API Changes, and KRaft Improvements

Apache Kafka 3.0.0 introduces numerous enhancements including deprecation of Java 8 and Scala 2.12 support, KRaft metadata snapshots, stronger default producer delivery guarantees, expanded Connect and Streams APIs, updated MirrorMaker 2 configuration, and many KIP-driven feature and API changes for improved streaming and event processing.

Apache KafkaEvent ProcessingKIP
0 likes · 15 min read
Apache Kafka 3.0.0 Release: New Features, API Changes, and KRaft Improvements
Big Data Technology & Architecture
Big Data Technology & Architecture
Sep 10, 2021 · Big Data

Understanding Flink Table API and SQL: Dependencies, Planners, and Practical Usage

This article provides a comprehensive guide to Apache Flink's Table API and SQL, covering required dependencies, the differences between old and Blink planners, program structure, table environment creation, catalog registration, query execution, conversion between DataStream and Table, update modes, and time attribute handling, with Scala code examples throughout.

FlinkSQLScala
0 likes · 26 min read
Understanding Flink Table API and SQL: Dependencies, Planners, and Practical Usage
Big Data Technology & Architecture
Big Data Technology & Architecture
Aug 24, 2021 · Big Data

Comprehensive Overview of Data Lake Technologies: Iceberg, Hudi, and Delta Lake

This article provides an in-depth overview of data lake concepts, definitions, and essential features, followed by detailed case studies of enterprise data lake implementations and comparative analysis of leading data lake table formats—Iceberg, Hudi, and Delta Lake—highlighting their architectures, capabilities, and trade‑offs.

Delta LakeFlinkHudi
0 likes · 19 min read
Comprehensive Overview of Data Lake Technologies: Iceberg, Hudi, and Delta Lake
Big Data Technology & Architecture
Big Data Technology & Architecture
Aug 21, 2021 · Big Data

Kafka Overview: Background, Core Concepts, Producer/Consumer Configuration, Core Principles, Operations, and Stream Processing

This article provides a comprehensive beginner-friendly guide to Apache Kafka, covering its background, core concepts, producer and consumer settings with code examples, underlying architecture, operational monitoring, integration with Spark and Flink, and an introduction to Kafka Streams.

ConsumerJavaProducer
0 likes · 19 min read
Kafka Overview: Background, Core Concepts, Producer/Consumer Configuration, Core Principles, Operations, and Stream Processing
iQIYI Technical Product Team
iQIYI Technical Product Team
Aug 13, 2021 · Game Development

Improving VR Video Clarity: PPD, Tile Encoding, and Future Directions

VR video clarity suffers because the required pixels‑per‑degree far exceed what 4K or 8K spherical streams can deliver, but tile‑based encoding that decodes only the viewport, combined with low motion‑to‑photon latency, distortion control, advanced codecs and AI‑driven projection, promises sharper, lower‑bitrate 6DoF experiences.

8KPPDStreaming
0 likes · 13 min read
Improving VR Video Clarity: PPD, Tile Encoding, and Future Directions
Ctrip Technology
Ctrip Technology
Aug 5, 2021 · Frontend Development

Understanding React Server Components: Concepts, Usage, and Implementation

This article explains the motivation, component types, naming conventions, runtime mechanism, streaming protocol, design goals, and practical considerations of React Server Components, illustrating how they reduce client bundle size and enable progressive server‑side rendering with code examples.

Code SplittingPerformanceReAct
0 likes · 15 min read
Understanding React Server Components: Concepts, Usage, and Implementation
Big Data Technology & Architecture
Big Data Technology & Architecture
Jul 30, 2021 · Big Data

Enterprise Big Data Platform Architecture: Insights from Taobao, Meituan, and Didi

This article examines the architecture of enterprise-level big data platforms at leading Chinese tech firms—Taobao, Meituan, and Didi—detailing their data sources, synchronization components, batch and streaming processing layers, scheduling systems, and common design patterns, while highlighting shared principles across these implementations.

Batch ProcessingStreamingarchitecture
0 likes · 9 min read
Enterprise Big Data Platform Architecture: Insights from Taobao, Meituan, and Didi
Big Data Technology & Architecture
Big Data Technology & Architecture
Jul 27, 2021 · Big Data

An Introduction to Apache Pulsar: Core Concepts, Architecture, and Key Features

Apache Pulsar is a cloud‑native distributed messaging platform that combines messaging, storage, and lightweight compute, featuring multi‑tenant support, geo‑replication, and high throughput, and this article introduces its core concepts, architecture components such as brokers, BookKeeper, ZooKeeper, and key design features.

Apache PulsarBookKeeperCloud Native
0 likes · 13 min read
An Introduction to Apache Pulsar: Core Concepts, Architecture, and Key Features
DataFunTalk
DataFunTalk
Jul 26, 2021 · Big Data

Accelerating Hive Daily Tables with Flink: A SmartNews Case Study

This article describes how SmartNews integrated Flink into its Airflow‑driven Hive batch pipeline to cut the actions table generation latency from three hours to about thirty‑four minutes, detailing the technical challenges, design decisions, and production results.

AWSBig DataData Pipeline
0 likes · 12 min read
Accelerating Hive Daily Tables with Flink: A SmartNews Case Study
Big Data Technology & Architecture
Big Data Technology & Architecture
Jul 20, 2021 · Big Data

Common Issues and Solutions for Flink CDC with MySQL

This article summarizes frequent problems encountered when using Flink CDC with MySQL—including Kafka version conflicts, checkpoint timeouts, permission errors, global lock issues, and DDL parsing failures—and provides practical configuration tweaks and code examples to resolve them.

CDCCheckpointDebezium
0 likes · 11 min read
Common Issues and Solutions for Flink CDC with MySQL
Open Source Linux
Open Source Linux
Jul 17, 2021 · Big Data

Master Kafka Basics: Topics, Partitions, Producers & Consumers Explained

This article provides a clear, visual guide to Kafka’s core concepts—including producers, consumers, topics, partitions, consumer groups, message ordering, and the underlying ZooKeeper‑managed cluster architecture—helping readers grasp how Kafka enables reliable, scalable stream processing.

Big DataConsumersProducers
0 likes · 6 min read
Master Kafka Basics: Topics, Partitions, Producers & Consumers Explained
Big Data Technology & Architecture
Big Data Technology & Architecture
Jul 9, 2021 · Big Data

Understanding Kafka: Use Cases, Reliability, Storage, Replication, Consumer Assignment, Transactions, and Exactly-Once Semantics

This article explains why Kafka is used, its buffering, decoupling, redundancy and robustness benefits, details the ack reliability levels, storage design, replica synchronization, ISR handling, consumer partition assignment strategies, transaction support, exactly‑once semantics, and why read‑write separation is not provided.

Consumer AssignmentMessage QueueReplication
0 likes · 20 min read
Understanding Kafka: Use Cases, Reliability, Storage, Replication, Consumer Assignment, Transactions, and Exactly-Once Semantics
Architect
Architect
Jul 7, 2021 · Big Data

Understanding Kafka High Availability and Resolving Consumer Offset Issues

This article explains Kafka's high‑availability architecture, including multi‑replica design, ISR synchronization, leader election, acks configuration, and how misconfigured __consumer_offset replication can cause consumer outages, offering practical steps to ensure reliable message delivery.

Consumer offsetReplicationStreaming
0 likes · 8 min read
Understanding Kafka High Availability and Resolving Consumer Offset Issues
Architecture Digest
Architecture Digest
Jul 3, 2021 · Fundamentals

Message Exchange Patterns: Architecture and Routing

This article explains the fundamental message exchange patterns—including publish‑subscribe, fan‑out, unidirectional and bidirectional streaming—as well as routing models such as unicast, broadcast, multicast, and anycast, illustrating each with common technology examples.

RoutingStreamingmessaging
0 likes · 8 min read
Message Exchange Patterns: Architecture and Routing
DataFunTalk
DataFunTalk
Jun 29, 2021 · Big Data

In-depth Analysis of Flink SQL 1.13 Features and Improvements

This article provides a comprehensive overview of Apache Flink SQL 1.13, detailing new Window TVF support, cumulate windows, performance optimizations, time‑zone handling, enhanced Hive compatibility, SQL client upgrades, DataStream‑Table conversion improvements, and outlines the roadmap for the upcoming 1.14 release.

DataStreamFlinkHive Integration
0 likes · 15 min read
In-depth Analysis of Flink SQL 1.13 Features and Improvements
360 Tech Engineering
360 Tech Engineering
Jun 25, 2021 · Big Data

Introducing ULTRON: A Real‑Time Data Warehouse Platform Powered by FlinkSQL

ULTRON is a one‑stop real‑time data‑warehouse development platform built on FlinkSQL that unifies data integration, asset management, cluster deployment, modeling, ETL, OLAP analysis and governance, addressing the limitations of traditional batch‑oriented warehouses and simplifying streaming data workflows for developers.

FlinkSQLReal-time Data WarehouseStreaming
0 likes · 13 min read
Introducing ULTRON: A Real‑Time Data Warehouse Platform Powered by FlinkSQL
Alibaba Terminal Technology
Alibaba Terminal Technology
Jun 25, 2021 · Frontend Development

Mastering Web Multimedia Front‑End: A Complete Beginner’s Guide

This comprehensive guide introduces multimedia front‑end development, explains W3C media standards and HTML elements, explores media APIs, outlines playback scenarios and solutions, and details both consumer‑facing live video systems and production‑side tools such as streaming and video‑editing, while sharing Alibaba’s roadmap for the field.

Streamingmedia APIsmultimedia
0 likes · 25 min read
Mastering Web Multimedia Front‑End: A Complete Beginner’s Guide
Sohu Tech Products
Sohu Tech Products
Jun 9, 2021 · Big Data

Real-time UV Counting with Flink, Hologres, and RoaringBitmap

This article explains how to implement both offline (T+1) and real‑time UV counting using Hologres with RoaringBitmap for high‑cardinality aggregation, and demonstrates a complete Flink‑Hologres pipeline—including table creation, streaming joins, windowed aggregation, and query examples—for fine‑grained user metric analysis.

FlinkHologresRoaringBitmap
0 likes · 11 min read
Real-time UV Counting with Flink, Hologres, and RoaringBitmap
Top Architect
Top Architect
Jun 8, 2021 · Backend Development

Architectural Messaging Patterns: Exchange Architectures and Routing Methods

This article explains the fundamental messaging exchange architectures such as Pub‑Sub, Fanout, Unidirectional and Bidirectional streaming, and the routing patterns including Unicast, Broadcast, Multicast and Anycast, illustrating how they are used in systems like Redis, Kafka, RabbitMQ and IBM MQ to simplify communication between producers and consumers.

FanoutRoutingStreaming
0 likes · 8 min read
Architectural Messaging Patterns: Exchange Architectures and Routing Methods
dbaplus Community
dbaplus Community
Jun 5, 2021 · Big Data

How Flink + Iceberg Transform Data Lakes for Real‑Time Streaming

This article explains the concept of data lakes, outlines a four‑layer open‑source architecture, presents several classic Flink‑Iceberg use cases, details why Iceberg was chosen, and describes the design of Flink’s streaming sink and upcoming community roadmap.

Apache FlinkApache IcebergBig Data
0 likes · 14 min read
How Flink + Iceberg Transform Data Lakes for Real‑Time Streaming
Big Data Technology & Architecture
Big Data Technology & Architecture
Jun 3, 2021 · Big Data

Comparing Apache Pulsar and Apache Kafka: Architecture, Performance, Use Cases, and Ecosystem

This article compares Apache Pulsar and Apache Kafka across performance, architecture, features, and real‑world use cases, highlighting Pulsar’s multi‑layer design, scalability, client language support, ecosystem integrations, and operational advantages while providing detailed analysis of storage, messaging models, and community resources.

Apache PulsarCloud NativeMessage Queue
0 likes · 28 min read
Comparing Apache Pulsar and Apache Kafka: Architecture, Performance, Use Cases, and Ecosystem
Big Data Technology & Architecture
Big Data Technology & Architecture
Jun 1, 2021 · Big Data

Understanding Idle State Retention Time in Flink SQL

Flink SQL's idle state retention time feature prevents state explosion by automatically cleaning up state for keys that remain inactive beyond a configurable time window, requiring both minimum and maximum retention settings, with implementation details involving CleanupState, timers, and KeyedProcessFunctionWithCleanupState.

FlinkIdle State RetentionSQL
0 likes · 8 min read
Understanding Idle State Retention Time in Flink SQL
New Oriental Technology
New Oriental Technology
May 31, 2021 · Fundamentals

Live Streaming Network Transmission: Protocols, Encoding, Decoding, and Synchronization

This article explains the end‑to‑end live‑streaming workflow, covering how a broadcaster pushes video and audio to a server, the various streaming protocols (RTMP, HTTP‑FLV, HLS, RTP), encoding formats, FFmpeg‑based decoding, hardware vs software decoding, and audio‑video synchronization techniques.

Audio-Video SyncFFmpegStreaming
0 likes · 26 min read
Live Streaming Network Transmission: Protocols, Encoding, Decoding, and Synchronization
IT Architects Alliance
IT Architects Alliance
May 30, 2021 · Big Data

NetEase Game Streaming ETL Architecture and Practices Based on Flink

This article presents NetEase Game's Flink‑based streaming ETL system, detailing business background, log classifications, specialized and generic ETL services, Python UDF integration, runtime optimizations, HDFS write tuning, SLA metrics, fault‑tolerance mechanisms, and future roadmap for unified data lakes and PyFlink support.

Big DataData IntegrationETL
0 likes · 19 min read
NetEase Game Streaming ETL Architecture and Practices Based on Flink
Byte Quality Assurance Team
Byte Quality Assurance Team
May 19, 2021 · Big Data

Streaming 102: The World Beyond Batch

This article extends the concepts introduced in Streaming 101 by deeply exploring data processing patterns for unbounded data, covering windowing, watermarks, triggers, accumulation modes, and their practical implications for building robust low‑latency streaming pipelines.

Big DataStreamingTriggers
0 likes · 14 min read
Streaming 102: The World Beyond Batch
vivo Internet Technology
vivo Internet Technology
May 12, 2021 · Backend Development

Understanding RTMP Protocol and Livego Source Code Analysis

The article explains RTMP’s multiplexed, packetized streaming over TCP, detailing its chunk structure, message types, handshake, and connection workflow, then demonstrates livego’s publishing and pulling processes, discusses typical latency sources, and offers mitigation strategies and reference resources for developers.

GoLivegoStreaming
0 likes · 28 min read
Understanding RTMP Protocol and Livego Source Code Analysis
DataFunTalk
DataFunTalk
May 11, 2021 · Big Data

Design and Practice of Baixin Bank's Flink‑Based Real‑Time Computing Platform and Hudi‑Powered Real‑Time Data Lake

This article details Baixin Bank's construction of a Flink‑driven real‑time computing platform integrated with Hudi as a real‑time data lake, covering background, architecture, data collection, transformation, storage layers, technical challenges, future roadmap, and practical lessons for similar big‑data initiatives.

Big DataFlinkHudi
0 likes · 12 min read
Design and Practice of Baixin Bank's Flink‑Based Real‑Time Computing Platform and Hudi‑Powered Real‑Time Data Lake
IT Architects Alliance
IT Architects Alliance
May 11, 2021 · Big Data

Demystifying Kafka: Core Concepts of Topics, Partitions, and Architecture

This article provides a clear, visual walkthrough of Kafka’s fundamental architecture, explaining how producers and consumers interact, the role of topics and partitions, consumer groups, and ZooKeeper’s coordination, helping readers grasp message flow, storage, ordering, and fault‑tolerance in a distributed streaming system.

Consumer GroupKafkaMessage Queue
0 likes · 6 min read
Demystifying Kafka: Core Concepts of Topics, Partitions, and Architecture
DataFunTalk
DataFunTalk
May 2, 2021 · Big Data

Continuous Optimization and Practice of Flink at Kuaishou

This article presents Kuaishou's comprehensive engineering practices for improving Flink's stability, task startup latency, and SQL performance, including high‑availability Kafka connectors, fault‑recovery mechanisms, I/O reductions, asynchronous job upgrades, aggregation optimizations, and future resource‑utilization plans.

Big DataFlinkKafka
0 likes · 10 min read
Continuous Optimization and Practice of Flink at Kuaishou
Programmer DD
Programmer DD
Apr 30, 2021 · Big Data

Kafka 2.8.0 Release: Say Goodbye to ZooKeeper with Raft Metadata Mode

Kafka 2.8.0, released on April 19, 2021, introduces the groundbreaking Raft Metadata mode that eliminates the need for ZooKeeper, alongside numerous new features, bug fixes, and enhancements such as API controls for stream threads, SASL_SSL mutual TLS, and IP rate limiting.

Big DataKafkaRaft
0 likes · 5 min read
Kafka 2.8.0 Release: Say Goodbye to ZooKeeper with Raft Metadata Mode
DataFunTalk
DataFunTalk
Apr 23, 2021 · Big Data

Building and Evolving Zhihu’s Flink‑Based Data Integration Platform

This article details Zhihu’s transition from a Sqoop‑driven data integration system to a Flink‑centric platform, covering business scenarios, historical architecture, design goals, technology choices, performance optimizations, and future plans for unified streaming‑batch processing across diverse storage systems.

Batch ProcessingBig DataData Integration
0 likes · 14 min read
Building and Evolving Zhihu’s Flink‑Based Data Integration Platform
Laravel Tech Community
Laravel Tech Community
Apr 22, 2021 · Big Data

Apache Kafka 2.8.0 Release Highlights and New Features

Apache Kafka 2.8.0 introduces several significant enhancements, including a new group API, mutual TLS authentication for SASL_SSL listeners, JSON request/response logging, broker connection rate limiting, topic identifiers, self‑managed quorum replacing ZooKeeper, and numerous improvements to Streams and Connect APIs for more reliable real‑time data pipelines.

Apache KafkaBig DataKafka 2.8.0
0 likes · 2 min read
Apache Kafka 2.8.0 Release Highlights and New Features
Tencent Cloud Developer
Tencent Cloud Developer
Apr 14, 2021 · Cloud Native

Apache Pulsar Meetup Shenzhen: Cloud-Native Distributed Messaging and Streaming Platform

The Apache Pulsar Meetup in Shenzhen on April 17, 2021, co‑hosted by Tencent Middleware and StreamNative, will showcase Pulsar’s cloud‑native messaging, streaming, and storage capabilities through sessions on KoP migration, big‑data and IoT use cases, cloud‑native deployments, and the StreamNative Cloud Pulsar‑as‑a‑Service offering.

Apache PulsarMeetupStreaming
0 likes · 7 min read
Apache Pulsar Meetup Shenzhen: Cloud-Native Distributed Messaging and Streaming Platform
ByteFE
ByteFE
Apr 13, 2021 · Frontend Development

Streaming Server‑Side Rendering in React: Concepts, lazy, Suspense, and Implementation

This article explains the principles of streaming server‑side rendering (SSR) in React, compares it with traditional client‑side rendering, and demonstrates how lazy loading and Suspense can be used together with streaming SSR to parallelize data and JavaScript delivery for faster first‑paint and improved user experience.

@LazyFrontendReAct
0 likes · 10 min read
Streaming Server‑Side Rendering in React: Concepts, lazy, Suspense, and Implementation
Programmer DD
Programmer DD
Mar 29, 2021 · Big Data

Mastering Kafka: High‑Throughput Distributed Messaging Explained

This comprehensive guide introduces Kafka as a high‑throughput, distributed, publish‑subscribe messaging system, detailing its core concepts, architecture, features, replication, log management, reliability guarantees, and typical use cases such as log collection, real‑time analytics, and cross‑cluster mirroring.

Big DataDistributed MessagingKafka
0 likes · 15 min read
Mastering Kafka: High‑Throughput Distributed Messaging Explained
Big Data Technology & Architecture
Big Data Technology & Architecture
Mar 18, 2021 · Big Data

Flink Job Troubleshooting and Performance Optimization: Data Skew, Kafka Configuration, Resource Management, and Checkpoint Issues

This article details common Flink streaming problems such as data skew causing task back‑pressure, oversized Kafka messages, high‑throughput ack settings, slot removal errors, checkpoint timeouts, and resource constraints, and provides concrete configuration changes and architectural adjustments to resolve them.

CheckpointData SkewFlink
0 likes · 18 min read
Flink Job Troubleshooting and Performance Optimization: Data Skew, Kafka Configuration, Resource Management, and Checkpoint Issues
DataFunTalk
DataFunTalk
Mar 15, 2021 · Big Data

Ten Gotchas When Migrating Spark Jobs to Flink

This article shares ten practical pitfalls encountered while moving hour‑level Spark session processing jobs to Apache Flink, covering parallelism skew, state TTL, checkpoint handling, logging, debugging, state migration, Reduce vs Process, input validation, event‑time handling, and the trade‑offs of storing data inside Flink.

Big DataFlinkPerformance
0 likes · 19 min read
Ten Gotchas When Migrating Spark Jobs to Flink
Big Data Technology & Architecture
Big Data Technology & Architecture
Mar 2, 2021 · Big Data

An Introduction to Kafka Connect: Architecture, Components, and Hands‑On Setup

This article introduces Kafka Connect, explaining its purpose as a scalable and reliable tool for moving data between Apache Kafka and external systems, detailing its core concepts, architecture, deployment modes, configuration files, and a step‑by‑step example that streams data from a file source to a file sink.

Data IntegrationETLKafka Connect
0 likes · 12 min read
An Introduction to Kafka Connect: Architecture, Components, and Hands‑On Setup
Laravel Tech Community
Laravel Tech Community
Feb 28, 2021 · Big Data

Apache Beam 2.28.0 Release Highlights and New Features

Apache Beam 2.28.0 introduces extensive Parquet support, new hash functions in BeamSQL and ZetaSQL, ApproximateDistinct via HLL, enhanced I/O connectors including SpannerIO for Numeric fields, ParquetIO schema support, KafkaTableProvider thrift, HadoopFormatIO key/value cloning skip, and various other improvements.

Apache BeamBig DataData Processing
0 likes · 3 min read
Apache Beam 2.28.0 Release Highlights and New Features
dbaplus Community
dbaplus Community
Feb 23, 2021 · Big Data

How NetEase Game Teams Built a Scalable Flink‑Based Streaming ETL Platform

This article explains how NetEase games collect heterogeneous logs, design a Flink‑driven streaming ETL pipeline, handle schema‑free sources, implement Python UDFs with Jython, optimize HDFS writes, manage real‑time and offline warehouses, and share practical tuning and fault‑tolerance techniques.

Data PipelineETLFlink
0 likes · 22 min read
How NetEase Game Teams Built a Scalable Flink‑Based Streaming ETL Platform
DataFunTalk
DataFunTalk
Feb 22, 2021 · Big Data

Optimizing Flink Real-Time Task Resources: Memory and Message Processing Perspectives

This article explores practical methods for optimizing Flink real‑time task resources on Kubernetes, focusing on memory usage analysis via GC logs and message‑processing capacity assessment, proposing automated detection of over‑provisioned memory and CPU, and outlining a workflow for resource adjustment to reduce costs.

Big DataFlinkGC Analysis
0 likes · 18 min read
Optimizing Flink Real-Time Task Resources: Memory and Message Processing Perspectives
dbaplus Community
dbaplus Community
Feb 18, 2021 · Big Data

How JD Search Scaled Real‑Time Analytics with Flink and Doris

This article details JD Search's journey from a Storm‑based pipeline to a Flink‑driven architecture backed by Apache Doris, covering business requirements, technical challenges, design trade‑offs, performance optimizations for massive traffic spikes, and future plans for their real‑time OLAP data warehouse.

Big DataDorisFlink
0 likes · 12 min read
How JD Search Scaled Real‑Time Analytics with Flink and Doris
Sohu Tech Products
Sohu Tech Products
Feb 17, 2021 · Big Data

Dynamic Broadcast State and Data Partitioning in an Apache Flink Fraud Detection Engine

This article demonstrates how to initialize, broadcast, and dynamically update rule sets in an Apache Flink fraud detection pipeline, using BroadcastProcessFunction and MapState to achieve runtime data partitioning without recompiling, and explains the underlying data exchange patterns such as forward, hash, rebalance, and broadcast.

Apache FlinkBroadcast StateDynamic Key Function
0 likes · 11 min read
Dynamic Broadcast State and Data Partitioning in an Apache Flink Fraud Detection Engine
DataFunTalk
DataFunTalk
Feb 17, 2021 · Big Data

Apache Iceberg 0.11.0: New Partition Support, SortOrder, Flink Streaming Reader, and Ecosystem Integrations

The article details Apache Iceberg 0.11.0's core enhancements—including partition changes, SortOrder, extensive Flink and Spark integrations, CDC/Upsert support, hash‑based write distribution to reduce small files, and upcoming 0.12.0 roadmap—while providing practical SQL and API examples for data‑lake practitioners.

Apache IcebergBig DataCDC
0 likes · 13 min read
Apache Iceberg 0.11.0: New Partition Support, SortOrder, Flink Streaming Reader, and Ecosystem Integrations
Big Data Technology & Architecture
Big Data Technology & Architecture
Feb 7, 2021 · Big Data

Building a Flink SQL Platform on Zeppelin: Installation, Configuration, and Advanced Use Cases

This guide walks through setting up Apache Zeppelin as a low‑cost, SQL‑centric development platform for Flink, covering environment preparation, installation, interpreter configuration, execution modes, verification, common pitfalls, dimension‑table joins, custom UDFs, Redis integration, and dual‑stream join techniques.

FlinkRedisSQL
0 likes · 24 min read
Building a Flink SQL Platform on Zeppelin: Installation, Configuration, and Advanced Use Cases
Full-Stack Internet Architecture
Full-Stack Internet Architecture
Feb 1, 2021 · Big Data

Kafka Overview: Architecture, Advantages, Disadvantages, and Core Concepts

This article provides a comprehensive introduction to Apache Kafka, covering its distributed publish‑subscribe architecture, its key components such as brokers, topics, partitions, producers, consumers, and ZooKeeper, as well as its advantages, drawbacks, storage mechanisms, partition assignment strategies, and reliability guarantees for high‑throughput big‑data streaming.

Big DataMessage QueueStreaming
0 likes · 20 min read
Kafka Overview: Architecture, Advantages, Disadvantages, and Core Concepts
DataFunTalk
DataFunTalk
Feb 1, 2021 · Big Data

Building a Real-Time Data Warehouse with Apache Flink and Apache Iceberg: Architecture, Challenges, and Best Practices

This article presents Tencent's experience of constructing a real‑time data warehouse by integrating Apache Flink with Apache Iceberg, covering background pain points, Iceberg's table format and capabilities, Flink‑Iceberg streaming and batch processing, practical implementations, and future roadmap for data‑lake acceleration.

Apache FlinkApache IcebergBig Data
0 likes · 21 min read
Building a Real-Time Data Warehouse with Apache Flink and Apache Iceberg: Architecture, Challenges, and Best Practices