BestHub
Sign in
Tagged articles

Streaming

586 articles · Page 2 of 6
iQIYI Technical Product Team
iQIYI Technical Product Team
Mar 27, 2025 · Big Data

Cost‑Effective Real‑Time Data Warehouse 2.0: Migrating from Kafka to Iceberg

iQIYI transformed its real‑time data warehouse by replacing a costly Kafka‑based Lambda stack with a unified stream‑batch Iceberg lake, cutting storage expenses by 90%, halving compute costs, extending data retention, and delivering minute‑level freshness for 90% of use cases while preserving second‑level processing where needed.

Cost OptimizationFlinkIceberg
0 likes · 11 min read
Cost‑Effective Real‑Time Data Warehouse 2.0: Migrating from Kafka to Iceberg
Airbnb Technology Team
Airbnb Technology Team
Mar 24, 2025 · Artificial Intelligence

Chronon: Open‑Source Feature Platform for Machine Learning – Architecture, Workflow, and Code Examples

Chronon is an open‑source ML feature platform that lets engineers declaratively define, compute, and serve both batch and real‑time features with built‑in observability, data‑quality checks, and a low‑latency retrieval API, ensuring online‑offline consistency while simplifying pipeline management and enabling future automation.

ChrononData pipelinesObservability
0 likes · 13 min read
Chronon: Open‑Source Feature Platform for Machine Learning – Architecture, Workflow, and Code Examples
Top Architecture Tech Stack
Top Architecture Tech Stack
Mar 24, 2025 · Backend Development

Using Spring's ResponseBodyEmitter for Real‑Time Log Streaming

This article explains how Spring Framework's ResponseBodyEmitter enables real‑time, chunked HTTP responses for use cases such as log streaming, progress updates, chat, and AI output, detailing its advantages over SSE, usage scenarios, core methods, a complete controller example, and best‑practice considerations.

JavaResponseBodyEmitterSpring Boot
0 likes · 9 min read
Using Spring's ResponseBodyEmitter for Real‑Time Log Streaming
Big Data Technology & Architecture
Big Data Technology & Architecture
Mar 24, 2025 · Big Data

Apache Kafka 4.0: Major New Features – KRaft Architecture, Consumer Group Protocol, Queue Mode, Java Upgrade, API Simplifications and More

Apache Kafka 4.0 introduces a groundbreaking KRaft architecture that removes ZooKeeper, a revamped consumer‑group protocol that dramatically speeds up rebalancing, a new queue mode for point‑to‑point messaging, upgraded Java requirements, streamlined APIs, and numerous performance and security enhancements, reshaping both development and operations for large‑scale streaming workloads.

Java 11Java 17KRaft
0 likes · 12 min read
Apache Kafka 4.0: Major New Features – KRaft Architecture, Consumer Group Protocol, Queue Mode, Java Upgrade, API Simplifications and More
Laravel Tech Community
Laravel Tech Community
Mar 23, 2025 · Big Data

Apache Kafka 4.0 Released: First Version Without ZooKeeper and New Features

Apache Kafka 4.0 has been officially released as the first major version that runs entirely without Apache ZooKeeper, introducing KRaft mode, a new consumer group protocol (KIP‑848), early‑access queue support (KIP‑932), updated Java requirements, and other enhancements aimed at improving scalability, operability, and messaging versatility.

Apache KafkaKIP-848KIP-932
0 likes · 3 min read
Apache Kafka 4.0 Released: First Version Without ZooKeeper and New Features
Alimama Tech
Alimama Tech
Mar 12, 2025 · Big Data

Design and Evolution of Alibaba Advertising Real-Time Data Warehouse

Alibaba Mama’s advertising platform migrated from a monolithic Flink‑Kafka pipeline to a layered Paimon lakehouse, adding DWS upsert support and multi‑layer storage, which delivers minute‑level data freshness, cuts latency by 2.5 hours, reduces resource use over 40 %, halves development effort and achieves ≥99.9 % availability.

AdvertisingAlibabaFlink
0 likes · 18 min read
Design and Evolution of Alibaba Advertising Real-Time Data Warehouse
Radish, Keep Going!
Radish, Keep Going!
Mar 12, 2025 · Backend Development

Why gRPC Beats REST: Performance, Contracts, and Streaming Explained

gRPC offers superior performance, efficient Protobuf encoding, strong API contracts, seamless streaming, cross‑language support, and HTTP/2 advantages, making it a compelling alternative to REST for modern web services, while tools like gRPC‑Gateway, ConnectRPC, and Buf streamline migration and ecosystem integration.

API contractsStreaminggRPC
0 likes · 10 min read
Why gRPC Beats REST: Performance, Contracts, and Streaming Explained
Java Architect Essentials
Java Architect Essentials
Mar 7, 2025 · Artificial Intelligence

Introducing DeepSeek4j 1.4: A Java Spring Boot Integration for DeepSeek AI with Chain‑of‑Thought and Streaming Support

The article introduces DeepSeek4j 1.4, a Java Spring Boot library that overcomes existing framework limitations by preserving DeepSeek's chain‑of‑thought capabilities, adding full reactive streaming, and providing a simple one‑line API along with quick‑start instructions and code examples.

AI integrationChain-of-ThoughtDeepSeek
0 likes · 5 min read
Introducing DeepSeek4j 1.4: A Java Spring Boot Integration for DeepSeek AI with Chain‑of‑Thought and Streaming Support
Code Ape Tech Column
Code Ape Tech Column
Mar 7, 2025 · Backend Development

Using Spring's ResponseBodyEmitter for Real‑Time Streaming Responses

This article introduces Spring Framework's ResponseBodyEmitter, explains its advantages over SSE for asynchronous HTTP responses, demonstrates practical usage with a real‑time log‑streaming example, outlines core methods, working principles, best‑practice considerations, and compares it with traditional Streaming and Server‑Sent Events.

JavaResponseBodyEmitterSSE
0 likes · 10 min read
Using Spring's ResponseBodyEmitter for Real‑Time Streaming Responses
Alibaba Cloud Infrastructure
Alibaba Cloud Infrastructure
Mar 6, 2025 · Big Data

Leveraging Apache Iceberg and AutoMQ for Real-Time Data Lake Ingestion: Architecture, Best Practices, and Cost Optimization

This article examines how Apache Iceberg’s snapshot‑based ACID transactions, logical‑physical partition evolution, and COW/MOR update modes enable efficient real‑time data lake ingestion, and demonstrates AutoMQ’s Kafka‑to‑Iceberg Table Topic solution that simplifies schema management, reduces latency, and cuts operational costs.

Apache IcebergAutoMQCloud Native
0 likes · 14 min read
Leveraging Apache Iceberg and AutoMQ for Real-Time Data Lake Ingestion: Architecture, Best Practices, and Cost Optimization
Python Programming Learning Circle
Python Programming Learning Circle
Feb 28, 2025 · Fundamentals

Techniques for Efficient Large File Processing in Python

Processing large files efficiently in Python requires techniques such as line-by-line iteration, chunked reads, generators, buffered I/O, and streaming, which help avoid memory errors, improve speed, and optimize resources for tasks like log analysis, data scraping, and real-time API handling.

File I/OLarge FilesStreaming
0 likes · 5 min read
Techniques for Efficient Large File Processing in Python
Cognitive Technology Team
Cognitive Technology Team
Feb 28, 2025 · Artificial Intelligence

Design and High‑Availability Architecture of Alibaba LangEngine AI Application Framework

This article introduces Alibaba's LangEngine, a pure Java AI application framework, detailing its high‑availability gateway architecture, communication protocols, streaming and non‑streaming output, multi‑level metadata caching, asynchronous and serverless designs, and future open‑source roadmap, offering practical guidance for building robust AI services.

AI FrameworkAsynchronous DesignHigh Availability
0 likes · 11 min read
Design and High‑Availability Architecture of Alibaba LangEngine AI Application Framework
Top Architect
Top Architect
Feb 21, 2025 · Artificial Intelligence

DeepSeek4j 1.4: Java Integration Framework for DeepSeek with Full Chain‑of‑Thought and Streaming Support

The article introduces DeepSeek4j 1.4, a Java‑based framework that overcomes Spring AI’s limitations by fully preserving DeepSeek’s chain‑of‑thought and billing features, adding reactive streaming, providing Spring Boot starter integration, and offering quick‑start code samples and configuration guidance.

AIChain-of-ThoughtDeepSeek
0 likes · 8 min read
DeepSeek4j 1.4: Java Integration Framework for DeepSeek with Full Chain‑of‑Thought and Streaming Support
Java Architecture Diary
Java Architecture Diary
Feb 10, 2025 · Artificial Intelligence

deepseek4j 1.3: Java SDK adds web search, streaming & multi‑channel AI

deepseek4j 1.3 introduces web‑search capability, streaming responses, system prompts, expanded multi‑platform support, enhanced SSE debugging, and upcoming features like API‑key rotation and resilience, enabling Java developers to integrate DeepSeek models effortlessly while focusing on business logic.

AIDeepSeekSDK
0 likes · 8 min read
deepseek4j 1.3: Java SDK adds web search, streaming & multi‑channel AI
macrozheng
macrozheng
Jan 24, 2025 · Backend Development

Boost Java Excel Performance with FastExcel: Features, Usage, and Comparison

This article introduces FastExcel, an upgraded Java library for high‑performance Excel read/write, outlines its key features, provides step‑by‑step code examples for entity creation, event listeners, writing, reading, PDF conversion, compares it with EasyExcel, and concludes with its suitability for large‑scale data processing.

Data ProcessingExcelFastExcel
0 likes · 8 min read
Boost Java Excel Performance with FastExcel: Features, Usage, and Comparison
Code Ape Tech Column
Code Ape Tech Column
Jan 24, 2025 · Backend Development

FastExcel: High‑Performance Java Library for Excel Read/Write – Features, Usage, and Comparison with EasyExcel

FastExcel is a Java library that builds on EasyExcel to provide higher performance, low‑memory streaming, and additional features such as PDF conversion, offering simple APIs, full compatibility, and detailed code examples for creating entity classes, listeners, and read/write operations.

Data ProcessingExcelFastExcel
0 likes · 9 min read
FastExcel: High‑Performance Java Library for Excel Read/Write – Features, Usage, and Comparison with EasyExcel
Alibaba Cloud Big Data AI Platform
Alibaba Cloud Big Data AI Platform
Jan 14, 2025 · Big Data

How Fluss Unifies Lake and Stream for Real‑Time Analytics: Architecture, Benefits, and Future Roadmap

This article summarizes a talk by Alibaba Cloud senior engineer and Flink Committer Luo Yuxia on the challenges of separating lake and stream storage, introduces the Fluss lake‑stream unified architecture, explains its technical benefits such as second‑level data freshness, unified metadata, efficient changelog generation, and outlines future plans for broader ecosystem integration.

FlinkFlussLakehouse
0 likes · 13 min read
How Fluss Unifies Lake and Stream for Real‑Time Analytics: Architecture, Benefits, and Future Roadmap
Ctrip Technology
Ctrip Technology
Jan 3, 2025 · Big Data

Design and Implementation of a Kafka Gatekeeper for FinOps Billing Data Quality Governance

This article describes the challenges of data quality in Ctrip’s hybrid‑cloud FinOps billing system and presents the design, implementation, and high‑availability deployment of a custom Kafka Gatekeeper proxy that performs pre‑validation, configurable rules, self‑service dashboards, and automated alerts to improve coverage, timeliness, and responsibility attribution.

Cloud NativeFinOpsStreaming
0 likes · 17 min read
Design and Implementation of a Kafka Gatekeeper for FinOps Billing Data Quality Governance
Big Data Technology & Architecture
Big Data Technology & Architecture
Jan 2, 2025 · Big Data

Apache Paimon: Core Capabilities, Table Types, LSM Tree, Buckets, Merge Engines, and Operational Details

This article provides a comprehensive overview of Apache Paimon, covering its real‑time lake ingestion, unified stream‑batch processing, table types (primary‑key and append‑only), LSM‑tree storage, bucket mechanisms, merge‑engine options, compaction strategies, concurrency control, consumption methods, tag management, data cleanup, and system tables for big‑data workloads.

Apache PaimonFlinkLSM Tree
0 likes · 25 min read
Apache Paimon: Core Capabilities, Table Types, LSM Tree, Buckets, Merge Engines, and Operational Details
Zhihu Tech Column
Zhihu Tech Column
Dec 31, 2024 · Cloud Native

Cloud Native Innovation Forum: AutoMQ Table Topic, OceanBase Integrated Database, and Observability Practices

The article recaps Zhihu's Cloud Native Innovation Forum where experts from AutoMQ, OceanBase, and Flashcat shared practical solutions on streaming data ingestion, unified database architectures, and AI‑driven observability, highlighting real‑world deployments, performance optimizations, and cost‑saving strategies.

AIAutoMQCloud Native
0 likes · 10 min read
Cloud Native Innovation Forum: AutoMQ Table Topic, OceanBase Integrated Database, and Observability Practices
Architect
Architect
Dec 15, 2024 · Databases

Efficient MySQL Queries for Millions of Rows: Regular, Stream, and Cursor

When processing massive MySQL result sets, loading all rows into JVM memory can cause OOM and slow performance, so this guide compares three approaches—regular pagination, streaming queries using server-side cursors, and cursor‑based fetchSize control—detailing their implementations, MyBatis configurations, and trade‑offs.

CursorDatabase QueryLarge Data
0 likes · 10 min read
Efficient MySQL Queries for Millions of Rows: Regular, Stream, and Cursor
MaGe Linux Operations
MaGe Linux Operations
Dec 14, 2024 · Big Data

Master Kafka: From Core Concepts to Real-World Deployment

This comprehensive guide explains Kafka’s architecture, core APIs, topics and partitions, deployment steps, multi‑broker clustering, and practical use cases such as messaging, log aggregation, stream processing, and data import/export with Kafka Connect, providing a hands‑on tutorial for developers and engineers.

InstallationStreamingTutorial
0 likes · 30 min read
Master Kafka: From Core Concepts to Real-World Deployment
Tencent Advertising Technology
Tencent Advertising Technology
Dec 6, 2024 · Big Data

Building a High‑Performance Advertising Feature Data Lake with Apache Iceberg at Tencent

Tencent's advertising team replaced a traditional HDFS‑Hive warehouse with an Apache Iceberg‑based data lake, adding primary‑key tables, multi‑stream merging, adaptive compaction, and Spark SPJ optimizations to achieve minute‑level feature update latency, 10× back‑fill speed, and up to 60% storage savings.

CDCIcebergStreaming
0 likes · 25 min read
Building a High‑Performance Advertising Feature Data Lake with Apache Iceberg at Tencent
Bilibili Tech
Bilibili Tech
Nov 26, 2024 · Big Data

Bilibili’s Iceberg‑Based Streaming‑Batch Integration: Architecture, Optimizations, and Practices

Bilibili migrated its massive user‑behavior, commercial AI training, and database synchronization pipelines from Hive and Kafka to an Iceberg‑based streaming‑batch architecture, using Flink and the Magnus optimizer to achieve minute‑level freshness, reduce CPU and memory usage by about 20‑22 %, save roughly 3.55 M CNY annually, and dramatically improve query latency and join performance.

Data IntegrationFlinkIceberg
0 likes · 20 min read
Bilibili’s Iceberg‑Based Streaming‑Batch Integration: Architecture, Optimizations, and Practices
DataFunSummit
DataFunSummit
Nov 23, 2024 · Big Data

Bilibili's Iceberg‑Based Streaming‑Batch Integration: Architecture, Optimizations, and Practice

This article presents Bilibili's end‑to‑end exploration of a streaming‑batch unified data pipeline built on Apache Iceberg, detailing the original and iterated architectures for massive user behavior transmission, online AI training, DB synchronization, and dimension‑join, along with performance gains, cost savings, and future plans.

Batch ProcessingFlinkIceberg
0 likes · 20 min read
Bilibili's Iceberg‑Based Streaming‑Batch Integration: Architecture, Optimizations, and Practice
CSS Magic
CSS Magic
Nov 8, 2024 · Artificial Intelligence

LLM Application Development Tips (3): Exploring LLM API Inputs and Outputs

This article explains how to configure key OpenAI chat completion parameters—such as temperature, top_p, streaming, response format, and tool selection—and walks through the structure of the API's JSON response, highlighting fields like id, model, choices, finish_reason, and usage for better control and cost estimation.

AI agentsAPI parametersJSON response
0 likes · 8 min read
LLM Application Development Tips (3): Exploring LLM API Inputs and Outputs
360 Zhihui Cloud Developer
360 Zhihui Cloud Developer
Oct 31, 2024 · Backend Development

Boosting Ozone Block Reads with gRPC Streaming: Up to 30% Faster

This article explains how a gRPC bidirectional streaming read method was added to Ozone to reduce chunk‑by‑chunk request gaps, describes the client‑side implementation, presents single‑ and multi‑threaded performance tests showing roughly 30% faster reads, and outlines future enhancements such as pre‑fetching.

OzoneStreamingblock storage
0 likes · 7 min read
Boosting Ozone Block Reads with gRPC Streaming: Up to 30% Faster
DataFunSummit
DataFunSummit
Sep 30, 2024 · Big Data

Apache Hudi from Zero to One: The Swiss Army Knife for Data Ingestion – Hudi Streamer (Part 9)

This article introduces Apache Hudi Streamer, a versatile Spark‑based data ingestion tool likened to a Swiss Army knife, detailing its core options—including table configuration, continuous mode, source classes, transformers, table services, catalog synchronization, and advanced features—while guiding users on practical pipeline setup.

Apache HudiData IngestionSpark
0 likes · 10 min read
Apache Hudi from Zero to One: The Swiss Army Knife for Data Ingestion – Hudi Streamer (Part 9)
JD Retail Technology
JD Retail Technology
Sep 3, 2024 · Backend Development

Design and Architecture of a New Video Review System with Streamlined Frame Extraction and Parallel Processing

This article presents the design goals, architecture, technology selection, and component details of a unified video review system that leverages FFmpeg for frame extraction, stream‑based parallel processing, and flexible synchronous/asynchronous workflows to achieve low latency and high scalability.

FFmpegStreamingcontent moderation
0 likes · 10 min read
Design and Architecture of a New Video Review System with Streamlined Frame Extraction and Parallel Processing
Java Tech Enthusiast
Java Tech Enthusiast
Sep 2, 2024 · Industry Insights

Why Major Pirate Streaming Sites Are Closing: Industry Trends and Copyright Crackdowns

A wave of shutdowns affecting popular free video and anime piracy platforms such as RARBG and Animeflix reveals how pandemic costs, legal pressures, court rulings, and coordinated anti‑piracy actions by industry alliances are reshaping the digital media landscape and pushing users toward legitimate services.

Industry TrendsStreamingcopyright enforcement
0 likes · 7 min read
Why Major Pirate Streaming Sites Are Closing: Industry Trends and Copyright Crackdowns
Big Data Technology & Architecture
Big Data Technology & Architecture
Aug 20, 2024 · Big Data

Practical Insights on Using Apache Paimon for Real-World Data Lake Scenarios

This article shares a personal, experience‑driven overview of Apache Paimon, highlighting its design simplicity, key capabilities such as schema evolution, stream‑batch unified processing, primary‑key support, and closed‑loop data handling, while discussing when its features are appropriate for production environments.

Apache PaimonBatch ProcessingSchema Evolution
0 likes · 5 min read
Practical Insights on Using Apache Paimon for Real-World Data Lake Scenarios
DeWu Technology
DeWu Technology
Jul 31, 2024 · Big Data

Custom Flink Scheduler Enhancements: Resource Balancing, Task Migration, and TmRestart Strategy

The article details Dewu’s custom Flink scheduler, DwScheduler, which adds JSON‑based resource specifications, per‑TaskManager slot sharing for balanced CPU use, hot TaskManager migration callbacks, and a new TmRestart strategy for rapid pod‑process recovery, offering practical techniques to enhance real‑time stream processing stability and performance.

Apache FlinkStreamingTask Migration
0 likes · 9 min read
Custom Flink Scheduler Enhancements: Resource Balancing, Task Migration, and TmRestart Strategy
Soul Technical Team
Soul Technical Team
Jul 23, 2024 · Big Data

Kafka Stability Challenges and Governance Framework at Soul

This article analyzes the role, application scenarios, stability challenges, and comprehensive governance framework of Apache Kafka at Soul, covering deployment, configuration, monitoring, standard controls, common misuse, and future directions toward cloud‑native solutions.

Streamingkafkamonitoring
0 likes · 30 min read
Kafka Stability Challenges and Governance Framework at Soul
Rare Earth Juejin Tech Community
Rare Earth Juejin Tech Community
Jul 22, 2024 · Big Data

Comprehensive Guide to Kafka: Architecture, Core Concepts, and Configuration

This article provides an in‑depth overview of Apache Kafka, covering its use cases, comparison with other message queues, versioning, performance mechanisms, core concepts such as topics, partitions, offsets, consumer groups, rebalancing, replication, leader election, idempotence, transactions, compression, interceptors, request handling, and practical configuration tips for reliable streaming applications.

Message QueueProducerStreaming
0 likes · 25 min read
Comprehensive Guide to Kafka: Architecture, Core Concepts, and Configuration
Tencent Cloud Developer
Tencent Cloud Developer
Jul 16, 2024 · Big Data

In‑Depth Exploration of Apache Kafka: Architecture, High Reliability, and High Performance

Apache Kafka achieves high‑throughput, fault‑tolerant messaging by combining a partitioned log architecture with leader‑follower replication, asynchronous producer pipelines, configurable acknowledgments, page‑cache‑based sequential writes, zero‑copy transfers, batching, compression, and a multi‑reactor network model that together ensure scalability, reliability, and performance.

Apache KafkaReliabilityStreaming
0 likes · 30 min read
In‑Depth Exploration of Apache Kafka: Architecture, High Reliability, and High Performance
Tencent Cloud Developer
Tencent Cloud Developer
Jul 2, 2024 · Big Data

Apache Flink Deployment with Pulsar Connector: Setup, Demos, and Best Practices

This guide shows how to deploy Apache Flink 1.17 in Docker, configure off‑heap memory, connect it to Pulsar via the 4.1.0‑1.17 connector, run example jobs that copy topics and perform windowed word‑count, and provides Maven dependencies, custom serialization tips, batching settings, and version‑specific best‑practice notes.

Apache FlinkDataStreamDocker deployment
0 likes · 20 min read
Apache Flink Deployment with Pulsar Connector: Setup, Demos, and Best Practices
Code Mala Tang
Code Mala Tang
Jun 29, 2024 · Frontend Development

Master WritableStream: Real-World Uses, Best Practices, and Common Pitfalls

This article introduces the JavaScript WritableStream API, explains its core methods and construction, demonstrates practical scenarios such as file uploads, logging, data transformation, and media handling, and discusses advanced considerations like chunk sizing, error recovery, concurrency control, and performance optimization.

StreamingWeb APIWritableStream
0 likes · 10 min read
Master WritableStream: Real-World Uses, Best Practices, and Common Pitfalls
Code Mala Tang
Code Mala Tang
Jun 27, 2024 · Frontend Development

Mastering ReadableStream: A Deep Dive into Web Streams API

This article introduces the concept of streams, explains the Web Streams API and its ReadableStream component, details constructors, methods, queuing strategies, back‑pressure handling, BYOB and byte streams, and provides practical code examples and usage scenarios for modern web development.

Front-endReadableStreamStreaming
0 likes · 20 min read
Mastering ReadableStream: A Deep Dive into Web Streams API
DataFunTalk
DataFunTalk
Jun 18, 2024 · Big Data

Real-time Data Warehouse Evolution with Data Lake: Architecture, Challenges, and Solutions

This article presents a comprehensive overview of the evolution from traditional Lambda‑based real‑time data warehouse solutions to a data‑lake‑integrated architecture, detailing the shortcomings of legacy designs, the iterative improvements made at JD Technology, and the technical and operational challenges encountered during implementation.

Lambda ArchitectureReal-time Data WarehouseStreaming
0 likes · 24 min read
Real-time Data Warehouse Evolution with Data Lake: Architecture, Challenges, and Solutions
Big Data Technology & Architecture
Big Data Technology & Architecture
Jun 16, 2024 · Big Data

Real-time Big Data Analytics with Apache Paimon and the Streaming Lakehouse Architecture

This article summarizes Wang Feng's presentation on the next‑generation Lakehouse architecture, explaining how Apache Paimon provides a unified, real‑time data lake format that bridges batch and streaming workloads, enabling low‑latency analytics and AI integration for modern big‑data applications.

Apache PaimonReal-Time AnalyticsStreaming
0 likes · 9 min read
Real-time Big Data Analytics with Apache Paimon and the Streaming Lakehouse Architecture
Sohu Tech Products
Sohu Tech Products
Jun 5, 2024 · Big Data

Why Kafka Is the Backbone of Modern Data Pipelines: Core Architecture and Use Cases

This article explains Kafka's role as a high‑throughput distributed message queue, detailing its core components, topic‑partition model, consumer groups, storage mechanisms, fault‑tolerance features, delivery guarantees, ZooKeeper coordination, and scalability strategies for building reliable real‑time data pipelines.

Message QueueStreamingbig data
0 likes · 14 min read
Why Kafka Is the Backbone of Modern Data Pipelines: Core Architecture and Use Cases
Su San Talks Tech
Su San Talks Tech
Jun 2, 2024 · Big Data

Mastering Kafka: Core Architecture, Use Cases, and Design Principles

This article provides a comprehensive overview of Apache Kafka, covering its role as a message queue, core components, topic and partition design, consumer groups, storage mechanisms, high‑availability features, delivery guarantees, ZooKeeper coordination, and scalability strategies for building robust real‑time data pipelines.

Streamingbig datakafka
0 likes · 15 min read
Mastering Kafka: Core Architecture, Use Cases, and Design Principles
DataFunTalk
DataFunTalk
May 16, 2024 · Big Data

Streaming Data Lake Warehouse Solution Based on USDP with Flink and Paimon

This article presents UCloud's USDP‑based streaming data lake warehouse solution that leverages Flink for real‑time processing and Paimon for lake storage, detailing its architecture, advantages, practical scenarios, and providing complete SQL and Flink CDC code snippets for end‑to‑end implementation.

CDCFlinkPaimon
0 likes · 27 min read
Streaming Data Lake Warehouse Solution Based on USDP with Flink and Paimon
Sohu Tech Products
Sohu Tech Products
May 15, 2024 · Artificial Intelligence

OpenAI Assistants API Quickstart Project for Next.js

OpenAI’s open‑source openai‑assistants‑quickstart project shows how to integrate the Assistants API into a Next.js app, offering streaming chat, code‑interpreter, file‑search, and function‑calling tools, and provides step‑by‑step setup instructions so developers can quickly build and customize AI assistants.

AI assistantAssistants APICode interpreter
0 likes · 4 min read
OpenAI Assistants API Quickstart Project for Next.js
Mike Chen's Internet Architecture
Mike Chen's Internet Architecture
May 11, 2024 · Big Data

Comprehensive Introduction to Apache Kafka: Architecture, Features, and Use Cases

This article provides a detailed overview of Apache Kafka, covering its core characteristics, distributed architecture, key components such as topics, partitions, brokers, producers, consumers, ZooKeeper, and common application scenarios like log collection, event‑driven architecture, real‑time analytics, and monitoring.

Message QueueStreamingarchitecture
0 likes · 7 min read
Comprehensive Introduction to Apache Kafka: Architecture, Features, and Use Cases
Big Data Technology & Architecture
Big Data Technology & Architecture
Apr 30, 2024 · Big Data

Apache Paimon Becomes a Top-Level Project: A Comprehensive Overview of Lakehouse Framework Capabilities and Future Trends

The article reviews Apache Paimon's graduation to an Apache Top-Level Project, outlines the essential capabilities of modern lakehouse frameworks—including streaming and batch I/O, multi‑engine integration, and advanced features—and discusses the problems they solve and the promising direction of the lakehouse ecosystem.

Apache PaimonBatch ProcessingReal-time Data Lake
0 likes · 5 min read
Apache Paimon Becomes a Top-Level Project: A Comprehensive Overview of Lakehouse Framework Capabilities and Future Trends
Bilibili Tech
Bilibili Tech
Apr 26, 2024 · Artificial Intelligence

2024 Bilibili Technology Patent Awards – Highlights of Ten Winning Innovations

On World Intellectual Property Day, Bilibili honored ten breakthrough patents that together enable billion‑scale video duplicate detection, AI‑driven story generation, synchronized live rhythm‑games, automatic OTT casting, knowledge‑graph‑based content moderation, glitch‑free multi‑audio streaming, modular playback integration, neural‑network resolution encoding, AV1 reference‑frame pruning, and fine‑grained GPU isolation.

Artificial IntelligenceStreamingencoding
0 likes · 6 min read
2024 Bilibili Technology Patent Awards – Highlights of Ten Winning Innovations
21CTO
21CTO
Apr 22, 2024 · Big Data

Inside Uber’s Real‑Time Data Infrastructure: How They Scale Streaming at Massive Scale

This article explores Uber’s sophisticated real‑time data infrastructure, detailing how the company leverages open‑source technologies such as Apache Kafka, Flink, Pinot, and Presto, and describing the architectural components, scaling challenges, multi‑region resilience, data back‑filling, and operational practices that enable low‑latency analytics for millions of daily rides and deliveries.

FlinkPinotStreaming
0 likes · 25 min read
Inside Uber’s Real‑Time Data Infrastructure: How They Scale Streaming at Massive Scale
Bilibili Tech
Bilibili Tech
Apr 12, 2024 · Backend Development

Design and Optimization of a High‑Throughput Long‑Connection Service for Live Streaming

The article details a Golang‑based high‑throughput long‑connection service for live‑streaming, describing its five‑layer architecture, multi‑protocol support, load‑balancing, message‑queue decoupling, aggregation with brotli compression, multi‑region deployment, priority channels, and future enhancements for observability and intelligent endpoint selection.

Backend ArchitectureHigh ThroughputStreaming
0 likes · 16 min read
Design and Optimization of a High‑Throughput Long‑Connection Service for Live Streaming
Bilibili Tech
Bilibili Tech
Apr 9, 2024 · Big Data

Optimizing Flink State Performance with RocksDB KV Separation and BlobDB

In large‑scale Flink double‑stream joins, terabyte‑sized RocksDB state caused severe compaction latency and CPU spikes, but enabling RocksDB BlobDB KV‑separation (and an inner‑compaction patch) dramatically shrank SST files, reduced read/write latencies to sub‑millisecond levels, and cut CPU spikes by about half.

FlinkKV SeparationRocksDB
0 likes · 12 min read
Optimizing Flink State Performance with RocksDB KV Separation and BlobDB
DataFunSummit
DataFunSummit
Apr 7, 2024 · Big Data

Li Auto’s Flink on Kubernetes Data Integration Practice

This article presents Li Auto’s end‑to‑end data integration journey, detailing the evolution of its data platform, the challenges of heterogeneous sources, and how a unified Flink‑on‑K8s solution with cloud‑native architecture, operator management, monitoring, and checkpointing addresses batch‑stream convergence and future scalability.

Batch ProcessingData IntegrationFlink
0 likes · 12 min read
Li Auto’s Flink on Kubernetes Data Integration Practice
Ctrip Technology
Ctrip Technology
Mar 22, 2024 · Mobile Development

Design and Implementation of the Cloud Touch Platform for Remote Mobile Device Control and Testing

The article presents the background, full‑scenario construction, core architecture, device‑pool strategy, remote iOS control via WebDriverAgent, screen‑sync using ffmpeg, streaming pipeline, data collection, and practical lessons of the Cloud Touch platform that enables unified remote testing and customer‑support workflows for mobile applications.

Cloud TouchFFmpegMobile Testing
0 likes · 14 min read
Design and Implementation of the Cloud Touch Platform for Remote Mobile Device Control and Testing
Didi Tech
Didi Tech
Mar 12, 2024 · Big Data

Understanding Flink Metrics System: Core Concepts, Elastic Design, and Practical Usage

The article explains Flink’s metrics architecture—core concepts, reporter interfaces, built‑in and custom metric types, elastic plugin design, and scheduled reporting—illustrated with a consumption‑latency example, and shows how Didi uses these metrics for real‑time UI curves, alerts, and intelligent task diagnosis.

FlinkStreamingbig data
0 likes · 11 min read
Understanding Flink Metrics System: Core Concepts, Elastic Design, and Practical Usage
Architect's Guide
Architect's Guide
Mar 2, 2024 · Fundamentals

RabbitMQ vs Kafka: Core Differences and When to Use Each

This article compares RabbitMQ and Apache Kafka across architecture, message ordering, routing, timing, retention, fault handling, scalability, and consumer complexity, and provides guidance on which platform suits specific use‑cases such as flexible routing, strict ordering, long‑term retention, or high throughput.

Message OrderingMessage QueueStreaming
0 likes · 19 min read
RabbitMQ vs Kafka: Core Differences and When to Use Each
Airbnb Technology Team
Airbnb Technology Team
Mar 1, 2024 · Big Data

Riverbed: A Scalable Data Framework for Real‑time and Batch Processing at Airbnb

Airbnb’s Riverbed framework unifies streaming CDC events and batch Spark jobs behind a GraphQL‑based declarative API to automatically build and maintain distributed materialized views, using Kafka‑partitioned ordering and version control to deliver billions of daily updates with low‑latency reads for features such as payments and search.

AirbnbApache SparkData Engineering
0 likes · 8 min read
Riverbed: A Scalable Data Framework for Real‑time and Batch Processing at Airbnb
MaGe Linux Operations
MaGe Linux Operations
Feb 20, 2024 · Big Data

Redis Streams vs Kafka: Which Is Better for Real‑Time Event Processing?

This article compares Redis Streams and Kafka, examining their architectures, ordering guarantees, consumer group models, scalability, and trade‑offs, and shows how Redis can emulate Kafka‑like semantics using the Runnel library, while highlighting memory‑speed benefits versus Kafka’s durable, unlimited log storage.

Event ProcessingRunnelStreaming
0 likes · 9 min read
Redis Streams vs Kafka: Which Is Better for Real‑Time Event Processing?
Open Source Tech Hub
Open Source Tech Hub
Jan 31, 2024 · Artificial Intelligence

How to Build Async OpenAI PHP Clients with Workerman & Webman

This guide shows how to install the OpenAI PHP async client and implement streaming and non‑streaming chat, image generation, audio speech, and embedding features using Workerman and Webman, including Azure OpenAI support, with complete code examples.

APIOpenAIPHP
0 likes · 6 min read
How to Build Async OpenAI PHP Clients with Workerman & Webman
StarRocks
StarRocks
Jan 30, 2024 · Big Data

How InLong Guarantees Exactly‑Once Real‑Time Writes to StarRocks

This article explains how Apache InLong provides automatic, secure, high‑performance real‑time data transfer to StarRocks, detailing the transactional Stream Load API, the two‑phase commit process, Flink‑based ingestion architecture, exactly‑once guarantees, and performance test results across different parallelism levels.

InLongStarRocksStreaming
0 likes · 11 min read
How InLong Guarantees Exactly‑Once Real‑Time Writes to StarRocks
MaGe Linux Operations
MaGe Linux Operations
Jan 21, 2024 · Big Data

Master Kafka: Core Concepts, Metrics, and Troubleshooting Guide

This article explains Kafka's fundamental components, version evolution, key monitoring metrics for producers, brokers, consumers and Zookeeper, and provides step‑by‑step troubleshooting methods for common issues such as slow topic throughput and message backlog.

Message QueueStreamingbig data
0 likes · 8 min read
Master Kafka: Core Concepts, Metrics, and Troubleshooting Guide
Rare Earth Juejin Tech Community
Rare Earth Juejin Tech Community
Jan 13, 2024 · Big Data

What Is Kafka? Overview, Architecture, Features, Deployment, and Sample Code

Kafka, an Apache‑developed distributed publish/subscribe messaging system, provides reliable, high‑throughput real‑time data streaming with producers, consumers, brokers, streams, and connectors, and the article explains its core concepts, architecture, advantages, deployment methods, use cases, and includes Java code examples for producers and consumers.

JavaMessage QueueStreaming
0 likes · 8 min read
What Is Kafka? Overview, Architecture, Features, Deployment, and Sample Code
FunTester
FunTester
Jan 5, 2024 · Big Data

An Overview of Apache Kafka and Kafka Streams Technical Features

This article introduces Apache Kafka as a high‑throughput, scalable, fault‑tolerant distributed streaming platform, explains why it is chosen for real‑time data pipelines, and details key Kafka Streams concepts such as stream processing, interactive queries, stateful processing, windowing, serialization, and testing.

Apache KafkaStreamingbig data
0 likes · 13 min read
An Overview of Apache Kafka and Kafka Streams Technical Features
Sohu Tech Products
Sohu Tech Products
Dec 27, 2023 · Big Data

Practical Implementation of Data Integration with Flink on Kubernetes at Li Auto

Li Auto built a cloud‑native data‑integration platform by deploying Flink on Kubernetes, unifying batch and streaming workloads with a storage layer (JuiceFS + BOS) and Flink Operator, enabling simple source‑sink pipelines, elastic scaling, automated checkpointing, and centralized monitoring while addressing earlier fragmentation and resource inefficiencies.

Cloud NativeData IntegrationFlink
0 likes · 11 min read
Practical Implementation of Data Integration with Flink on Kubernetes at Li Auto
ITPUB
ITPUB
Dec 24, 2023 · Backend Development

Why Kafka Is the Backbone of Modern Messaging, Streaming, and Data Pipelines

This article explains how Kafka serves as a high‑throughput, durable messaging system, a reliable storage layer, a log‑aggregation hub, a stream‑processing engine, and a core component for CDC, system migration, monitoring, and event‑sourcing architectures.

CDCStreamingevent sourcing
0 likes · 9 min read
Why Kafka Is the Backbone of Modern Messaging, Streaming, and Data Pipelines
DataFunTalk
DataFunTalk
Dec 15, 2023 · Big Data

Flink Forward Asia 2023: New Flink Releases, Apache Paimon, and Flink CDC 3.0

The Flink Forward Asia 2023 conference showcased major updates to Apache Flink (versions 1.17 and 1.18), introduced the Apache Paimon lakehouse project, announced Flink CDC 3.0, and highlighted community growth, cloud‑native deployments, and real‑time data‑warehouse use cases across industry leaders.

Apache FlinkApache PaimonFlink CDC
0 likes · 17 min read
Flink Forward Asia 2023: New Flink Releases, Apache Paimon, and Flink CDC 3.0
ITPUB
ITPUB
Dec 14, 2023 · Big Data

How to Build a Python‑Hadoop Word Count on a Single‑Node Cluster

This step‑by‑step guide shows how to install and configure a single‑node Hadoop 3.2.0 environment on CentOS 7, set up Python 3.7, write MapReduce mapper and reducer scripts in Python, and run a word‑count job using Hadoop streaming, illustrating core Hadoop concepts and their relevance today.

HadoopMapReducePython
0 likes · 21 min read
How to Build a Python‑Hadoop Word Count on a Single‑Node Cluster
DataFunTalk
DataFunTalk
Dec 8, 2023 · Big Data

Zhihu Bridge Platform: Architecture, Capabilities, and Future Trends of Content Operations

This article presents a comprehensive overview of Zhihu's Bridge platform, detailing its content‑operation architecture—including content pool, management, analysis, monitoring, and intervention modules—explaining the underlying streaming and batch technologies such as Flink, Doris, and Elasticsearch, and outlining future automation and AI‑driven workflow directions.

AIStreamingarchitecture
0 likes · 17 min read
Zhihu Bridge Platform: Architecture, Capabilities, and Future Trends of Content Operations
ITPUB
ITPUB
Dec 2, 2023 · Backend Development

Why Did My Flink Kafka Job Lose Data? Uncovering Misconfigured Bootstrap Servers

A Flink job that reads from Kafka and writes to Elasticsearch was losing data because the bootstrap.servers list mixed production and pre‑release clusters, causing random server selection, partition discovery failures, and offset mismatches, which were resolved by correcting the server configuration.

Bootstrap ServersData lossFlink
0 likes · 8 min read
Why Did My Flink Kafka Job Lose Data? Uncovering Misconfigured Bootstrap Servers
JavaEdge
JavaEdge
Nov 24, 2023 · Backend Development

Why Kafka Is the Ultimate Backbone for Modern Backend Systems

This article explores how Kafka serves as a versatile backbone for messaging, durable storage, log aggregation, monitoring, commit logs, recommendation pipelines, stream processing, CDC, system migration, and event sourcing, highlighting its performance, reliability, and practical deployment patterns.

Data PipelineMessage QueueStreaming
0 likes · 10 min read
Why Kafka Is the Ultimate Backbone for Modern Backend Systems
Alibaba Cloud Big Data AI Platform
Alibaba Cloud Big Data AI Platform
Nov 23, 2023 · Big Data

Why Apache Paimon Is Revolutionizing Streaming Lakehouse Architecture with Flink

The article traces the shift from traditional Hive‑based warehouses to modern lakehouse architectures, explains the advantages of lake formats, introduces Apache Paimon as a streaming‑first data lake integrated with Flink, presents performance benchmarks showing its superiority over Hudi, and demonstrates a real‑time streaming lakehouse workflow.

Apache PaimonFlinkLakehouse
0 likes · 15 min read
Why Apache Paimon Is Revolutionizing Streaming Lakehouse Architecture with Flink
Alibaba Cloud Big Data AI Platform
Alibaba Cloud Big Data AI Platform
Nov 22, 2023 · Big Data

Real-Time Data Integration with Flink CDC: Core Tech and Alibaba Cloud Solutions

This article, based on a presentation by Flink CDC and Apache Flink community leaders, explores CDC real‑time integration challenges, delves into Flink CDC’s core technologies such as incremental snapshot and lock‑free processing, and demonstrates Alibaba Cloud’s enterprise‑grade solutions for end‑to‑end real‑time data pipelines.

Alibaba CloudChange Data CaptureFlink CDC
0 likes · 21 min read
Real-Time Data Integration with Flink CDC: Core Tech and Alibaba Cloud Solutions
Big Data Technology Architecture
Big Data Technology Architecture
Nov 14, 2023 · Big Data

Open Source Big Data Platform 3.0: Streaming Lakehouse, Serverless Architecture, and AI Integration

The talk outlines the evolution of Alibaba Cloud's open‑source big data platform from Hadoop‑based EMR to a 3.0 architecture featuring a streaming lakehouse, full serverless compute and storage, AI‑driven operations, and upcoming vector search services, highlighting technical motivations, challenges, and product releases.

LakehouseServerlessStreaming
0 likes · 14 min read
Open Source Big Data Platform 3.0: Streaming Lakehouse, Serverless Architecture, and AI Integration
DataFunSummit
DataFunSummit
Nov 9, 2023 · Big Data

Spark 3.4 New Features Overview: Community Updates, SQL Enhancements, PySpark, Streaming, and AI Ecosystem

This article presents a comprehensive overview of Spark 3.4, covering community growth statistics, major SQL improvements such as default column values and timestamp handling, new PySpark and streaming capabilities, and the emerging AI ecosystem that integrates natural‑language interfaces and Spark AI services.

DatabricksPySparkStreaming
0 likes · 14 min read
Spark 3.4 New Features Overview: Community Updates, SQL Enhancements, PySpark, Streaming, and AI Ecosystem
macrozheng
macrozheng
Nov 9, 2023 · Big Data

7 Real-World Kafka Use Cases Every Engineer Should Know

This article explains Kafka's core components and features, then details seven practical scenarios—including log processing, recommendation streams, monitoring, CDC, system migration, event sourcing, and message queuing—showing how Kafka powers modern distributed systems.

Message QueueStreamingbig data
0 likes · 12 min read
7 Real-World Kafka Use Cases Every Engineer Should Know
ITPUB
ITPUB
Nov 7, 2023 · Big Data

7 Real-World Kafka Use Cases That Power Modern Distributed Systems

This article introduces Apache Kafka’s core components and key features, then details seven practical use cases—including log processing, recommendation streams, monitoring, CDC, system migration, event sourcing, and message queuing—illustrated with diagrams and step‑by‑step workflows for distributed systems.

Message QueueStreamingbig data
0 likes · 10 min read
7 Real-World Kafka Use Cases That Power Modern Distributed Systems