Tagged articles

high throughput

77 articles · Page 1 of 1

Programmer1970

Aug 16, 2026 · Backend Development

Why Disruptor’s Ring Buffer Beats BlockingQueue: Zero‑Copy, False‑Sharing Elimination, and Cache‑Line Padding

The article dissects how Disruptor’s ring‑buffer architecture outperforms traditional BlockingQueue by eliminating lock contention, dynamic allocation, and cache‑line interference through pre‑allocated zero‑copy buffers, CAS‑based sequencing, cache‑line padding, batch processing, and tailored wait strategies, achieving millions of TPS.

BlockingQueueCache line paddingDisruptor

0 likes · 13 min read

Why Disruptor’s Ring Buffer Beats BlockingQueue: Zero‑Copy, False‑Sharing Elimination, and Cache‑Line Padding

Cloud Architecture

Aug 7, 2026 · Backend Development

Building a Trillion‑Message Queue: Kafka‑Level Architecture and Implementation

This article explains why and how to build a Kafka‑grade message‑queue kernel from scratch, detailing functional and non‑functional goals, core design principles, storage layout, replication, consumer‑group coordination, performance optimizations, deployment on Kubernetes, and a step‑by‑step roadmap to production‑grade reliability.

Distributed LogJavaKafka

0 likes · 40 min read

Building a Trillion‑Message Queue: Kafka‑Level Architecture and Implementation

21CTO

Aug 4, 2026 · Backend Development

How Zalando Achieved 1M RPS with an In‑Process Client‑Side Load Balancer

Zalando’s engineering team redesigned its high‑throughput product‑read API by moving 100‑fold internal fan‑out routing into an in‑process client‑side load balancer, cutting tail latency, reducing infrastructure costs by over 75%, and improving observability while keeping the external Skipper edge router unchanged.

KubernetesZalandoclient-side load balancer

0 likes · 7 min read

How Zalando Achieved 1M RPS with an In‑Process Client‑Side Load Balancer

Random Bulletin

Jul 17, 2026 · Backend Development

Ten‑Million QPS Architecture: From Strong to Eventual Consistency with Layered Design

The article examines why ultra‑high‑throughput systems must move from costly strong consistency to eventual consistency, explaining consistency layering, the trade‑offs of coordination latency, availability, and tail latency, and how local message tables, idempotent delivery, compensation (Saga) and periodic reconciliation together ensure data eventually aligns without sacrificing performance.

SAGAcompensationconsistency

0 likes · 17 min read

Ten‑Million QPS Architecture: From Strong to Eventual Consistency with Layered Design

Random Bulletin

Jun 30, 2026 · Operations

Preventing Message Queue Backlog at Ten‑Million QPS: From Reactive to Proactive

The article explains how a ten‑million‑QPS messaging system can shift from reactive fire‑fighting to proactive backlog prevention by using long‑, mid‑, and short‑term defense layers, capacity planning, predictive scaling, backpressure, and regular chaos engineering to eliminate user‑visible latency.

BackpressureCapacity Planningbacklog prevention

0 likes · 19 min read

Preventing Message Queue Backlog at Ten‑Million QPS: From Reactive to Proactive

Coder Trainee

Jun 28, 2026 · Fundamentals

Java Performance Tuning Part 5: Hands‑On GC Optimization from G1 to ZGC

This article walks through Java GC tuning by defining low‑latency and high‑throughput goals, comparing major collectors, presenting G1 and ZGC configuration examples, and demonstrating a real‑world payment system case where pause times were reduced from 150‑200 ms to under 50 ms.

G1GCJVM

0 likes · 8 min read

Java Performance Tuning Part 5: Hands‑On GC Optimization from G1 to ZGC

Random Bulletin

Jun 17, 2026 · Backend Development

Rebalance Optimization for Million‑QPS Kafka Clusters: From Default to Custom Assignors

At a 5,000‑instance, 20,000‑partition Kafka consumer cluster, the default eager Rebalance caused a 90‑second stop‑the‑world pause, illustrating how a single GC‑induced heartbeat miss can flood brokers with tens of millions of messages; the article dissects eager vs cooperative protocols, built‑in assignor limits, and four custom assignor patterns to scale Rebalance from 100 K to tens of millions of QPS.

AssignorCooperative ProtocolCustom Scheduler

0 likes · 28 min read

Rebalance Optimization for Million‑QPS Kafka Clusters: From Default to Custom Assignors

Random Bulletin

Jun 15, 2026 · Backend Development

From Auto to Manual: Mastering Consumer Offsets for Ten‑Million QPS Systems

The article examines a real payment loss incident caused by Kafka's default automatic offset commit, explains why automatic commits become a hidden trap at high traffic, and provides a step‑by‑step guide to switching to manual commits with async‑first, sync‑fallback, idempotency, dead‑letter handling, batch strategies, and concurrency controls for reliable ten‑million QPS consumption.

Kafkadead-letter queuehigh throughput

0 likes · 15 min read

From Auto to Manual: Mastering Consumer Offsets for Ten‑Million QPS Systems

Random Bulletin

Jun 14, 2026 · Backend Development

Scaling to Millions of QPS: How Batch Consumption Beats Single-Message Processing

The article explains why single‑message consumption stalls under high QPS due to fixed per‑message overhead, and how merging pull, processing, and commit into batch operations dramatically boosts throughput while introducing trade‑offs in latency, memory, and failure handling, with practical guidelines for batch size selection and dynamic tuning.

KafkaPerformance Optimizationbatch consumption

0 likes · 16 min read

Scaling to Millions of QPS: How Batch Consumption Beats Single-Message Processing

Random Bulletin

Jun 13, 2026 · Backend Development

Scaling Consumer Parallelism: From Single‑Thread to Multi‑Thread for Ten‑Million QPS

The article analyzes why single‑threaded message consumption hits processing, I/O/CPU mismatch, and fault‑tolerance limits, then walks through a step‑by‑step evolution—pull/worker split, key‑based routing, sliding‑window offset commits, backpressure, and multi‑layer parallelism—to achieve stable ten‑million‑QPS throughput.

Backpressureconsumer parallelismhigh throughput

0 likes · 22 min read

Scaling Consumer Parallelism: From Single‑Thread to Multi‑Thread for Ten‑Million QPS

Random Bulletin

Jun 9, 2026 · Backend Development

From Synchronous to Asynchronous Sending: Achieving 10 Million QPS with Kafka Producer

The article analyses how synchronous Kafka producer calls block business threads on ACK, limiting throughput, and shows that moving to asynchronous sending with the Accumulator decouples submission from confirmation, unlocking batch, compression, and back‑pressure mechanisms that enable stable operation at tens of millions of QPS.

BackpressureKafkaasynchronous

0 likes · 21 min read

From Synchronous to Asynchronous Sending: Achieving 10 Million QPS with Kafka Producer

Cloud Architecture

Jun 7, 2026 · Cloud Native

Deep Dive into Kubernetes Scheduler: From Scheduling Cycle to Managing Ten‑Thousands of Pods

This guide explains why the kube‑scheduler becomes a bottleneck in large clusters, walks through its architecture, scheduling cycle, queue mechanics, constraint handling, plugin framework, multi‑scheduler designs, production‑grade tuning, observability, and real‑world case studies for achieving stable, high‑throughput pod placement.

Production Tuninghigh throughputkube-scheduler

0 likes · 37 min read

Deep Dive into Kubernetes Scheduler: From Scheduling Cycle to Managing Ten‑Thousands of Pods

Random Bulletin

Jun 5, 2026 · Backend Development

Designing Kafka Topics at Ten‑Million QPS: From Ad‑hoc to Standardized Governance

At ten‑million QPS scale, a Kafka topic becomes a cross‑team contract rather than a simple name, requiring standardized naming, granularity, partitioning, schema registry, lifecycle management, and governance platforms to transform ad‑hoc topics into machine‑verifiable infrastructure assets.

KafkaMessage StreamingSchema Registry

0 likes · 24 min read

Designing Kafka Topics at Ten‑Million QPS: From Ad‑hoc to Standardized Governance

Random Bulletin

Jun 2, 2026 · Backend Development

Queue Selection at Ten‑Million QPS: From General‑Purpose to Specialized Queues

In systems handling ten‑million QPS, generic message queues like Kafka hit scalability and latency limits, so the article breaks down a three‑question framework—loss tolerance, latency sensitivity, and ordering needs—to match five specialized queue types and guide a transition from a single‑queue approach to a queue matrix.

KafkaRocketMQdistributed systems

0 likes · 18 min read

Queue Selection at Ten‑Million QPS: From General‑Purpose to Specialized Queues

Su San Talks Tech

May 19, 2026 · Interview Experience

Designing a Hundred‑Billion‑Scale Message Queue: A ByteDance Interview Walkthrough

This article walks through the interview question of designing a message queue that handles billions of messages daily and peaks at millions of QPS, covering traffic calculations, core roles, storage and throughput techniques, scalability, high availability, observability, framework comparisons, a real‑world case study, and key follow‑up interview topics.

KafkaPulsarRocketMQ

0 likes · 12 min read

Designing a Hundred‑Billion‑Scale Message Queue: A ByteDance Interview Walkthrough

Machine Heart

May 7, 2026 · Artificial Intelligence

Photo‑Level Simulation Bridges Vision Gap for Robot Learning (GS‑Playground, RSS 2026)

GS‑Playground is a next‑generation visual‑high‑fidelity robot simulator that cuts photo‑level rendering cost, automates asset creation, and narrows the Sim2Real gap, achieving up to 10,000 FPS on RTX 4090 and outperforming MuJoCo by 32× while supporting full‑stack parallel physics, 3DGS rendering, and end‑to‑end Real2Sim pipelines.

3D Gaussian SplattingSimulationhigh throughput

0 likes · 10 min read

Photo‑Level Simulation Bridges Vision Gap for Robot Learning (GS‑Playground, RSS 2026)

LuTiao Programming

Mar 23, 2026 · Backend Development

Stop Using Cron: From Single-Node to Distributed Scheduling to Reach >1k Tasks/s

The article explains why traditional Cron‑based scheduling fails under high load, walks through three architectural stages—from database polling to Redis delay queues and finally a full distributed scheduler—showing code examples, design trade‑offs, and performance numbers that enable thousands of tasks per second.

CronJavaRedis

0 likes · 8 min read

Stop Using Cron: From Single-Node to Distributed Scheduling to Reach >1k Tasks/s

Architect Chen

Mar 20, 2026 · Big Data

How Kafka Achieves Million‑Message Throughput: Partitions, Zero‑Copy, and Page Cache Explained

This article breaks down Kafka's high‑throughput design by detailing how partitioning enables parallelism, sequential disk writes and zero‑copy reduce I/O overhead, and OS page cache maximizes memory efficiency, together allowing millions of messages per second.

Partitioninghigh throughputpage cache

0 likes · 4 min read

How Kafka Achieves Million‑Message Throughput: Partitions, Zero‑Copy, and Page Cache Explained

Code Mala Tang

Feb 17, 2026 · Backend Development

How to Build a High‑Throughput HTTP Server with Node.js

This article explains what high‑throughput means for an HTTP server, why Node.js’s event‑driven, non‑blocking architecture makes it ideal, and provides step‑by‑step code examples—including a basic server, blocking vs. async I/O, clustering, keep‑alive, JSON optimization, and streaming—to help developers design scalable, low‑latency services.

HTTPNode.jshigh throughput

0 likes · 8 min read

How to Build a High‑Throughput HTTP Server with Node.js

Alibaba Cloud Observability

Jan 26, 2026 · Cloud Native

How LoongCollector Delivers 10× Throughput and 80% Resource Savings in Cloud‑Native Observability

LoongCollector, the open‑source cloud‑native collector behind Alibaba Cloud's Simple Log Service, achieves ten‑fold higher throughput, up to 80% lower CPU and memory usage, near‑linear scaling, zero‑copy processing, lock‑free event pools and adaptive concurrency, while guaranteeing enterprise‑grade reliability for petabyte‑scale log and metric ingestion.

Lock-Freeadaptive-concurrencycloud-native

0 likes · 16 min read

How LoongCollector Delivers 10× Throughput and 80% Resource Savings in Cloud‑Native Observability

Mike Chen's Internet Architecture

Dec 3, 2025 · Big Data

Kafka High‑Throughput Tricks: Sequential Writes, Zero‑Copy, Partitioning

The article explains how Kafka achieves high throughput by writing messages sequentially to disk, leveraging OS page cache and zero‑copy system calls, using partitioned topics for parallelism, batching and compressing records on both producer and broker sides, and employing asynchronous replication with configurable persistence strategies.

CompressionKafkaPartitioning

0 likes · 4 min read

Kafka High‑Throughput Tricks: Sequential Writes, Zero‑Copy, Partitioning

Mike Chen's Internet Architecture

Nov 25, 2025 · Big Data

How Kafka Achieves High Throughput: Sequential Writes, Zero‑Copy, Partitioning, and More

This article explains the key techniques Kafka uses to maximize throughput, including sequential disk writes with zero‑copy, partition‑based parallelism, batch processing with compression, and configurable asynchronous replication for balancing performance and reliability.

Batch ProcessingCompressionKafka

0 likes · 5 min read

How Kafka Achieves High Throughput: Sequential Writes, Zero‑Copy, Partitioning, and More

Su San Talks Tech

Nov 9, 2025 · Backend Development

Designing a Scalable Real‑Time Video Ranking System for Billions of Views

This article walks through the complete system‑design process for a Douyin‑style video leaderboard, covering requirement analysis, functional and non‑functional specs, scale estimation, single‑machine baseline, fault‑tolerance, sharding, time‑window handling with a double‑pointer method, and caching strategies to achieve low latency, high throughput, and precise results at massive scale.

Distributed ArchitectureSystem Designhigh throughput

0 likes · 26 min read

Designing a Scalable Real‑Time Video Ranking System for Billions of Views

Architect Chen

Sep 29, 2025 · Big Data

How Kafka Achieves Ultra‑High Throughput: Sequential Writes, Page Cache, Batching & Zero‑Copy

This article explains the core Kafka techniques—sequential disk writes, Linux page cache usage, producer/broker batching with compression, and zero‑copy data transfer via sendfile—that together enable massive concurrency and near‑memory write performance in large‑scale streaming architectures.

Kafkabatchinghigh throughput

0 likes · 4 min read

How Kafka Achieves Ultra‑High Throughput: Sequential Writes, Page Cache, Batching & Zero‑Copy

Architect Chen

Sep 10, 2025 · Big Data

How Kafka Achieves Million‑Message Throughput: Sequential Writes, Page Cache, Batching & Zero‑Copy

The article explains how Kafka attains high‑throughput performance by using sequential disk writes, leveraging the OS page cache, employing producer and consumer batching with configurable parameters, and utilizing zero‑copy sendfile to minimize CPU and memory overhead, enabling stable million‑message per second rates.

KafkaSequential Writebatching

0 likes · 5 min read

How Kafka Achieves Million‑Message Throughput: Sequential Writes, Page Cache, Batching & Zero‑Copy

Tencent Cloud Developer

Aug 13, 2025 · Backend Development

How to Build a Streaming Reconciliation Engine for Billions of Transactions per Hour

This article explains the fundamentals of financial reconciliation, outlines the two‑step reconciliation process, and details a streaming reconciliation architecture capable of handling 30 billion USD per hour (≈27 777 TPS) with high concurrency, distributed parsing, and efficient result aggregation.

FinancialStreamingdistributed-systems

0 likes · 8 min read

How to Build a Streaming Reconciliation Engine for Billions of Transactions per Hour

Mike Chen's Internet Architecture

Jul 24, 2025 · Big Data

Boost Kafka Consumer Throughput: Multi‑Threading, Consumer Groups & Config Tuning

This guide explains why Kafka consumer throughput matters, then details practical techniques—including multi‑threaded consumption, scaling with consumer groups, client parameter tuning, and batch processing—to dramatically increase throughput while maintaining reliability in high‑concurrency, large‑scale data pipelines.

ConsumerKafkahigh throughput

0 likes · 5 min read

Boost Kafka Consumer Throughput: Multi‑Threading, Consumer Groups & Config Tuning

Mike Chen's Internet Architecture

Jul 17, 2025 · Big Data

How Kafka Achieves Million-Message Writes: Inside Its Sequential Append & Page Cache Magic

This article explains how Apache Kafka attains million‑level write throughput by sequentially appending messages to log files, leveraging operating‑system page cache and asynchronous disk flushing, and details the step‑by‑step data flow from producer to broker to storage.

Kafkahigh throughputmessage queue

0 likes · 4 min read

How Kafka Achieves Million-Message Writes: Inside Its Sequential Append & Page Cache Magic

Mike Chen's Internet Architecture

Jul 7, 2025 · Big Data

How Kafka Achieves Billion-Message Throughput: Sequential Disk Writes, Page Cache, and Zero‑Copy

This article explains how Kafka sustains massive traffic by writing logs sequentially to disk, leveraging the operating system’s page cache for fast in‑memory writes, employing zero‑copy techniques like sendfile to avoid user‑space copying, and batching messages to reduce network overhead, thereby delivering high‑throughput, low‑latency streaming.

Kafkabatchinghigh throughput

0 likes · 4 min read

How Kafka Achieves Billion-Message Throughput: Sequential Disk Writes, Page Cache, and Zero‑Copy

Code Mala Tang

Jul 3, 2025 · Databases

How We Achieved 50,000 TPS with PostgreSQL & Node.js: 6 Proven Optimizations

By systematically applying connection pooling, batch processing, WAL tuning, table partitioning, query refactoring, and OS-level tweaks, we transformed a PostgreSQL‑Node.js stack to sustain 50,000 transactions per second with sub‑100 ms latency, while maintaining data consistency on modest hardware.

Node.jsPostgreSQLdatabase tuning

0 likes · 10 min read

How We Achieved 50,000 TPS with PostgreSQL & Node.js: 6 Proven Optimizations

Instant Consumer Technology Team

Jun 30, 2025 · Operations

How 3FS Revolutionizes AI Storage with High‑Throughput Distributed Filesystem

3FS, DeepSeek’s high‑performance parallel file system, is engineered for AI workloads, offering ultra‑low latency, high‑throughput storage via RDMA, CRAQ consistency, and seamless cloud‑native integration, with detailed architecture, deployment steps, performance benchmarks, and cost‑saving strategies for large‑scale model training and inference.

AI storageRDMAdistributed file system

0 likes · 28 min read

How 3FS Revolutionizes AI Storage with High‑Throughput Distributed Filesystem

Senior Tony

Jun 8, 2025 · Backend Development

Why Kafka Handles Millions of Messages Per Second: Batch, Partition, Zero‑Copy, and Compression Explained

This article breaks down the core techniques that give Kafka its high‑throughput capability, including producer batch settings (batch.size, linger.ms), broker append‑only writes, consumer poll configuration, partition distribution, zero‑copy data transfer, dual‑thread processing, and configurable compression algorithms.

Batch ProcessingCompressionConsumer

0 likes · 8 min read

Why Kafka Handles Millions of Messages Per Second: Batch, Partition, Zero‑Copy, and Compression Explained

Raymond Ops

Jun 1, 2025 · Backend Development

RocketMQ vs Others: Key Features, Reliability, and Performance

RocketMQ, an Apache‑licensed distributed message queue from Alibaba, offers low latency, high reliability, and high throughput through features like distributed architecture, synchronous disk flushing, master‑slave replication, and flexible messaging models, and it compares favorably against Kafka, RabbitMQ, and ActiveMQ in various scenarios.

Distributed ArchitectureHigh reliabilityRocketMQ

0 likes · 11 min read

RocketMQ vs Others: Key Features, Reliability, and Performance

Mike Chen's Internet Architecture

May 22, 2025 · Big Data

Optimizing Kafka Producer for High Throughput: Batching, Asynchronous Sending, Compression, and Concurrency

This article details practical techniques for boosting Kafka producer throughput, covering batch sending optimization, linger.ms tuning, asynchronous send handling, compression options, and concurrent sending strategies, while providing code examples and performance considerations for high‑throughput streaming applications.

CompressionKafkaasynchronous

0 likes · 4 min read

Optimizing Kafka Producer for High Throughput: Batching, Asynchronous Sending, Compression, and Concurrency

Mike Chen's Internet Architecture

Apr 28, 2025 · Big Data

How Kafka Hits Million‑Message Throughput with Disk Writes, Caching & Zero‑Copy

Kafka attains million‑level message throughput by writing logs sequentially to disk, leveraging OS page cache for batch flushing, sending messages in large batches, and employing zero‑copy techniques that move data directly from disk to network sockets, dramatically reducing I/O overhead and CPU usage.

Kafkabatchingdisk I/O

0 likes · 4 min read

How Kafka Hits Million‑Message Throughput with Disk Writes, Caching & Zero‑Copy

Ma Wei Says

Mar 28, 2025 · Backend Development

Choosing the Right Message Queue: Kafka vs RocketMQ vs RabbitMQ Explained

This article compares Kafka, RocketMQ, and RabbitMQ, detailing their architectures, performance characteristics, strengths, and ideal use‑cases to help engineers select the most suitable message‑queue solution for high‑throughput, fault‑tolerant, and real‑time processing scenarios.

Event StreamingKafkaRabbitMQ

0 likes · 11 min read

Choosing the Right Message Queue: Kafka vs RocketMQ vs RabbitMQ Explained

Huolala Tech

Nov 8, 2024 · Backend Development

How Huolala Built a Scalable Real‑Time Reconciliation Platform for Millions of Daily Transactions

Huolala’s real‑time reconciliation platform tackles massive daily transaction volumes by addressing distributed system consistency, high‑throughput data ingestion, dynamic cluster scaling, and security safeguards, enabling sub‑second settlement verification across hundreds of services.

Data ConsistencyDynamic Scalingbackend architecture

0 likes · 10 min read

How Huolala Built a Scalable Real‑Time Reconciliation Platform for Millions of Daily Transactions

Tongcheng Travel Technology Center

Apr 17, 2024 · Backend Development

In-Depth Analysis of Apache RocketMQ Architecture, Operation Principles, and High‑Throughput Mechanisms

This article provides a comprehensive overview of Apache RocketMQ, detailing its core components, producer and consumer workflows, storage strategies, master‑slave synchronization, Raft‑based half‑write and leader election mechanisms, and best‑practice recommendations for high‑throughput, fault‑tolerant messaging systems.

RaftRocketMQbackend development

0 likes · 22 min read

In-Depth Analysis of Apache RocketMQ Architecture, Operation Principles, and High‑Throughput Mechanisms

Bilibili Tech

Apr 12, 2024 · Backend Development

Design and Optimization of a High‑Throughput Long‑Connection Service for Live Streaming

The article details a Golang‑based high‑throughput long‑connection service for live‑streaming, describing its five‑layer architecture, multi‑protocol support, load‑balancing, message‑queue decoupling, aggregation with brotli compression, multi‑region deployment, priority channels, and future enhancements for observability and intelligent endpoint selection.

Long ConnectionStreamingbackend architecture

0 likes · 16 min read

Design and Optimization of a High‑Throughput Long‑Connection Service for Live Streaming

Programmer DD

Jan 5, 2024 · Operations

Master the ‘Three Highs’: Availability, Throughput, and Scalability in System Design

This article explains the essential "three high" goals of system design—high availability, high throughput, and high scalability—detailing their meanings, common architectural patterns such as Hot‑Hot, Hot‑Warm, leader‑based clusters, and practical techniques like caching, async processing, and micro‑service isolation to build robust, scalable services.

System Designbackend architecturehigh availability

0 likes · 6 min read

Master the ‘Three Highs’: Availability, Throughput, and Scalability in System Design

Java High-Performance Architecture

Aug 11, 2023 · Operations

Unlock 10M+ HTTP Connections per Second: dperf’s DPDK‑Powered Performance

dperf, an open‑source Baidu tool built on DPDK, delivers ultra‑high network performance—supporting millions of HTTP connections per second, hundreds of Gbps throughput, and detailed statistics—making it ideal for load‑balancer, cloud VM, NIC, and CPU network testing, with step‑by‑step deployment instructions.

DPDKHTTP benchmarkingLoad Testing

0 likes · 7 min read

Unlock 10M+ HTTP Connections per Second: dperf’s DPDK‑Powered Performance

JD Tech

Jun 15, 2023 · Big Data

Event Bus: Architecture, Technical Challenges, and Solutions for High‑Throughput Data Standardization

This article introduces the event bus as a data pipeline for risk insight, explains its source‑transform‑sink architecture, outlines key technical challenges such as data heterogeneity and high‑throughput parsing, and presents solutions including standardized data models, plugin extensibility, low‑code hot‑loading, dynamic grouping, one‑click degradation, and traffic monitoring.

Data PipelineEvent BusPlugin Architecture

0 likes · 14 min read

Event Bus: Architecture, Technical Challenges, and Solutions for High‑Throughput Data Standardization

JD Cloud Developers

Apr 20, 2023 · Operations

How to Build a Cost‑Effective, High‑Throughput Log Collection System with ClickHouse

This article examines the challenges of scaling log storage and retrieval for high‑traffic services, analyzes the cost and performance limits of traditional ELK‑based pipelines, and presents a streamlined, UDP‑driven architecture using ClickHouse that dramatically reduces hardware expenses while handling hundreds of gigabytes per second.

ClickHouseCost OptimizationLog Collection

0 likes · 16 min read

How to Build a Cost‑Effective, High‑Throughput Log Collection System with ClickHouse

ITPUB

Oct 4, 2022 · Big Data

How Kafka Achieves Million‑TPS with Sequential I/O, MMAP, and Zero‑Copy

This article explains how Kafka attains million‑level transactions per second by leveraging sequential disk writes, memory‑mapped files, zero‑copy data transfer, and batch processing, detailing each technique's mechanics and performance impact.

Sequential I/Obig datahigh throughput

0 likes · 10 min read

How Kafka Achieves Million‑TPS with Sequential I/O, MMAP, and Zero‑Copy

Alibaba Cloud Big Data AI Platform

Aug 18, 2022 · Big Data

How Hologres Achieves Ultra‑High‑Throughput Real‑Time Writes in Big Data Warehouses

This article explores the challenges of real‑time data ingestion at massive scale, explains Hologres' innovative upsert model, performance optimizations, and compares its capabilities with other data‑warehouse solutions, demonstrating how it delivers low‑latency, high‑throughput writes for big‑data analytics.

HologresReal-time Data WarehouseUpsert

0 likes · 24 min read

How Hologres Achieves Ultra‑High‑Throughput Real‑Time Writes in Big Data Warehouses

58 Tech

Aug 11, 2022 · Backend Development

WLock: High‑Reliability, High‑Throughput Distributed Lock Service Based on WPaxos

WLock is an open‑source distributed lock service built on the WPaxos consensus algorithm and RocksDB storage, offering multiple lock types, flexible acquisition modes, high concurrency optimizations, and strong reliability and throughput for coordinating shared resources in distributed systems.

RocksDBWPaxosdistributed lock

0 likes · 12 min read

WLock: High‑Reliability, High‑Throughput Distributed Lock Service Based on WPaxos

Laravel Tech Community

Apr 21, 2022 · Fundamentals

What Is Wi‑Fi 7? Features, Speeds, Frequencies, and Release Timeline

Wi‑Fi 7 (802.11be) is a forthcoming wireless standard that promises peak speeds over 40 Gbps—up to four times faster than Wi‑Fi 6—by using wider 320 MHz channels, 4096‑QAM, multi‑link operation, and advanced OFDMA and MU‑MIMO technologies across 2.4 GHz, 5 GHz, and 6 GHz bands, with devices expected to appear by late 2023 or early 2024.

802.11beMLOWi-Fi 7

0 likes · 6 min read

What Is Wi‑Fi 7? Features, Speeds, Frequencies, and Release Timeline

JD Tech

Apr 8, 2022 · Big Data

Designing a High‑Throughput Log Collection System with ClickHouse, UDP and Compression

The article analyses the massive cost and performance challenges of traditional log‑storage pipelines at JD.com, then proposes a streamlined architecture that eliminates disk and MQ stages, uses in‑memory buffering, UDP transport, Snappy/ZSTD compression, and ClickHouse storage to achieve multi‑gigabyte per‑second throughput with over 70% hardware cost reduction.

ClickHouseLog Collectiondistributed systems

0 likes · 15 min read

Designing a High‑Throughput Log Collection System with ClickHouse, UDP and Compression

dbaplus Community

Dec 14, 2021 · Backend Development

How to Build a Cost‑Effective High‑Throughput Log Collection System with ClickHouse and UDP

This article analyzes the challenges of massive log storage and retrieval, calculates the bandwidth and hardware costs of traditional pipelines, and presents a streamlined architecture that uses in‑memory buffering, UDP transport, compression, and ClickHouse to achieve petabyte‑scale throughput while cutting storage costs by over 75%.

BackendLog CollectionUDP

0 likes · 16 min read

How to Build a Cost‑Effective High‑Throughput Log Collection System with ClickHouse and UDP

Open Source Linux

Nov 29, 2021 · Fundamentals

What Makes Wi‑Fi 7 Faster? Exploring the New 802.11be Features and Benefits

Wi‑Fi 7 (IEEE 802.11be EHT) introduces 320 MHz bandwidth, 4096‑QAM, Multi‑RU, Multi‑Link, enhanced MU‑MIMO and coordinated multi‑AP scheduling, delivering up to 30 Gbps throughput and ultra‑low latency for demanding applications such as 8K video, AR/VR, gaming, cloud and industrial IoT.

802.11beEHTWi-Fi 7

0 likes · 8 min read

What Makes Wi‑Fi 7 Faster? Exploring the New 802.11be Features and Benefits

JD Retail Technology

Nov 22, 2021 · Backend Development

Designing a High‑Performance Log Collection System with UDP, Compression, and ClickHouse

The article analyzes the high cost and scalability challenges of traditional log collection pipelines and proposes a streamlined architecture that uses in‑memory buffering, UDP transport, aggressive compression, and ClickHouse storage to achieve massive throughput while drastically reducing hardware and operational expenses.

ClickHouseCompressionLog Collection

0 likes · 15 min read

Designing a High‑Performance Log Collection System with UDP, Compression, and ClickHouse

Programmer DD

Aug 30, 2021 · Big Data

Why Is Kafka So Fast? Unveiling the Secrets Behind Its High Throughput

This article explains how Kafka achieves remarkable speed and massive throughput by using sequential disk I/O, OS page cache, zero‑copy transfers, partitioned log segments with indexes, batch processing, and efficient compression, making it a cornerstone of modern big‑data pipelines.

Kafkabig datahigh throughput

0 likes · 9 min read

Why Is Kafka So Fast? Unveiling the Secrets Behind Its High Throughput

ITFLY8 Architecture Home

Aug 11, 2021 · Backend Development

Building a High‑Throughput, Highly Available Message Center with RocketMQ, ES, and Kubernetes

This article outlines the technical and business goals, design architecture, technology selection, and operational considerations for creating a message center capable of handling 10,000 messages per second upstream and 1,000 downstream, while ensuring 100% high availability and easy integration with third‑party push services.

DevOpsRocketMQbackend architecture

0 likes · 5 min read

Building a High‑Throughput, Highly Available Message Center with RocketMQ, ES, and Kubernetes

macrozheng

Dec 15, 2020 · Big Data

How Kafka Achieves Million‑TPS Through Sequential I/O, MMAP, and Zero‑Copy

Kafka can sustain millions of transactions per second by writing data sequentially to disk, leveraging memory‑mapped files, employing zero‑copy DMA transfers, and batching messages, each technique reducing I/O overhead and CPU involvement, which together enable its high‑throughput performance in big‑data pipelines.

KafkaSequential I/Obig data

0 likes · 11 min read

How Kafka Achieves Million‑TPS Through Sequential I/O, MMAP, and Zero‑Copy

Programmer DD

Aug 14, 2020 · Big Data

How Kafka Achieves High Throughput: Architecture, GC Tweaks, and Memory Buffer Pools

This article explains how Kafka’s architecture, use of OS page cache, Sendfile optimization, and a custom memory buffer pool work together to minimize JVM garbage collection overhead and deliver the massive throughput required by big‑data messaging workloads.

GC optimizationKafkahigh throughput

0 likes · 23 min read

How Kafka Achieves High Throughput: Architecture, GC Tweaks, and Memory Buffer Pools

ITPUB

Jul 23, 2020 · Databases

How to Achieve 570,000 MySQL Writes per Second with TokuDB

This article details a real‑world test that loads 200 million rows into a MySQL TokuDB table in under six minutes, achieving roughly 570 k writes per second, and compares the results with InnoDB while providing configuration tips and bulk‑loader considerations.

Database OptimizationMySQLPerformance

0 likes · 6 min read

How to Achieve 570,000 MySQL Writes per Second with TokuDB

Java Backend Technology

Jun 16, 2020 · Big Data

How Kafka’s Architecture and Memory Pool Reduce JVM GC for High Throughput

This article explains how Kafka’s design—its broker architecture, use of sequential disk I/O, PageCache, Sendfile, and a custom memory buffer pool—optimizes JVM garbage collection and achieves massive throughput in big‑data messaging scenarios.

GC optimizationKafkabig data

0 likes · 21 min read

How Kafka’s Architecture and Memory Pool Reduce JVM GC for High Throughput

Tencent Cloud Developer

Apr 27, 2020 · Cloud Computing

High‑Throughput Cloud File Storage (CFS) Design for Kuaishou's Spring Festival Red‑Envelope Campaign

During Kuaishou's 2020 Spring Festival red‑envelope campaign, Tencent Cloud's Cloud File Storage (CFS) provided a high‑throughput, read‑heavy NFS solution that handled up to 5 GB/s across 1 700 pods, delivering 100% availability, low latency, and massive interaction volumes for billions of users.

CFSKuaishouNFS

0 likes · 22 min read

High‑Throughput Cloud File Storage (CFS) Design for Kuaishou's Spring Festival Red‑Envelope Campaign

Java Backend Technology

Mar 29, 2020 · Backend Development

How to Build a High‑Throughput Coupon Push System with a Redis‑Based Scheduled Task Cluster

This article explains how to design and implement a Redis‑driven, second‑level scheduled‑task cluster that reliably pushes millions of coupon notifications in real time, addressing latency, throughput, and single‑point‑failure challenges.

Coupon PushRedisdistributed system

0 likes · 9 min read

How to Build a High‑Throughput Coupon Push System with a Redis‑Based Scheduled Task Cluster

ITFLY8 Architecture Home

Mar 7, 2020 · Big Data

How Kafka’s Reactor Thread Model Powers High‑Throughput Messaging

Kafka’s high‑throughput network architecture—built on NIO, a Reactor thread model, and a TCP‑based protocol—evolves from a simple synchronous processor design to a decoupled handler‑pool system, offering valuable lessons for designing scalable backend communication layers in big‑data applications.

KafkaNIOReactor Model

0 likes · 7 min read

How Kafka’s Reactor Thread Model Powers High‑Throughput Messaging

Architecture Digest

Jan 19, 2020 · Big Data

Why Kafka Is So Fast: Sequential Writes, Memory‑Mapped Files, and Zero‑Copy

This article explains how Kafka achieves high throughput by using sequential disk writes, memory‑mapped files, batch compression, and zero‑copy sendfile for reads, while also covering data retention policies and the role of offsets in consumer processing.

Data StreamingMemory Mapped FilesSequential Write

0 likes · 10 min read

Why Kafka Is So Fast: Sequential Writes, Memory‑Mapped Files, and Zero‑Copy

Big Data Technology Architecture

Aug 3, 2019 · Big Data

Kafka Architecture: Design Principles for High Throughput and Reliability

This article explains Kafka's design background, persistence mechanisms, disk sequential I/O optimizations, network and compression strategies, and stability features such as partitioning, replication, and ISR, illustrating how these techniques enable high‑throughput, low‑latency real‑time log processing in big‑data environments.

Kafkadisk I/Ohigh throughput

0 likes · 9 min read

Kafka Architecture: Design Principles for High Throughput and Reliability

Architecture Digest

Aug 2, 2019 · Backend Development

Design and Implementation of a High‑Throughput WeChat Red‑Envelope System Simulating 10 Billion Requests

This article describes how a single‑machine prototype was built in Go to simulate a WeChat red‑envelope service capable of handling up to 6 × 10⁴ QPS, supporting one million concurrent connections, and processing 10 billion shake‑red‑envelope requests through careful load‑balancing, monitoring, and performance analysis.

Performancedistributed-systemsgolang

0 likes · 17 min read

Design and Implementation of a High‑Throughput WeChat Red‑Envelope System Simulating 10 Billion Requests

Big Data Technology Architecture

Jul 9, 2019 · Big Data

Kafka Best Practices for High Throughput: 20 Recommendations

This article presents New Relic's 20 best‑practice recommendations for operating Apache Kafka at high throughput, covering partitions, consumers, producers, and brokers, and explains key concepts, configuration tuning, monitoring, and architectural considerations to ensure reliable, scalable streaming pipelines.

Best PracticesBrokersConsumers

0 likes · 14 min read

Kafka Best Practices for High Throughput: 20 Recommendations

ITPUB

Jul 1, 2019 · Big Data

Why Kafka Handles Millions of Writes per Second: I/O Secrets Revealed

This article explains how Kafka achieves ultra‑high throughput by using sequential disk writes, memory‑mapped files, zero‑copy sendfile reads, and batch compression, while also describing its data retention policies and the trade‑offs of synchronous versus asynchronous producer modes.

Memory Mapped Filesbatch compressionhigh throughput

0 likes · 9 min read

Why Kafka Handles Millions of Writes per Second: I/O Secrets Revealed

dbaplus Community

Jun 20, 2019 · Big Data

How Kafka Hits Million‑Message Throughput Using Page Cache and Zero‑Copy

Kafka achieves its ultra‑high throughput and low latency by writing data to the OS page cache, performing sequential disk writes, and employing zero‑copy techniques that eliminate unnecessary data copies during consumption, enabling tens of thousands to millions of messages per second.

Kafkabig datahigh throughput

0 likes · 8 min read

How Kafka Hits Million‑Message Throughput Using Page Cache and Zero‑Copy

dbaplus Community

May 7, 2019 · Big Data

Why Kafka Achieves Million‑Level Throughput: Sequential Writes, mmap, and Zero‑Copy

This article explains how Kafka attains high throughput by using sequential disk writes, memory‑mapped files, sendfile zero‑copy, and batch compression, detailing both write and read path optimizations and their impact on performance.

KafkaMemory Mapped FilesSequential Write

0 likes · 8 min read

Why Kafka Achieves Million‑Level Throughput: Sequential Writes, mmap, and Zero‑Copy

Programmer DD

May 5, 2019 · Operations

20 Proven Kafka Best Practices to Scale High‑Throughput Streams

This article presents New Relic’s 20 best‑practice recommendations for Apache Kafka, covering partitions, consumers, producers, and brokers, to help engineers design, configure, and monitor high‑throughput, reliable streaming pipelines at scale.

Apache Kafkahigh throughput

0 likes · 13 min read

20 Proven Kafka Best Practices to Scale High‑Throughput Streams

Java Backend Technology

Nov 23, 2018 · Big Data

20 Proven Kafka Best Practices for High‑Throughput Enterprise Deployments

This article presents 20 practical best‑practice recommendations—from partition sizing and consumer tuning to producer configuration and broker optimization—to help engineers build scalable, reliable, and high‑throughput Apache Kafka clusters for large‑scale applications.

Apache KafkaBrokersConsumers

0 likes · 16 min read

20 Proven Kafka Best Practices for High‑Throughput Enterprise Deployments

dbaplus Community

Nov 20, 2018 · Backend Development

20 Proven Kafka Best Practices for High‑Throughput Clusters

This guide presents New Relic’s 20 practical best‑practice recommendations—covering partitions, consumers, producers, and brokers—to help engineers design, tune, and monitor Apache Kafka deployments for reliable, high‑throughput performance.

BrokersConsumersKafka

0 likes · 15 min read

20 Proven Kafka Best Practices for High‑Throughput Clusters

Alibaba Cloud Infrastructure

Aug 29, 2018 · Artificial Intelligence

Alibaba's FPGA-Based Ultra‑Low Latency, High‑Throughput Machine Learning Processor

Alibaba unveiled an FPGA‑designed machine‑learning accelerator that achieves sub‑millisecond inference latency and thousands of frames‑per‑second throughput, demonstrating how integrated hardware‑software optimizations can deliver real‑time AI performance surpassing conventional GPU and ASIC solutions.

AI acceleratorFPGAResNet18

0 likes · 5 min read

Alibaba's FPGA-Based Ultra‑Low Latency, High‑Throughput Machine Learning Processor

Architects' Tech Alliance

Jul 6, 2017 · Fundamentals

Understanding RDMA: Principles, Advantages, and Implementation Details

The article provides a comprehensive overview of RDMA technology, explaining its low‑latency, high‑throughput benefits, origins in InfiniBand, extensions like RoCE and iWARP, the Verbs API, communication models, and why it is poised for broader adoption in modern networking and storage systems.

RDMARoCEhigh throughput

0 likes · 9 min read

Understanding RDMA: Principles, Advantages, and Implementation Details

Architecture Digest

Apr 11, 2017 · Backend Development

Design and Practice of a High‑Throughput Spring Festival Red‑Packet System Supporting One Million Connections

This article describes how to design, implement, and evaluate a backend system that simulates the Spring Festival red‑packet service, achieving up to 6 × 10⁴ QPS on a single server while handling one million concurrent connections, and discusses the hardware, software, architecture, and performance results.

BackendQPSdistributed systems

0 likes · 18 min read

Design and Practice of a High‑Throughput Spring Festival Red‑Packet System Supporting One Million Connections

dbaplus Community

Jun 14, 2016 · Backend Development

How LeTV Scaled Its Order System to One Million Transactions per Second

The talk explains how LeTV’s BOSS platform used sharding, a custom distributed ORM called Mango, and Snowflake‑style ID generation together with cluster‑level request routing to expand its order processing from hundreds of thousands to over a million transactions per second.

BackendSnowflakedistributed-orm

0 likes · 5 min read

How LeTV Scaled Its Order System to One Million Transactions per Second

Java High-Performance Architecture

Feb 27, 2016 · Backend Development

How Kafka Achieves Ultra‑High Throughput: Sequential I/O, Zero‑Copy, and More

This article explains how Kafka’s design—using sequential disk reads/writes, zero‑copy system calls, file segmentation, batch sending, and message compression—delivers massive throughput while minimizing performance loss and network load.

Distributed Messagingbatch sendingdata compression

0 likes · 4 min read

How Kafka Achieves Ultra‑High Throughput: Sequential I/O, Zero‑Copy, and More

21CTO

Nov 25, 2015 · Backend Development

Why Alipay Chose MetaQ: Inside Its High‑Performance Distributed Messaging System

Metamorphosis (MetaQ), Alipay’s open‑source Java‑based distributed messaging middleware, offers high throughput, ordered storage, transaction support, and HA replication, positioning itself as a Kafka‑like solution tailored for massive log, broadcast, and sequential data flows, now powering billions of daily messages for Alipay and Taobao.

JavaKafka AlternativeMetaQ

0 likes · 6 min read

Why Alipay Chose MetaQ: Inside Its High‑Performance Distributed Messaging System

21CTO

Aug 10, 2015 · Backend Development

Boost Java Application Performance: Buffering, Caching, and Object Reuse Techniques

This article explains how to improve the throughput and response time of Java systems by using buffers, caches, object‑reuse pools, and time‑space trade‑off algorithms, providing concrete code examples and performance measurements for each technique.

BufferingJavaPerformance Optimization

0 likes · 20 min read

Boost Java Application Performance: Buffering, Caching, and Object Reuse Techniques