This article walks through Java GC tuning by defining low‑latency and high‑throughput goals, comparing major collectors, presenting G1 and ZGC configuration examples, and demonstrating a real‑world payment system case where pause times were reduced from 150‑200 ms to under 50 ms.
This article walks through the interview question of designing a message queue that handles billions of messages daily and peaks at millions of QPS, covering traffic calculations, core roles, storage and throughput techniques, scalability, high availability, observability, framework comparisons, a real‑world case study, and key follow‑up interview topics.
GS‑Playground is a next‑generation visual‑high‑fidelity robot simulator that cuts photo‑level rendering cost, automates asset creation, and narrows the Sim2Real gap, achieving up to 10,000 FPS on RTX 4090 and outperforming MuJoCo by 32× while supporting full‑stack parallel physics, 3DGS rendering, and end‑to‑end Real2Sim pipelines.
The article explains why traditional Cron‑based scheduling fails under high load, walks through three architectural stages—from database polling to Redis delay queues and finally a full distributed scheduler—showing code examples, design trade‑offs, and performance numbers that enable thousands of tasks per second.
This article breaks down Kafka's high‑throughput design by detailing how partitioning enables parallelism, sequential disk writes and zero‑copy reduce I/O overhead, and OS page cache maximizes memory efficiency, together allowing millions of messages per second.
This article explains what high‑throughput means for an HTTP server, why Node.js’s event‑driven, non‑blocking architecture makes it ideal, and provides step‑by‑step code examples—including a basic server, blocking vs. async I/O, clustering, keep‑alive, JSON optimization, and streaming—to help developers design scalable, low‑latency services.
LoongCollector, the open‑source cloud‑native collector behind Alibaba Cloud's Simple Log Service, achieves ten‑fold higher throughput, up to 80% lower CPU and memory usage, near‑linear scaling, zero‑copy processing, lock‑free event pools and adaptive concurrency, while guaranteeing enterprise‑grade reliability for petabyte‑scale log and metric ingestion.
The article explains how Kafka achieves high throughput by writing messages sequentially to disk, leveraging OS page cache and zero‑copy system calls, using partitioned topics for parallelism, batching and compressing records on both producer and broker sides, and employing asynchronous replication with configurable persistence strategies.
This article explains the key techniques Kafka uses to maximize throughput, including sequential disk writes with zero‑copy, partition‑based parallelism, batch processing with compression, and configurable asynchronous replication for balancing performance and reliability.
This article walks through the complete system‑design process for a Douyin‑style video leaderboard, covering requirement analysis, functional and non‑functional specs, scale estimation, single‑machine baseline, fault‑tolerance, sharding, time‑window handling with a double‑pointer method, and caching strategies to achieve low latency, high throughput, and precise results at massive scale.
This article explains the core Kafka techniques—sequential disk writes, Linux page cache usage, producer/broker batching with compression, and zero‑copy data transfer via sendfile—that together enable massive concurrency and near‑memory write performance in large‑scale streaming architectures.
The article explains how Kafka attains high‑throughput performance by using sequential disk writes, leveraging the OS page cache, employing producer and consumer batching with configurable parameters, and utilizing zero‑copy sendfile to minimize CPU and memory overhead, enabling stable million‑message per second rates.
This article explains the fundamentals of financial reconciliation, outlines the two‑step reconciliation process, and details a streaming reconciliation architecture capable of handling 30 billion USD per hour (≈27 777 TPS) with high concurrency, distributed parsing, and efficient result aggregation.
This guide explains why Kafka consumer throughput matters, then details practical techniques—including multi‑threaded consumption, scaling with consumer groups, client parameter tuning, and batch processing—to dramatically increase throughput while maintaining reliability in high‑concurrency, large‑scale data pipelines.
This article explains how Apache Kafka attains million‑level write throughput by sequentially appending messages to log files, leveraging operating‑system page cache and asynchronous disk flushing, and details the step‑by‑step data flow from producer to broker to storage.
This article explains how Kafka sustains massive traffic by writing logs sequentially to disk, leveraging the operating system’s page cache for fast in‑memory writes, employing zero‑copy techniques like sendfile to avoid user‑space copying, and batching messages to reduce network overhead, thereby delivering high‑throughput, low‑latency streaming.
By systematically applying connection pooling, batch processing, WAL tuning, table partitioning, query refactoring, and OS-level tweaks, we transformed a PostgreSQL‑Node.js stack to sustain 50,000 transactions per second with sub‑100 ms latency, while maintaining data consistency on modest hardware.
3FS, DeepSeek’s high‑performance parallel file system, is engineered for AI workloads, offering ultra‑low latency, high‑throughput storage via RDMA, CRAQ consistency, and seamless cloud‑native integration, with detailed architecture, deployment steps, performance benchmarks, and cost‑saving strategies for large‑scale model training and inference.
This article breaks down the core techniques that give Kafka its high‑throughput capability, including producer batch settings (batch.size, linger.ms), broker append‑only writes, consumer poll configuration, partition distribution, zero‑copy data transfer, dual‑thread processing, and configurable compression algorithms.
RocketMQ, an Apache‑licensed distributed message queue from Alibaba, offers low latency, high reliability, and high throughput through features like distributed architecture, synchronous disk flushing, master‑slave replication, and flexible messaging models, and it compares favorably against Kafka, RabbitMQ, and ActiveMQ in various scenarios.
This article details practical techniques for boosting Kafka producer throughput, covering batch sending optimization, linger.ms tuning, asynchronous send handling, compression options, and concurrent sending strategies, while providing code examples and performance considerations for high‑throughput streaming applications.
Kafka attains million‑level message throughput by writing logs sequentially to disk, leveraging OS page cache for batch flushing, sending messages in large batches, and employing zero‑copy techniques that move data directly from disk to network sockets, dramatically reducing I/O overhead and CPU usage.
This article compares Kafka, RocketMQ, and RabbitMQ, detailing their architectures, performance characteristics, strengths, and ideal use‑cases to help engineers select the most suitable message‑queue solution for high‑throughput, fault‑tolerant, and real‑time processing scenarios.
Huolala’s real‑time reconciliation platform tackles massive daily transaction volumes by addressing distributed system consistency, high‑throughput data ingestion, dynamic cluster scaling, and security safeguards, enabling sub‑second settlement verification across hundreds of services.
This article provides a comprehensive overview of Apache RocketMQ, detailing its core components, producer and consumer workflows, storage strategies, master‑slave synchronization, Raft‑based half‑write and leader election mechanisms, and best‑practice recommendations for high‑throughput, fault‑tolerant messaging systems.
The article details a Golang‑based high‑throughput long‑connection service for live‑streaming, describing its five‑layer architecture, multi‑protocol support, load‑balancing, message‑queue decoupling, aggregation with brotli compression, multi‑region deployment, priority channels, and future enhancements for observability and intelligent endpoint selection.
This article explains the essential "three high" goals of system design—high availability, high throughput, and high scalability—detailing their meanings, common architectural patterns such as Hot‑Hot, Hot‑Warm, leader‑based clusters, and practical techniques like caching, async processing, and micro‑service isolation to build robust, scalable services.
dperf, an open‑source Baidu tool built on DPDK, delivers ultra‑high network performance—supporting millions of HTTP connections per second, hundreds of Gbps throughput, and detailed statistics—making it ideal for load‑balancer, cloud VM, NIC, and CPU network testing, with step‑by‑step deployment instructions.
This article introduces the event bus as a data pipeline for risk insight, explains its source‑transform‑sink architecture, outlines key technical challenges such as data heterogeneity and high‑throughput parsing, and presents solutions including standardized data models, plugin extensibility, low‑code hot‑loading, dynamic grouping, one‑click degradation, and traffic monitoring.
This article examines the challenges of scaling log storage and retrieval for high‑traffic services, analyzes the cost and performance limits of traditional ELK‑based pipelines, and presents a streamlined, UDP‑driven architecture using ClickHouse that dramatically reduces hardware expenses while handling hundreds of gigabytes per second.
This article explains how Kafka attains million‑level transactions per second by leveraging sequential disk writes, memory‑mapped files, zero‑copy data transfer, and batch processing, detailing each technique's mechanics and performance impact.
This article explores the challenges of real‑time data ingestion at massive scale, explains Hologres' innovative upsert model, performance optimizations, and compares its capabilities with other data‑warehouse solutions, demonstrating how it delivers low‑latency, high‑throughput writes for big‑data analytics.
WLock is an open‑source distributed lock service built on the WPaxos consensus algorithm and RocksDB storage, offering multiple lock types, flexible acquisition modes, high concurrency optimizations, and strong reliability and throughput for coordinating shared resources in distributed systems.
Wi‑Fi 7 (802.11be) is a forthcoming wireless standard that promises peak speeds over 40 Gbps—up to four times faster than Wi‑Fi 6—by using wider 320 MHz channels, 4096‑QAM, multi‑link operation, and advanced OFDMA and MU‑MIMO technologies across 2.4 GHz, 5 GHz, and 6 GHz bands, with devices expected to appear by late 2023 or early 2024.
The article analyses the massive cost and performance challenges of traditional log‑storage pipelines at JD.com, then proposes a streamlined architecture that eliminates disk and MQ stages, uses in‑memory buffering, UDP transport, Snappy/ZSTD compression, and ClickHouse storage to achieve multi‑gigabyte per‑second throughput with over 70% hardware cost reduction.
This article analyzes the challenges of massive log storage and retrieval, calculates the bandwidth and hardware costs of traditional pipelines, and presents a streamlined architecture that uses in‑memory buffering, UDP transport, compression, and ClickHouse to achieve petabyte‑scale throughput while cutting storage costs by over 75%.
Wi‑Fi 7 (IEEE 802.11be EHT) introduces 320 MHz bandwidth, 4096‑QAM, Multi‑RU, Multi‑Link, enhanced MU‑MIMO and coordinated multi‑AP scheduling, delivering up to 30 Gbps throughput and ultra‑low latency for demanding applications such as 8K video, AR/VR, gaming, cloud and industrial IoT.
The article analyzes the high cost and scalability challenges of traditional log collection pipelines and proposes a streamlined architecture that uses in‑memory buffering, UDP transport, aggressive compression, and ClickHouse storage to achieve massive throughput while drastically reducing hardware and operational expenses.
This article explains how Kafka achieves remarkable speed and massive throughput by using sequential disk I/O, OS page cache, zero‑copy transfers, partitioned log segments with indexes, batch processing, and efficient compression, making it a cornerstone of modern big‑data pipelines.
This article outlines the technical and business goals, design architecture, technology selection, and operational considerations for creating a message center capable of handling 10,000 messages per second upstream and 1,000 downstream, while ensuring 100% high availability and easy integration with third‑party push services.
Kafka can sustain millions of transactions per second by writing data sequentially to disk, leveraging memory‑mapped files, employing zero‑copy DMA transfers, and batching messages, each technique reducing I/O overhead and CPU involvement, which together enable its high‑throughput performance in big‑data pipelines.
This article explains how Kafka’s architecture, use of OS page cache, Sendfile optimization, and a custom memory buffer pool work together to minimize JVM garbage collection overhead and deliver the massive throughput required by big‑data messaging workloads.
This article details a real‑world test that loads 200 million rows into a MySQL TokuDB table in under six minutes, achieving roughly 570 k writes per second, and compares the results with InnoDB while providing configuration tips and bulk‑loader considerations.
This article explains how Kafka’s design—its broker architecture, use of sequential disk I/O, PageCache, Sendfile, and a custom memory buffer pool—optimizes JVM garbage collection and achieves massive throughput in big‑data messaging scenarios.
During Kuaishou's 2020 Spring Festival red‑envelope campaign, Tencent Cloud's Cloud File Storage (CFS) provided a high‑throughput, read‑heavy NFS solution that handled up to 5 GB/s across 1 700 pods, delivering 100% availability, low latency, and massive interaction volumes for billions of users.
This article explains how to design and implement a Redis‑driven, second‑level scheduled‑task cluster that reliably pushes millions of coupon notifications in real time, addressing latency, throughput, and single‑point‑failure challenges.
Kafka’s high‑throughput network architecture—built on NIO, a Reactor thread model, and a TCP‑based protocol—evolves from a simple synchronous processor design to a decoupled handler‑pool system, offering valuable lessons for designing scalable backend communication layers in big‑data applications.
This article explains how Kafka achieves high throughput by using sequential disk writes, memory‑mapped files, batch compression, and zero‑copy sendfile for reads, while also covering data retention policies and the role of offsets in consumer processing.
This article explains Kafka's design background, persistence mechanisms, disk sequential I/O optimizations, network and compression strategies, and stability features such as partitioning, replication, and ISR, illustrating how these techniques enable high‑throughput, low‑latency real‑time log processing in big‑data environments.
This article describes how a single‑machine prototype was built in Go to simulate a WeChat red‑envelope service capable of handling up to 6 × 10⁴ QPS, supporting one million concurrent connections, and processing 10 billion shake‑red‑envelope requests through careful load‑balancing, monitoring, and performance analysis.
This article presents New Relic's 20 best‑practice recommendations for operating Apache Kafka at high throughput, covering partitions, consumers, producers, and brokers, and explains key concepts, configuration tuning, monitoring, and architectural considerations to ensure reliable, scalable streaming pipelines.
This article explains how Kafka achieves ultra‑high throughput by using sequential disk writes, memory‑mapped files, zero‑copy sendfile reads, and batch compression, while also describing its data retention policies and the trade‑offs of synchronous versus asynchronous producer modes.
Kafka achieves its ultra‑high throughput and low latency by writing data to the OS page cache, performing sequential disk writes, and employing zero‑copy techniques that eliminate unnecessary data copies during consumption, enabling tens of thousands to millions of messages per second.
This article explains how Kafka attains high throughput by using sequential disk writes, memory‑mapped files, sendfile zero‑copy, and batch compression, detailing both write and read path optimizations and their impact on performance.
This article presents New Relic’s 20 best‑practice recommendations for Apache Kafka, covering partitions, consumers, producers, and brokers, to help engineers design, configure, and monitor high‑throughput, reliable streaming pipelines at scale.
This article presents 20 practical best‑practice recommendations—from partition sizing and consumer tuning to producer configuration and broker optimization—to help engineers build scalable, reliable, and high‑throughput Apache Kafka clusters for large‑scale applications.
This guide presents New Relic’s 20 practical best‑practice recommendations—covering partitions, consumers, producers, and brokers—to help engineers design, tune, and monitor Apache Kafka deployments for reliable, high‑throughput performance.
Alibaba unveiled an FPGA‑designed machine‑learning accelerator that achieves sub‑millisecond inference latency and thousands of frames‑per‑second throughput, demonstrating how integrated hardware‑software optimizations can deliver real‑time AI performance surpassing conventional GPU and ASIC solutions.
The article provides a comprehensive overview of RDMA technology, explaining its low‑latency, high‑throughput benefits, origins in InfiniBand, extensions like RoCE and iWARP, the Verbs API, communication models, and why it is poised for broader adoption in modern networking and storage systems.
This article describes how to design, implement, and evaluate a backend system that simulates the Spring Festival red‑packet service, achieving up to 6 × 10⁴ QPS on a single server while handling one million concurrent connections, and discusses the hardware, software, architecture, and performance results.
The talk explains how LeTV’s BOSS platform used sharding, a custom distributed ORM called Mango, and Snowflake‑style ID generation together with cluster‑level request routing to expand its order processing from hundreds of thousands to over a million transactions per second.
This article explains how Kafka’s design—using sequential disk reads/writes, zero‑copy system calls, file segmentation, batch sending, and message compression—delivers massive throughput while minimizing performance loss and network load.
Metamorphosis (MetaQ), Alipay’s open‑source Java‑based distributed messaging middleware, offers high throughput, ordered storage, transaction support, and HA replication, positioning itself as a Kafka‑like solution tailored for massive log, broadcast, and sequential data flows, now powering billions of daily messages for Alipay and Taobao.
This article explains how to improve the throughput and response time of Java systems by using buffers, caches, object‑reuse pools, and time‑space trade‑off algorithms, providing concrete code examples and performance measurements for each technique.
