This article shares a year‑long, hands‑on experience of improving backend alert governance at Tencent Meeting, covering why alerts are hard, designing segmented error codes, building unified alert policies, driving team silence‑up, measuring progress, and the tools that make the process sustainable.
The article analyzes a real‑world financing workflow where a failed electronic contract generation exposed legal, loss, and reputation risks, and it outlines critical ordering, dependency, and rollback cost considerations to improve service orchestration reliability.
This article explains how to configure Nginx for rate limiting, including request rate control, burst handling, whitelisting, connection limits, and upload/download bandwidth throttling, with detailed directives, examples, and code snippets to ensure proper service stability.
This article records a systematic test of Elasticsearch refresh latency, revealing that update operations, a high proportion of deleted documents, and the soft‑delete setting significantly increase refresh time, while the large‑segment strategy and disabling soft delete can reduce latency without harming overall performance.
Baidu’s Aladdin search service safeguards stability during massive traffic spikes—such as Gaokao, the Tokyo and Beijing Olympics—by mapping dependencies, deploying multi‑dimensional monitoring, adding scaling layers like multi‑region Redis, and establishing rapid‑response on‑call teams, achieving over 99.99 % uptime and near‑real‑time data updates.
This article introduces the open‑source traffic‑replay tool TCPCopy, explains its 1.0 architecture—including the tcpcopy and intercept components—covers its packet‑capture and injection methods (raw socket vs pcap), TCP state handling, routing challenges, intercept role, and performance characteristics, providing practical insights for backend testing and operations.
This article breaks down WeChat’s 2018 overload control system for massive microservices, explaining the problem of service overload, detection via average waiting time, and a multi‑level priority‑based mitigation strategy that dynamically adjusts admission thresholds to keep billions of daily requests stable.
This article guides developers through classifying system‑level and business‑level bugs, using Linux utilities like perf, ps, and vmstat for quick root‑cause analysis, and outlines effective code‑design patterns and architectural strategies—caching, rate‑limiting, and high‑availability—to prevent and resolve production incidents.
Developers often flood systems with INFO logs, causing massive files that strain operations; this article outlines practical log‑slimming strategies—printing only essential logs, merging entries, using abbreviations, and context‑aware level switches—illustrated with a concrete case that reduced daily log volume from 5 GB to under 1 GB.
Arthas, Alibaba’s open‑source Java diagnostic tool, enables dynamic code tracing, real‑time JVM monitoring, and on‑the‑fly debugging without stopping applications; this guide covers installation, common scenarios, and core commands such as stack, jad, sc, watch, trace, jobs, logger, dashboard, and redefine for effective troubleshooting.
The Nginx Unit 1.27.0 release introduces HTTP‑to‑HTTPS redirection using $request_uri, configurable default filenames for pure‑path URIs, numerous bug fixes, expanded Linux distribution support, and updated Docker images with the latest language runtimes.
This article explains why Redis needs high availability, defines failure scenarios, compares common HA solutions, and walks through four deployment patterns—from a single instance to a three‑Sentinel architecture—highlighting their trade‑offs and practical implementation details.
This article examines common business scenarios that require timed tasks, introduces single‑machine and distributed scheduling solutions such as Timer, ScheduledExecutorService, Spring, Quartz, TBSchedule, elastic‑job, Saturn, and XXL‑Job, and provides a detailed feature‑by‑feature comparison to help choose the most suitable framework.
This article examines common business scenarios requiring timed tasks, compares single‑machine and distributed scheduling frameworks such as Timer, Spring, Quartz, TBSchedule, Elastic‑Job, Saturn and XXL‑Job, and provides guidance on selecting the most suitable solution.
When a Kafka cluster reaches load limits or experiences sudden traffic spikes, urgent expansion is needed, but data migration can be time‑consuming and risky; this guide outlines several practical techniques—including adjusting retention, adding partitions, leader switching, and single‑replica operation—to quickly scale clusters while minimizing data movement and service disruption.
This article analyzes various high‑availability strategies for stateful backend services—covering cold backup, dual‑machine hot standby, same‑city active‑active, remote active‑active, and multi‑region active‑active architectures—detailing their benefits, limitations, and practical implementation considerations.
This article explains how JD’s Open Platform’s Business Message Queue (BMQ) architecture, dynamic channels, retry and downgrade mechanisms, and real‑time monitoring ensure reliable, low‑risk message delivery across thousands of merchants while simplifying integration and scaling for future growth.
This article outlines comprehensive internal resource governance techniques—including degradation, circuit breaking, isolation, async conversion, thread‑pool management, JVM and hardware metric monitoring, and daily operational practices—to enhance system stability and high availability in large‑scale backend services.
This guide walks through a real‑world incident where a data platform’s CPU spiked to 98.94%, showing step‑by‑step how to identify the high‑load process, pinpoint the offending Java thread, analyze the root cause in the time‑utility code, and implement a performance‑focused solution that reduced load by thirtyfold.
The article describes how a high‑traffic payment platform achieves 99.999% availability by avoiding single points of failure, applying fail‑fast principles, implementing resource limits, building real‑time monitoring and alerting, and automating fault detection, routing, and recovery to ensure continuous 7×24 operation.
The article recounts a production incident where a GC alert triggered due to excessive object creation in a scheduled ad‑transaction sync task, and explains how the problem was diagnosed, mitigated by task splitting, and finally resolved through data sharding across multiple machines.
This article walks through a real‑world incident where an e‑commerce site suffered intermittent outages due to TCP SYN and accept queue overflows, explains the underlying handshake mechanics, shows how kernel and Nginx parameters can be tuned, and provides Python scripts for testing and SYN‑flood simulation.
This article explains how to create a lightweight yet reliable Python‑based log analysis tool that parses nginx logs with regular expressions, stores detailed metrics in MySQL, and provides fine‑grained performance and anomaly reports for web services.
This article shares practical methods for frontline engineers to quickly understand, assess, and resolve online system problems by categorizing system layers, evaluating impact, using essential Linux monitoring tools, and applying systematic troubleshooting and design‑for‑failure strategies to minimize downtime.
This article explores comprehensive service degradation techniques—including automatic and manual switchovers, read/write and multi‑level fallback strategies, and practical examples like timeout, failure count, and traffic throttling—to ensure core functionality remains available during traffic spikes or component failures in high‑concurrency systems.
This article traces the evolution of image server architectures—from a simple single‑directory setup on Windows/.NET, through cluster‑based real‑time synchronization and shared‑storage solutions, to modern FastDFS‑backed distributed file systems with CDN acceleration—highlighting scalability, reliability, and migration challenges.
This article explains how stateless application servers combined with load balancers form scalable clusters, detailing request routing, scaling strategies, and common implementations such as Nginx for HTTP forwarding and LVS for IP‑level balancing.
This article explains how small‑team internet products evolve from a single‑server setup to a scalable, clustered web architecture by separating services, adding dedicated servers, and employing caching, CDNs, and other components to handle growing user traffic.
