GitHub’s uptime has fallen sharply, with hundreds of incidents—including dozens of major outages—largely fueled by AI‑driven code generation, prompting high‑profile users to migrate, leadership to prioritize availability, and a costly overhaul of capacity and architecture.
This article walks through the complete interview‑style design of a billion‑scale URL shortener, covering requirements, capacity estimation, API definitions, database schema, short‑code generation algorithms, sharding, caching, load balancing, rate limiting, and expiration handling, while illustrating each step with concrete examples and calculations.
This guide walks SRE and senior operations engineers through a real-world service‑avalanche incident, detailing alert hierarchy design, fault‑location commands, emergency SOPs, capacity‑baseline building, and post‑mortem best practices to dramatically reduce MTTR in distributed micro‑service environments.
Meituan's database team created a sandbox‑based capacity assessment platform that replays live traffic, uses accelerated replay to discover performance bottlenecks, and closes the loop with capacity monitoring and automated operations, dramatically improving stability and resource utilization.
This guide walks through diagnosing intermittent 502 Bad Gateway errors in Nginx by analyzing error logs, checking upstream and FastCGI timeout settings, reviewing PHP‑FPM configuration, performing performance tuning, and outlining advanced troubleshooting, monitoring, and capacity‑planning strategies to ensure stable high‑traffic deployments.
This article breaks down performance testing into three essential dimensions—time behavior, resource utilization, and capacity—explains their key metrics, demonstrates a detailed e‑commerce flash‑sale case study, and shows how systematic testing and optimization can dramatically improve response times, throughput, and scalability.
This article provides a comprehensive framework for evaluating system support capability by defining QPS excellence thresholds across industry benchmarks, business types, response time, resource efficiency, performance metrics, architectural guidelines, optimization tactics, and real‑world case studies, culminating in a practical calculation formula.
This article explains why interviewers probe sharding, walks through the typical chain of questions, compares partition tables and read/write splitting, shows how to decide between sharding strategies, estimate capacity, plan expansion, and safely migrate data with traffic replication.
This guide explains what QPS is, outlines three practical methods to estimate it—including business‑scenario modeling, historical data analysis, and industry benchmarking—covers key influencing factors, shows formulas linking QPS, concurrency and response time, and recommends tools and best‑practice tips for reliable capacity planning.
This guide walks senior backend engineers through the strategic reasoning, capacity estimation, and step‑by‑step migration techniques for sharding databases, covering when to split, choosing between partitioning, read/write splitting, or full sharding, and how to plan safe expansions.
The article reveals the hidden gap between textbook Kubernetes knowledge and real production failures, offering six practical skills—from interpreting pod symptoms and debugging without logs to capacity planning and treating events as first‑class signals—essential for engineers to survive on‑call crises that interview questions never cover.
Meituan’s Database Capacity Assessment System uses online traffic replay, accelerated load testing, and automated analysis to safely evaluate and optimize database read/write capacity, providing real‑time operation insights, flexible scaling, and reliable change‑risk mitigation for large‑scale production environments.
This guide explains how to unleash vLLM’s full potential by optimizing batch size, leveraging 4‑bit quantization, tuning concurrency parameters, planning capacity with token‑per‑second metrics, and implementing robust monitoring to balance latency, cost, and scalability in production deployments.
Meituan’s database team introduced a capacity evaluation system that uses online traffic replay in sandbox environments, accelerated replay to uncover performance bottlenecks, and a capacity‑operation loop to monitor and manage cluster resources, thereby improving database stability, safety of changes, and resource utilization.
This comprehensive SRE guide walks through a real-world service avalanche incident, detailing alert triggering, root‑cause analysis, step‑by‑step recovery, capacity baseline creation, layered alert design, automated scripts, and post‑mortem best practices to help engineers prevent and resolve large‑scale outages.
This article explores how integrating AI‑driven AIOps with the TiDB distributed database can automate monitoring, enable proactive anomaly detection, streamline root‑cause analysis, and optimize capacity planning, ultimately shifting database operations from manual firefighting to intelligent, data‑driven management.
This article explores how AIOps transforms traditional IT operations by using AI for anomaly detection, root‑cause analysis, capacity forecasting, and self‑healing, offering a step‑by‑step roadmap, real‑world code examples, toolchain recommendations, common pitfalls, and future trends for building intelligent, automated operations.
This article details Qunar's end‑to‑end solution for forecasting traffic spikes, estimating required CPU resources, and automatically scaling hotel services using a combined flow‑calendar, algorithmic prediction, and Ops‑driven auto‑scaling pipeline, improving stability and operational efficiency.
During the Double‑11 shopping festival, a leading e‑commerce platform suffered a P0 outage when its Eureka service‑discovery cluster overloaded, triggering a full‑chain failure that lasted 2 hours 42 minutes and caused losses exceeding 1.2 billion yuan; the article dissects the timeline, root causes, capacity mis‑planning, monitoring gaps, and remediation strategies.
Many managers blame the workshop when capacity falls short, but the real issue lies in misaligned production, planning, procurement, and logistics; this article explains how to diagnose bottlenecks, use data-driven methods, and implement coordinated supply‑chain strategies to boost throughput.
This article explains how to determine a 100 000‑QPS rate‑limiting threshold by covering the purpose of throttling, the three core elements of limiting, common algorithms, target dimensions, capacity estimation for single‑service and full‑link scenarios, pressure‑testing techniques, monitoring data, and adaptive configuration strategies.
Effective technical optimization requires moving from isolated, point‑style ideas to a comprehensive, measurable framework that quantifies goals, assesses gaps, designs capacity, monitors key services and links, and establishes clear compensation and incident‑handling procedures, ensuring a complete, closed‑loop solution.
The article walks through why sharding is needed, outlines IO and CPU bottlenecks, presents two design principles, shows how to estimate capacity from existing data and growth trends, compares range, modulo, consistent‑hash and Snowflake sharding schemes, and details migration strategies for expanding nodes without downtime.
This comprehensive guide outlines six critical areas of modern system operations—including real‑time monitoring, security safeguards, automation, fault diagnosis, collaborative teamwork, and process optimization—offering practical strategies and tools such as Zabbix, Prometheus, ELK, Redis, Ansible, and capacity planning to ensure stable, efficient enterprise services.
This article explains why many companies struggle with inaccurate plans, defines supply chain planning as the dynamic coordination of demand, capacity, and inventory, and provides a step‑by‑step ERP‑based framework—including demand forecasting, capacity analysis, inventory control, and execution—to achieve reliable, data‑driven operations.
This document describes a comprehensive capacity‑planning solution that predicts traffic‑peak impacts for hotel services, automatically estimates required CPU resources, creates timed scaling tasks, and evaluates performance using detailed metrics, thereby improving operational efficiency and reducing manual effort during events such as exam‑ticket printing and holiday travel surges.
This article explains the key performance metrics TPS (transactions per second) and QPS (queries per second), provides formulas for calculating them, and demonstrates practical calculations for multi-node deployments, illustrating how request latency, thread pools, and instance count affect overall system concurrency and throughput.
This article presents a comprehensive guide to building high‑availability systems, covering availability metrics, fault prevention, detection and recovery, capacity evaluation, layered architecture design, service tiering, resilience mechanisms, and operational best practices for reliable service delivery.
This article provides a systematic overview of high‑availability system design, covering availability metrics, fault prevention, detection, recovery, capacity planning, service tiering, data layer resilience, monitoring, and the responsibilities of architects, SREs, and developers to ensure reliable, scalable services.
The article reviews major 2024 service outages—from Alibaba Cloud to OpenAI—highlights their root causes, and offers practical operations strategies such as disaster recovery, regular backups, load balancing, monitoring, performance tuning, and capacity planning to reduce future downtime.
This comprehensive guide explains how to design, build, and operate high‑availability systems by covering availability metrics, fault‑tolerance strategies, capacity planning, code and data layer architecture, automated testing, monitoring, and clear role responsibilities to ensure services stay reliable and resilient under load.
This article explains how system stability depends on architecture and code details, defines SLA and the “nines” metric, outlines Google’s SRE hierarchy, and provides practical governance steps—including development and release processes, high‑availability design, capacity planning, monitoring, incident response, and team culture—to achieve reliable, high‑availability services.
This article chronicles the evolution of Huolala’s Kafka infrastructure—from an integrated compute‑storage design to a separated compute‑storage model with multi‑tenant deployment, and finally to a cloud‑native elastic architecture—detailing the challenges of capacity awareness, alarm configuration, and cost‑effective performance optimization.
Bilibili’s 18‑month, multi‑regional data‑center migration moved tens of thousands of servers using a high‑frequency rolling strategy, combining meticulous planning, cross‑team coordination, automated rack placement and rigorous checklists to achieve significant cost savings, higher utilization, improved stability and greener operations.
This article details HuoLala's three‑year journey of systematic business‑peak assurance, covering goal definition, project‑management practices, technical risk mitigation, cloud‑provider coordination, and post‑event reviews that together delivered zero‑fault high‑traffic periods and continuously improving system stability.
This comprehensive guide outlines the seven key pillars—operations, high‑availability architecture, capacity governance, change management, risk governance, fault management, and chaos engineering—that together form a systematic approach to building and maintaining a reliable, 24‑hour backend system.
An October 2023 outage of the bfe‑customer‑application‑query‑svc revealed how traffic spikes, downstream latency, and a mis‑configured custom thread pool saturated Tomcat's worker threads, causing health‑check failures, pod restarts, and a complete zone failure.
This article demystifies Site Reliability Engineering (SRE) by explaining its origins, core responsibilities, essential skill sets, and key practices such as observability, incident response, testing, capacity planning, automation, user support, on‑call duties, and the definition of SLI/SLO/SLA, providing a comprehensive guide for modern operations teams.
iQIYI implemented full‑link load testing of its playback service using LoadMaker for traffic generation and Rover for link control, mapping the topology, creating weighted user scenarios, and safely pressurizing production‑like environments, which validated multi‑times historical peak capacity, uncovered bottlenecks, and enabled several performance and disaster‑recovery improvements without impacting real users.
This article explains the definitions of stability and high availability, distinguishes their relationship, outlines key performance indicators, and provides a comprehensive framework—including fault prevention, detection, and recovery, as well as design, coding, testing, monitoring, and emergency response practices—to help teams build reliable, highly available systems.
This article details ICBC Software Development Center's comprehensive approach to performance capacity testing, covering background challenges, a structured quality practice plan, enhanced test scope evaluation, result analysis, tool support, implementation outcomes, and future directions for ensuring system stability and scalability.
This article explains the concept of story points as a relative estimation unit, why agile teams use them, how they are applied across Scrum ceremonies, and answers common questions about their relationship to effort, value, and managerial decision‑making.
This article details Bilibili's end‑to‑end technical planning, traffic‑estimation models, and concrete optimizations—including hotspot caching, traffic dispersion, long‑connection isolation, and automated fault‑injection—that enabled the S13 League of Legends finals to serve over 1.2 billion viewers with stable, low‑latency streaming.
This article recounts the author's experience preparing JD's Customer Data Platform (CDP) for the Double 11 shopping festival, detailing the platform's capabilities, business scenarios, capacity planning, stability and performance challenges, disaster‑recovery measures, and personal reflections on the intensive technical effort involved.
Kapacity V0.2 introduces an AI‑powered, traffic‑driven replica prediction algorithm for cloud‑native autoscaling, featuring a Linear‑Residual model, a lightweight Swish Net time‑series forecaster, custom metric support, and open‑source tools, aiming to improve resource efficiency and reduce operational risk.
This article breaks down high‑availability system design into six critical layers—architecture, development standards, application services, storage, product safeguards, and operations—offering concrete practices such as capacity planning, fault‑tolerant patterns, monitoring, and incident‑response strategies to achieve four‑nine (99.99%) uptime.
This article details Huolala's big‑data architecture and the practical measures—ranging from background analysis and stability challenges to case studies, discovery mechanisms, capacity planning, high‑availability, and automation—that the company employs to guarantee Doris's reliability and performance across its rapidly growing logistics platform.
This guide explains the Linux kernel parameters that limit the number of open files and TCP connections on a server, shows how to adjust those limits, calculates practical connection capacities for both servers and clients, and offers troubleshooting tips for the "too many open files" error.
This article presents HuoLala's practical approach to guaranteeing the stability of the Doris OLAP engine within its large‑scale big data platform, covering background, challenges, case studies, capability building, process standards, and future planning.
This article details Tencent's cloud load‑testing approach for handling massive traffic spikes during Chinese New Year, covering background challenges, model selection, script authoring options, data construction, report analysis, and real‑world case studies that demonstrate capacity planning and performance optimization.
This article explains how to achieve 24/7 service availability through comprehensive monitoring, fault handling, release management, and capacity planning, covering alarm types, batch processing, traffic and resource metrics, fault causes and mitigation, deployment strategies, scaling commands, and service degradation techniques.
This article explains why automation testing often lags behind product changes, outlines essential load‑testing concepts such as bottleneck analysis and capacity planning, and provides hands‑on guidance for using Wrk and GoReplay tools within an SRE‑driven operations workflow.
This guide explains how enterprises can select appropriate Alibaba Cloud ECS instance types using spec families or API queries, match them to workloads such as web services, big data, or AI, and verify capacity and performance through systematic PTS load testing and capacity planning.
The article outlines a comprehensive high‑availability architecture framework across six layers—development standards, application services, storage, product fallback, operations deployment, and emergency response—detailing design principles such as stateless services, elastic scaling, redundant storage, robust monitoring, gray releases, and chaos engineering to ensure resilient, continuously available systems.
This article explains how micro‑enterprise SaaS leader Weimeng built a full‑link load‑testing platform to simulate real‑world traffic for major shopping festivals, detailing the challenges, architecture, capabilities, results, and future plans for ensuring system stability and performance at scale.
The article outlines a comprehensive stability‑assurance framework for large‑scale promotional events—detailing planning, capacity and pressure‑test rehearsals, strict change‑freeze, internal gray releases, coordinated on‑call response, thorough link and capacity analysis, monitoring, emergency procedures, cross‑team collaboration, external partner coordination, and post‑event review to ensure resilient system performance.
This article breaks down the essential design and operational considerations for achieving high availability across six layers—development standards, application services, storage, product strategy, operations deployment, and incident response—providing concrete practices, metrics, and safeguards to reach four‑nine (99.99%) uptime.
This article explains what stability assurance is, outlines a systematic workflow—including anomaly identification, monitoring configuration, impact assessment, and solution planning—and provides practical methods such as capacity estimation, traffic limiting, load testing, scaling, and pre‑heating to ensure services remain stable during both daily operations and high‑traffic events.
This article explains how enterprises can design and implement a comprehensive observability platform—covering metrics, logs, tracing, fault response, post‑mortems, testing, capacity planning, and automation—to improve system reliability and user experience.
This article systematically explores high‑availability system design from development standards, capacity planning, application services, storage, product strategies, operations deployment, to incident response, presenting key concepts, architectural patterns, and practical guidelines for building resilient services.
The technical‑assurance team secured Bilibili’s high‑write live‑stream gift system by expanding capacity, isolating hot keys, refactoring pipelines, adding asynchronous writes, employing horizontal scaling and full‑link load testing, converting uncertain dependencies into graceful fallbacks, and deploying dual‑active, chaos‑engineered disaster‑resilience architecture aligned with business usage patterns.
This article analyzes how semiconductor design firms can balance compute capacity and cash flow by modeling monthly core‑hour demand across project phases, presenting beginner and experienced algorithms, realistic usage conversion, and strategic choices between local, cloud, and hybrid resources.
High availability—measured in nines of uptime—requires partitioning systems, decoupling components, choosing robust technologies, deploying redundant instances with automatic failover, capacity planning, rapid scaling, traffic shaping, resource isolation, global protection, observability, and disciplined change management to achieve stable, resilient services.
This article details Bilibili's SRE approach to large‑scale live events, covering background, activity scenarios, resource planning, performance testing, chaos‑engineering drills, technical safeguards such as DCDN, SLB, WAF, PaaS, cache and DB, pre‑plan capabilities, post‑mortem analysis, and future outlook, illustrating how systematic capacity management and automated resilience practices enable stable operation for events with tens of millions of concurrent users.
Bilibili’s SRE team ensures flawless large‑scale online events by meticulously gathering activity details, provisioning DNS, CDN, networking and compute resources, conducting multi‑stage performance tests and chaos‑engineering drills, applying layered traffic controls, maintaining historical checklists, executing predefined contingency responses, and iterating post‑mortems to drive continuous automation and reliability.
This article outlines a comprehensive, cloud‑native methodology for preparing large‑scale sales events like the 618 promotion, covering uncertainty challenges, capacity assessment, performance testing, pre‑heating strategies, flow‑control, and MSE service‑governance techniques to ensure stable, cost‑effective operation.
This article walks through a scenario‑driven capacity assessment for a production‑grade Kafka cluster, covering QPS calculations, storage needs, physical machine count, disk choices, memory, CPU, network bandwidth, deployment steps, and a final resource summary.
This article explains the importance of conducting full‑link load testing in production environments, outlines the evolution and solution architecture, describes key technologies such as traffic coloring, data isolation and risk control, and shares practical implementation steps and customer case studies from Alibaba.
This article explains how the Havok full‑link load testing platform was designed and built to replay real traffic safely, provide capacity‑assessment data, support multiple test types, and offer real‑time monitoring and circuit‑breaker protection for large‑scale online services.
This article compiles core architectural concepts—including ROI, SOLID principles, system splitting, isolation, ACID, CAP/BASE, distributed transactions, and capacity estimation—explaining their definitions, practical examples, trade‑offs, and how they guide architects in making informed technical decisions.
This article explains the SRE role originated by Google, outlines its core responsibilities such as automation, observability, incident response, testing, capacity planning, and SLI/SLO/SLA management, and highlights the skills and cultural practices needed for reliable service operations.
The article presents a case study of how Alibaba Cloud’s network operations team tackled the massive, unpredictable traffic spikes of the 2021 Double 11 shopping festival by identifying transaction‑promotion traffic risks early through AI‑powered analysis, overcoming the limitations of manual rule‑based detection, and achieving precise, automated capacity risk control.
The article reviews the Xi'an “One Code Pass” health‑code platform, covering its award recognition, recent service outages, capacity‑planning calculations, security‑platform procurement, Ministry engineer inspection, and the identified technical bottlenecks such as lack of CDN for static assets and insufficient outbound bandwidth.
This article explains how to design and evaluate system capacity by defining key metrics such as QPS, TPS and concurrency, describing when capacity assessment is needed, and outlining a step‑by‑step methodology—including traffic analysis, peak estimation, stress testing and redundancy planning—to ensure reliable performance under varying load conditions.
This article details the end‑to‑end preparation, execution, reporting, and retrospective of a large‑scale production‑environment load test for the Double 11 shopping festival, covering data preparation, QPS target calculation, multi‑scenario testing, issue analysis, and continuous improvement practices.
This article shares practical experiences and best‑practice guidelines for deploying and operating Prometheus in Kubernetes, covering version selection, inherent limitations, exporter choices, metric design, multi‑cluster scraping, memory and storage planning, GPU monitoring, timezone handling, and alerting considerations.
The article explains the reasons for scaling, outlines hardware and component expansion strategies, introduces the AKF splitting principle for distributed systems, discusses database clustering and distributed ID generation methods such as UUID and Snowflake, and describes elastic scaling challenges and solutions.
This article explains why capacity expansion is necessary, outlines hardware and component scaling strategies, introduces the AKF splitting principle for Redis clusters, discusses challenges of distributed scaling such as data consistency and high concurrency, and reviews database clustering and distributed ID generation methods like UUID and Snowflake.
The article explains why performance optimization has limits and outlines practical scaling methods—including whole‑machine and component upgrades, AKF splitting, database clustering, distributed ID generation (UUID and Snowflake), and elastic scaling—while also discussing the challenges each approach introduces.
This article explains why capacity expansion is necessary, outlines hardware and component scaling strategies, introduces AKF splitting principles, discusses database clustering and distributed ID generation methods such as UUID and Snowflake, and highlights the benefits and challenges of elastic scaling.
The article explains how to assess system capacity by analyzing daily traffic, calculating average and peak QPS, applying the 80/20 rule, conducting stress tests, and adjusting resources, illustrated with a sports event example and a book reservation system case study.
This article explains how to design and evaluate system capacity—covering concepts like QPS, concurrency, peak‑traffic estimation, stress testing, and practical steps to ensure your services can handle both normal loads and sudden traffic spikes without failure.
This article explains why sizing a database cluster is challenging, outlines a systematic capacity‑planning process, examines workload characteristics, query‑operation mapping, consistency trade‑offs, and maintenance considerations, and demonstrates how the open‑source NoSQL database ScyllaDB can be used to model and simplify these decisions.
This article presents a comprehensive, step‑by‑step guide on improving system reliability through early fault detection, scope reduction, frequency reduction, and rapid incident handling, using real‑world practices from Baidu's commercial hosting platform.
This article explains how to estimate and plan system capacity—covering concepts like QPS, concurrency, peak traffic calculation, load testing, and practical steps with examples—to ensure services can handle both regular and surge workloads effectively.
This article explains how to assess and design system capacity by translating a real‑world sports event scenario into concrete metrics such as daily visits, average and peak QPS, concurrency, and instance limits, while outlining practical steps, formulas, and a book‑reservation case study.
This article explains how to design and evaluate system capacity by using a sports event example, defining key metrics such as QPS, TPS and concurrency, applying the 80/20 rule, performing stress tests, and calculating required instances for reliable operation.
This talk shares practical high‑availability strategies learned from years of building Taobao’s massive e‑commerce platform and migrating to Alibaba Cloud, covering traditional IDC stability mechanisms, cache and disaster‑recovery designs, cloud‑native fault‑tolerance, capacity planning, rate‑limiting, graceful degradation, and multi‑region resilience.
Shopee Shake’s architecture separates admin and user sides into three layers—access, application, and resource—and uses horizontal scaling, bucketed Redis coin pools, multi‑level caching, asynchronous message queues, precise capacity formulas, and comprehensive monitoring and chaos‑engineered runbooks to reliably handle over 300,000 QPS during major promotional events.
The article explains why large‑scale services must conduct production full‑link load testing, describes its evolution from ad‑hoc trials to standardized monthly practices, and details the technical and procedural steps—including traffic modeling, JMeter usage, middleware tagging, and responsibility mapping—that ensure reliable capacity planning and risk mitigation.
This document outlines a comprehensive SRE‑driven operational framework for ensuring stable, high‑availability online education services during peak summer and winter periods, detailing pre‑, during‑, and post‑maintenance phases, architectural principles, load testing, monitoring, capacity management, safety hardening, chaos engineering, incident response, and post‑mortem practices.
Alibaba’s eight‑year Double 11 journey illustrates how the company tackled exponential business growth by inventing high‑availability middleware, precise capacity planning, unit‑based deployment, online stress testing, hybrid‑cloud elasticity, and intelligent runtime control to balance throughput, cost, and user experience during the midnight peak.
This article explains how to estimate and design system capacity by analyzing daily traffic, calculating average and peak QPS, applying the 80/20 rule, performing stress tests, and adjusting resources to handle temporary spikes, initial launches, and growing workloads.
This article explains how to design and evaluate system capacity by defining key concepts such as design capacity, TPS, QPS, and concurrency, outlining when capacity assessment is needed, and providing a step‑by‑step methodology with real‑world examples and calculations for accurate performance planning.
The article analyzes WeChat Moments' massive image and video services, detailing its OC/IDC architecture, holiday traffic challenges, software and hardware safeguards, disaster‑recovery mechanisms, retry policies, and a series of flexible strategies—including compression format changes, bitrate reduction, buffer pools, and timeline throttling—to sustain billions of daily accesses.
This article analyzes the large‑scale architecture of WeChat Moments, detailing image and video traffic characteristics, hardware and software safeguards, disaster‑recovery mechanisms, capacity assessment, and a series of flexible strategies such as compression format changes, bitrate reduction, buffer pools, and timeline throttling to handle holiday spikes.
This article explains how to assess and design system capacity by analyzing real‑world scenarios—such as a company sports event—calculating required concurrency, average and peak QPS using the 80/20 rule, performing load tests, and determining instance counts to ensure reliable performance under varying traffic spikes.
This article clarifies key performance metrics such as QPS, TPS, PV, UV, DAU, MAU, concurrent users, and system throughput, explains their differences, relationships, and how they impact capacity planning, while also outlining essential performance testing concepts and evaluation methods for robust system design.
This article explains how to estimate and design system capacity—including traffic, concurrency, QPS, and resource requirements—by illustrating concepts, analytical formulas, peak‑load assessment techniques, performance testing, and a practical case study of a book‑reservation system.
This guide explains Kubernetes capacity planning, showing how to detect idle CPU and memory, identify wasteful namespaces, use open‑source tools like kube‑state‑metrics and cAdvisor, and apply PromQL queries to optimize resource requests and measure the impact of your improvements.
