This guide teaches operations engineers a four‑step “Wang‑Wen‑Wen‑Qie” methodology—observing system health, listening via alerts, questioning changes, and pulse‑checking with profiling tools—to rapidly diagnose and resolve high‑impact incidents while maintaining clear communication and post‑mortem learning.
A MySQL server experienced max_connection errors due to lingering sleep connections caused by mismatched timeout settings between LVS and HAProxy, and adjusting HAProxy's timeout resolved the issue and prevented further connection exhaustion.
This article explains various service degradation strategies—including interface and page refusal, delayed persistence, and persistent‑layer restrictions—along with management approaches and implementation points such as middleware control, NGINX+LUA page blocking, and data‑operation rules, to keep core functions running under high server pressure.
This article summarizes Alipay’s evolution into a super‑app, detailing its multi‑stage architecture, performance and power optimizations, stability improvements, and the comprehensive operations system that monitors and mitigates issues across millions of users.
This article explains why servers often run far below their theoretical capacity, defines the concept of highest usable resource utilization, and offers practical and advanced techniques—such as multithreading, workload consolidation, resource layering, and overselling—to improve utilization while weighing performance, cost, and reliability impacts.
The article explains DevOps as the integration of development and operations, tracing its origins, outlining its cultural and technical challenges, detailing its benefits such as faster, more reliable releases, and reviewing the tools and global adoption trends, including a new Chinese survey initiative.
This article explains Nginx's role as a high‑performance HTTP and reverse‑proxy server, its event‑driven design, key features, internal process model, request handling flow, and real‑world deployments, providing a comprehensive overview for developers and operations engineers.
This talk details Ctrip's application operations framework, covering data‑center scale, multi‑application deployment on Windows, high availability goals, capacity‑prediction models, disaster‑recovery design, incident response, and the evolution from manual tooling to automated, intelligent operations.
This article curates five technical reads—covering network operations, Google’s production environment, massive cost‑saving strategies, IDC automation, and Docker‑based RDS—each presented as a “medicine” with a brief description and a link for deeper insight.
This guide outlines a step‑by‑step strategy for designing a highly available, secure, and scalable website architecture, covering domain acquisition, CDN deployment, image caching, data center selection, monitoring, DDoS mitigation, redundancy, server configuration, database replication, testing environments, and operational best practices.
This article details Meituan's technical approach to handling massive traffic spikes during large‑scale promotions, covering background of the O2O deal platform, traffic‑model construction, capacity‑budget calculations, micro‑service architecture evolution, pressure‑test strategies, and the PTP performance‑testing environment used to validate system limits.
Twitter’s widgets.js, serving 300,000 requests per second, uses a carefully engineered deployment pipeline that emphasizes instant rollbacks, progressive rollouts, and real‑time visibility through DNS routing, CDN distribution, and origin management to minimize risk and ensure reliability.
This article explains why yum install commands return 404 errors on CentOS due to an undefined $releasever variable, analyzes the root cause in yum configuration, and provides a step‑by‑step solution and useful troubleshooting tips for operations engineers.
This article explains the concept of precise operations, detailing how integrating business demand as a variable can make IT maintenance more proactive, value‑driven, and synchronized with business needs, and outlines a step‑by‑step framework with real‑world examples.
When a Linux server’s load spikes, you can quickly pinpoint the root cause within a minute by running ten key commands that reveal CPU, memory, I/O, and kernel‑level metrics, enabling fast, data‑driven troubleshooting.
This article presents Hong Guangming's detailed overview of Ctrip's payment engine reconstruction, covering the legacy system's challenges, the new split‑service architecture, channel‑specific services, settlement reconciliation, and the multi‑layered strategies employed to achieve high availability and operational stability.
This article dissects WhatsApp’s high‑reliability architecture that supports 450 million users, detailing its Erlang‑based backend, hardware choices, scaling techniques, monitoring tools, and the engineering lessons learned from pushing a single server to two‑million concurrent connections.
This article summarizes Ouyang Chen's GDevOps 2016 talk, covering the definition of DevOps, four personal viewpoints, Microsoft's three‑phase transformation, Xiaomi's rapid release pipeline, key principles, metrics such as time‑to‑detect, and essential tools for building an efficient DevOps culture.
Over the past two years WhatsApp has dramatically expanded its user base, hardware, and traffic while maintaining a tiny engineering team, highlighting the challenges of massive scalability, Erlang‑based distributed design, Mnesia database bottlenecks, decoupling strategies, and operational patches required to keep the service reliable.
The article explains why a Huawei AR2240 gateway using PPPoE on an on‑demand dial‑bundle fails to obtain an IP address, identifies the misconfiguration of the internal Ethernet interface, and provides step‑by‑step commands to switch the PPPoE mode to permanent online for successful dialing.
This article explores how historical monitoring alerts can be analyzed and predicted using linear regression, guiding operations engineers to preprocess data, build regression models, and forecast future alert trends to reduce manual alarm handling and improve system stability.
This article explains how to extract valuable performance and traffic insights from Nginx access logs using shell commands and awk, covering request volume, peak rates, bandwidth usage, slow‑query detection, URL normalization, and practical optimization recommendations for web operations.
This article, excerpted from the upcoming Chinese edition of “SRE: Google Site Reliability Engineering”, examines how Google’s SRE guiding philosophies—disaster planning, post‑mortem culture, automation, and data‑driven decision‑making—are adopted, adapted, or contrasted in sectors such as manufacturing, aerospace, nuclear, telecommunications, healthcare, and finance, highlighting key similarities, differences, and lessons for Google and the broader tech industry.
The article details Dangdang's 15‑year evolution from a monolithic platform to a distributed, SOA‑based architecture, describing system tiering, front‑end and back‑end scaling techniques, asynchronous processing, data‑flow optimization, and operational practices that enable stable handling of ten‑fold traffic spikes during major sales events.
This article walks readers through practical techniques for identifying and resolving common “unavailable” app problems—such as white screens, slow loading, or error codes—by using the lightweight Ping&DNS Android tool to check domain names, IP connectivity, DNS resolution, and traceroute data, empowering both novice users and professionals.
The article explains when MySQL sharding is needed, outlines five practical principles for deciding to split databases or tables, provides real‑world examples, and shares operational tips for implementing horizontal partitioning to improve performance, availability, and manageability.
The article examines agentless backup technology, comparing its implementation in virtualized and physical environments, detailing supported interfaces, evaluating a real‑world Asigra Cloud Backup case, and discussing security risks, performance impacts, and when traditional agents remain necessary.
This guide demonstrates how to simulate a high‑traffic scenario between two machines and use an iptables rule to reject connections from a specific host when its concurrent requests exceed ten, including command syntax, execution steps, and result analysis.
An automated database operations platform shifts routine DBA tasks to developer self‑service and fully automates processes such as cluster provisioning, scaling, backup, migration, and sharding, using a stateless workflow center, job queue, and unified SqlEditor to improve efficiency, safety, and auditability.
Tencent's SNG team shares six practical capacity‑management techniques—performance, density, feature, fragmentation, barrel, and hardware selection methods—that helped reduce operational expenses by over a hundred million yuan annually while supporting hundreds of millions of daily active users.
This presentation details Ctrip's payment engine reconstruction, describing the limitations of the legacy system, the new service‑oriented architecture separating payment planning and execution, channel‑specific services, the reconciliation system, and the operational measures taken to achieve high payment availability.
The article describes how Ctrip’s network security team built an automated, centralized firewall management platform that handles multi‑brand firewalls, streamlines policy queries, generation, and deployment, integrates with change‑ticket workflows, and dramatically improves operational efficiency while reducing human error.
This article details Tencent's practical approaches to reducing user‑perceived latency in mobile QQ and Qzone by analyzing server, network, and client delays, employing private protocols, multi‑path connection strategies, real‑time monitoring, and big‑data clustering to identify and fix performance bottlenecks.
This article explains the comprehensive concept of high‑availability systems, covering redundancy, failover, consistency challenges, various technical solutions, SLA definitions, and the organizational and engineering practices required to achieve multiple “9s” of availability.
This article examines common causes of Redis data loss, walks through a real‑world incident where 90,000 keys vanished, and provides concrete monitoring, configuration, and operational safeguards to detect and avoid such failures.
This article details Ant Financial's evolution from supporting 20,000 transactions per minute in 2010 to 86,000 transactions per second in 2015, describing its multi‑active architecture, financial‑grade operation platform, and organizational mechanisms that enable high‑availability, automated capacity management and fault handling in a cloud‑native environment.
This article explains how to build a comprehensive mobile app quality monitoring framework by defining quality standards, setting capability indicators, and leveraging data acquisition, analysis, and visualization to continuously improve product reliability and user experience across different development stages.
This article details a MySQL high‑availability solution built on Pacemaker, Corosync and MHA, explains why earlier keepalived‑based designs suffered split‑brain issues, and walks through the architecture, quorum handling, resource agents, failover workflow, testing methodology, and practical lessons learned.
This guide walks you through setting up a Docker Swarm cluster on three Ubuntu 16.04 servers, covering environment preparation, Docker installation, configuring a Consul discovery backend, creating manager and worker nodes, and managing containers across the swarm using the Docker remote API.
This article outlines the evolution of enterprise operations through four stages, identifies seven common operational pain points, and presents a comprehensive model for building an automated operations platform that integrates design, deployment, monitoring, optimization, and troubleshooting.
The article describes how Alibaba's game integration platform achieved business‑oriented high availability by abandoning traditional system‑centric designs and implementing a three‑dimensional architecture that combines clear HA goals, multi‑active deployment, client‑side retries, functional isolation, automated monitoring, and rapid fault recovery, ultimately meeting a 3‑minute issue‑location and 5‑minute business‑recovery target.
This article explains the design and implementation of Harbor's policy‑based Docker image replication, detailing its architecture, job service workflow, state‑machine handling, and how it reduces storage‑specific dependencies while simplifying large‑scale container registry synchronization.
This article provides a detailed walkthrough of MHA's core configuration parameters—including server scopes, connection settings, candidate master rules, failover scripts, and monitoring options—explaining where each setting belongs and how to fine‑tune MySQL high‑availability behavior.
This article details how Tencent's operations team orchestrated a seamless, zero‑impact migration of over 70 million users across three data centers during the 2015 Tianjin explosion, highlighting the four key capabilities—distribution, scheduling, data synchronization, and automated operations—that enabled multi‑active disaster recovery at massive scale.
This article explains how Alibaba’s NineGame platform achieved ultra‑high availability by shifting from system‑centric to business‑centric design, defining measurable goals (3‑minute issue detection, 5‑minute recovery, bi‑monthly incidents) and implementing a layered, automated, visual monitoring, client‑side retry, HTTP‑DNS, functional isolation, and multi‑site active‑active architecture.
This article outlines the importance of store data in O2O products and presents a comprehensive workflow—including single‑attribute rules, multi‑attribute cross‑validation, and auxiliary checks—to detect and remediate low‑quality or erroneous store information, thereby improving user experience.
This article explores practical service‑degradation techniques—including automatic and manual switches, read/write fallback, and multi‑level strategies—to ensure core functionality remains available during traffic spikes, failures, or resource constraints in high‑concurrency systems for.
This guide walks Java developers through practical performance analysis using the Hprof agent and pidstat tool, demonstrates step‑by‑step command usage with real code examples, and presents a memory‑leak case study that explains GC overhead limits, dump analysis, and concrete remediation steps.
Zhang Yue, a Qunar operations engineer, discusses the design, selection, architecture, scalability challenges, visualization, alert strategies, and future plans of the company's in‑house monitoring platform Watcher, highlighting lessons learned from migrating from Cacti to a Graphite‑based, Grafana‑enhanced solution.
This article recounts a seven‑year‑old Oracle 10g database migration, detailing project background, team turmoil, topology redesign, security constraints, data‑sync strategies, custom tools, high‑fidelity testing, unexpected failures, and the lessons learned for reliable operations.
The article explains how traditional centralized backup indexes become performance bottlenecks as data grows, and details Simpana's two‑level distributed indexing architecture—primary and secondary indexes—showing how it enhances backup speed, reduces network load, and simplifies recovery across multi‑site environments.
This article details Alibaba Games' journey to achieve four‑nine reliability through a business‑focused high‑availability architecture, including system analysis, a four‑layer design, HTTP‑DNS client retry, service decoupling, multi‑active deployment, comprehensive monitoring, and measurable operational goals.
The article presents a business‑oriented, three‑layer high‑availability architecture for a large‑scale game access platform, detailing measurable goals, client‑side retry with HTTP‑DNS, functional separation and degradation, multi‑region active‑active deployment, and automated, visual monitoring to achieve rapid fault detection, isolation, and recovery.
Leveraging Tencent's extensive experience in social platform operations, this talk explores intelligent monitoring practices—covering active, passive, and side‑channel techniques, full‑link observability, data processing pipelines, and alert convergence—to enhance reliability, availability, and user experience while reducing noise for ops teams.
This article details how a Tencent game operations team reduced a major online game's scheduled maintenance window from 1.5 hours to just 0.3 hours by redesigning the checklist, separating pre‑ and post‑maintenance tasks, and switching to a rename‑based update method across thousands of servers.
The article outlines a comprehensive O2O data quality assurance workflow for online movie seat selection, detailing background challenges, a three‑stage process, evaluation metrics, and a concrete case study that demonstrates how real‑time data monitoring and issue handling improve user experience.
This article shares real‑world ops stories about a disputed memory upgrade, explains how Linux calculates usable memory, clarifies common subnet‑mask misunderstandings, and demonstrates why Telnet cannot test UDP ports, highlighting practical troubleshooting lessons for system administrators.
The article explains Jenkins' single‑node and master‑slave deployment models, outlines the scalability and resource challenges of traditional setups, and proposes replacing test machines with Docker containers managed by Kubernetes or Swarm to improve efficiency, maintainability, and resource utilization.
This guide explains how to simplify and accelerate Tomcat web application migration by separating static binaries from external configurations, using Docker containers or Juju packages, applying layered configuration, managing persistent data with volumes, and automating deployment, scaling, and rollback operations.
This article guides operations engineers through tcpdump fundamentals, advanced filtering techniques, the design of an automated packet‑capture tool, and how to transform captured traffic data into a visual business architecture tree for efficient fault isolation and resource optimization.
This article walks through a systematic, multi‑stage approach to identifying, reproducing, and fixing server‑side problems using Linux tools such as strace, lsof, and netstat, illustrated with real‑world case studies and practical command examples.
This guide shows ops engineers how to shift from routine maintenance to DevOps expertise by adopting the right mindset, mastering open‑source community resources, contributing code, and understanding design patterns, concurrency, modularity, data structures, algorithms, and system calls.
This article details the technical background and evolution of the Duosuo English learning app, covering initial architecture, bandwidth estimation, risk control, database sharding, code refactoring, and operational lessons learned over four months of scaling.
Service overload caused by cache failures can cripple dependent systems, but by adopting smart cache get patterns, proactive client‑side checks, traffic throttling, service degradation, and dynamic scaling, developers can both prevent overload and recover gracefully when it occurs.
The article explains how modern call centers evolve to multi‑site active‑active deployments, detailing the network design, disaster‑recovery mechanisms, login and monitoring logic, key features supporting automatic failover for thousands of agents, and future expansion possibilities.
The article examines JD.com's strategic focus on user experience—covering pricing, logistics, service, and product quality—and explains how its integrated systems and "no‑no" policy drive operational efficiency and competitive advantage in China's e‑commerce market.
This article explains how large‑scale internet companies can reduce alarm storms and speed up incident resolution by creating an operations ecosystem centered on automated fault root‑cause localization, detailing the challenges, architecture, decision‑tree algorithms, and a four‑step implementation guide.
This article presents a curated list of 18 lesser‑known command‑line utilities for Linux/Unix administrators, explaining their purpose, typical usage scenarios, and how they help monitor system resources, network activity, and security events.
A deep dive into a long‑standing PHP‑CGI deadlock that left deleted log files occupying disk space, explaining how signal‑unsafe functions caused the lock, how the issue was diagnosed with lsof, strace and gdb, and the practical steps to eliminate the deadlock.
This article explores how large‑scale internet operations can achieve order and efficiency by applying entropy theory, standardizing configuration and monitoring, adopting automated deployment practices, and leveraging open‑source tools like Open‑Falcon to build a fully automated, resilient infrastructure.
A tiny typo in an ifconfig command set all IPs to 0.0.0.0, causing an Oracle RAC 10.2.0.4 cluster on Solaris 10 to collapse instantly, illustrating the critical need for meticulous command‑level precision in system operations.
This article shares VIPShop's extensive production experience with Redis Cluster, covering use cases, storage architecture evolution, detailed best‑practice guidelines, common pitfalls, operational automation, monitoring strategies, and useful open‑source tools for large‑scale deployments.
This article explains the DevOps toolchain by breaking it into three core components—SCM, automation, and cloud—detailing their roles, typical tools, and how they interoperate to enable continuous delivery and scalable, self‑service infrastructure.
Twitter keeps its platform stable during massive traffic spikes by regularly performing large‑scale stress and extreme tests, analyzing performance metrics, and maintaining detailed contingency plans that guide rapid response to unexpected events such as the record‑breaking “Sky City” incident.
This article presents a curated list of twelve powerful command-line utilities—such as lsof, htop, iotop, IPTraf, Monit, netHogs, iftop, and Monitorix—that Linux system administrators can use to monitor, diagnose, and optimize system and network performance.
This article explains what strace is, how it works, and provides step‑by‑step examples—including fixing a failed service start, tracing nginx, diagnosing process crashes, shared‑memory errors, and performance analysis—to help operations engineers quickly locate and resolve Linux system problems.
This article recounts two real‑world operations incidents—a network outage caused by an improperly configured portfast on a trunk link and an NFS failure that crippled an API service—then distills practical lessons on pre‑incident procedures, monitoring, fault handling, recovery, and post‑mortem practices.
This guide explains how to use the Linux locate and find commands, compares their speed and database usage, details common options, pattern syntax, file‑type and size filters, time‑based searches, permission checks, and shows practical examples of combining conditions and actions.
This article explains a practical architecture for aggregating logs from distributed Docker containers by employing Flume NG as a lightweight log collector, Kafka as a high‑throughput message bus, and custom sinks to store logs per service, module and day with low latency and minimal resource impact.
This article shares the author's experience and lessons learned while building a log‑monitoring framework with the ELK stack, discussing performance issues, configuration of Logstash filters using Grok, and practical tips for deploying ElasticSearch, Logstash, and Kibana in production environments.
This article shares Peng Lihang's practical insights on operations automation, covering the essential trio of configuration, state, and change management, the evolution of ops practices, platform design principles, and concrete steps for building scalable, business‑driven ops platforms.
This article outlines ten practical guidelines for operations engineers—ranging from mandatory rollback testing and cautious handling of destructive commands to robust backup verification, vigilant monitoring, and disciplined handover procedures—to dramatically reduce system outages and improve overall reliability.
This article shares eBay’s seven core scalability best practices—including functional partitioning, horizontal sharding, avoiding distributed transactions, asynchronous decoupling, stream processing, virtualization, and smart caching—to help architects design highly available, cost‑effective systems that can handle billions of daily requests.
The article analyzes distributed storage replication methods, explains how large‑scale replica recovery can trigger an avalanche effect, and proposes operational safeguards such as cross‑rack replica selection, flow‑control mechanisms, predictive fault handling, and a safety mode to maintain system stability.
A misconfigured mount_maxsize limited each Data Server to 20 GB, causing 96% storage usage, and after correction led to block corruption that required a custom script to clean up, illustrating the importance of proper storage settings and automated remediation in TFS operations.
The article examines Alibaba's Zeus resource scheduling platform, detailing its background, problem analysis, container‑based virtualization, distributed architecture, strategies for improving resource utilization such as overselling and hybrid deployment, as well as stability measures and automation for large‑scale operations.
This article explains the Mesos distributed system kernel, its master‑slave architecture, fine‑grained resource scheduling, and how Qunar leverages Mesos and Marathon for log processing, Spark, Alluxio, and multi‑tenant services while addressing framework unification, HA, service discovery, and operational challenges.
This article compiles 73 practical Linux shell tips covering network checks, process control, file manipulation, system monitoring, version control, and various command-line shortcuts, providing concise examples and commands to enhance productivity and troubleshooting for system administrators and developers.
This article explains why log analysis platforms are essential as systems grow, outlines the benefits of centralized logging, presents Ctrip’s real‑world requirements and challenges, and introduces the ELK stack as a scalable solution for collecting, storing, and visualizing massive log data.
This article chronicles Spotify's evolution in server operations—from a manual Ops team and ad‑hoc tools in the early years, through automated DNS, provisioning, and self‑service platforms, to a hybrid cloud strategy that reduced resource‑request turnaround from weeks to minutes.
This article chronicles Spotify's journey from a manual, fire‑fighting Ops team and the early ServerDb tool to automated DNS updates, provisioning systems like provcannon, Neep and Sid, and finally a cloud‑native migration using Google Cloud Platform, highlighting the challenges, solutions, and impact on resource delivery speed.
A collection of real‑world programming mishaps—from mixing test and production data to dangerous rm commands—illustrates why strict environment separation, cautious command execution, and disciplined code management are essential for reliable software operations.
An in‑depth overview of Tencent’s Blue Whale system reveals its positioning, design philosophy, PaaS and SaaS components, and how it enables scalable, unmanned operations across cloud and on‑premise environments, illustrating practical automation stages from scripting to intelligent orchestration.
The article explains why operators should start from clear goals rather than events, defines meaningful metrics such as user retention and API call volume, shows how to break down and evaluate these metrics, and offers practical advice on data collection, benchmarking, and continuous improvement.
This article examines how a large‑scale e‑commerce transaction platform achieved high availability through iterative architectural evolution—from early .NET monoliths to vertical and horizontal micro‑service splits—highlighting practical strategies for fault detection, rapid recovery, scaling, and operational best‑practices.
This article explains why Docker services need monitoring, distinguishes black‑box and white‑box approaches, compares self‑hosted and SaaS monitoring stacks, and reviews key components and popular tools such as Prometheus, InfluxDB, Grafana, Datadog, and Sysdig.
A comprehensive overview of common Linux daemon processes, detailing each service’s purpose, typical use cases, and key configuration notes for system administrators seeking to understand and manage background services effectively.
This article shares the author’s experience building fault‑tolerance for Tencent’s activity operations platform, covering retry strategies, automatic removal of unhealthy machines, timeout tuning, asynchronous processing, anti‑replay mechanisms, service degradation, service decoupling, and business‑level safeguards to reduce manual alarm handling and improve system robustness.
This article, based on the Efficient Operations Community Talk, outlines the evolution of e‑commerce systems, the challenges faced during rapid growth, and how LeTV’s e‑commerce cloud leverages micro‑service architecture, container technology, and hybrid cloud solutions to address scalability, security, and operational efficiency.
This article explains what deployment specifications are, outlines the key components of a good spec, shares a real-world CodeDeploy example, and provides practical steps for designing, building, and rolling out deployment standards that balance flexibility, non‑intrusiveness, and ease of use.
