This article details how Alibaba Group transformed its GitLab deployment from a single‑node bottleneck into a horizontally scalable, sharded architecture that handles millions of daily requests with high availability, improved performance, and robust data safety.
This article compiles practical experiences from system administrators about common operational pitfalls, their root causes, and concrete mitigation steps, ranging from misconfigured HAProxy timeouts and risky rm commands to ansible async quirks and cron‑job failures.
The article reflects on the challenges faced by operations engineers in small companies, illustrating why they often become scapegoats, and offers practical advice on learning, risk control, communication, and disaster‑recovery drills to increase their value and effectiveness.
This guide explains why server IO utilization spikes above 60% during early‑morning hours, covering hardware self‑test, RAID battery failures, cache policy misconfigurations, and step‑by‑step commands for MegaRAID and HP servers, plus BIOS adjustments and best‑practice recommendations to prevent performance degradation.
This article chronicles Alibaba's evolution of the full‑link pressure testing platform—from its 2013 inception tackling massive Double 11 traffic, through data construction, isolation, traffic generation, and platform upgrades—to a mature, automated, cloud‑native solution that safeguards large‑scale e‑commerce stability.
This article walks through a vehicle‑networking company's four‑stage journey—selection, requirement analysis, implementation, and evolution—to replace fragmented SSH passwords with a centralized OpenLDAP authentication platform, covering cost decisions, deployment steps, security hardening, and management automation.
The article explains gray release as a smooth, risk‑mitigating deployment method, outlines why it is needed, describes its limitations, and compares four practical gray‑release solutions—including code‑level flags, pre‑release machines, SET isolation, and dynamic routing—before recommending a combined approach.
Based on a survey of 300 IT professionals, this report reveals growing DevOps adoption, key motivations such as quality and cost reduction, major obstacles like resource shortages, measurable benefits including cost savings and faster releases, preferred tools, error‑handling practices, and future investment plans.
This article explains why distributed systems need tracing, outlines design goals, compares major implementations like Zipkin, EagleEye, and Hydra, and details the data collection, storage, and analysis pipelines that enable end‑to‑end visibility and performance optimization in large‑scale services.
The article reflects on eight years of operations experience, highlights the bright industry outlook, and outlines four key career paths—operations development, platform R&D, database engineering, and management—showing why skilled ops engineers are increasingly in demand.
This article traces the evolution of software engineering tools leading to DevOps, highlights common transformation pitfalls, outlines core DevOps practices such as autonomous small teams, traceable toolchains, real‑time metrics, and describes the surrounding ecosystem, offering practical guidance for organizations adopting DevOps.
This article presents a comprehensive server‑side monitoring solution covering functional and performance requirements, monitoring objects, design choices between self‑monitoring and centralized reporting, system architecture, API definitions, key challenges such as key collisions, data formats, storage options, and operational considerations.
This article explains the origins and background of DevOps, clarifies common misconceptions about its role and title, outlines its cultural principles, surveys the essential toolchain, and discusses how organizations can adopt DevOps practices beyond just development and operations.
This article explains how 360’s DoctorStarange system combines time‑series forecasting, neural‑network predictions, alarm correlation, and a machine‑health scoring model to reduce false alerts, automate remediation, and maximize resource utilization across thousands of production servers.
This article warns about ten powerful Linux commands that, when misused—especially with root privileges—can irreversibly delete data, corrupt disks, or crash the entire system, and offers practical safeguards to prevent accidental disasters.
The article reports on the SRECon17 Americas conference in San Francisco, summarizing keynote talks, panel sessions, and practical insights from industry leaders such as Stripe, Netflix, Google, and IBM on topics ranging from traffic control and container management to on‑call practices and cost considerations for Site Reliability Engineering.
The article examines how Tencent’s Blue Whale platform enables traditional operations teams to evolve into DevOps‑focused, cloud‑native units by automating release, change, and incident processes, integrating big‑data decision support, and delivering low‑cost SaaS tools for a wide range of internal stakeholders.
This article recounts how Tencent's operations team tackled the urgent migration of aging data‑center infrastructure to preserve service for 8 million legacy QQ users, detailing the challenges, strategic choices, IP‑level network relocation, and the DevOps practices that ensured a successful cut‑over.
The article outlines a typical six‑segment workday of a Tencent operations engineer, detailing how they review past results, tackle urgent issues, take breaks, perform routine tasks, debug scripts, and handle after‑hours responsibilities, offering practical insight for aspiring sysadmins.
This article explains the SAN Boot concept, its advantages and limitations, and provides detailed step‑by‑step instructions for configuring HBA cards and server BIOS (including legacy mode) to enable SAN Boot on Linux, HP‑UX, AIX, and Solaris systems.
This article explains the origins of DevOps, presents the CALMS and Three Ways frameworks, and organizes practical DevOps principles into four dimensions—people, product, process, and tools—illustrating how they collectively enable continuous, on‑demand delivery of business value.
The article shares frontline IT operations experiences, emphasizing careful command execution, mandatory operation logs, two‑person verification, and backup strategies to prevent disastrous errors, illustrated by real incidents like a massive Deutsche Bank loss caused by a simple input mistake.
This article explains how to use the Linux top command for real‑time system and process monitoring, covering its interface, statistical and process sections, interactive shortcuts, command‑line options, and internal commands to customize and sort the displayed information.
This article explains how to control work‑in‑progress using the lake‑water‑rock metaphor, outlines realistic and useful principles for setting WIP limits, describes common limiting methods such as swim‑lane caps, stage caps, and personal caps, and offers practical ways to determine initial limit values.
This article explains why the operations role exists in the software lifecycle, outlines the core responsibilities of an operations engineer—including process standardization, automation, monitoring, security, and technology adoption—and highlights the vital personal qualities needed for success.
The article explains how to control work‑in‑progress by postponing new starts and concentrating on finishing existing tasks, emphasizing that WIP should be measured in delivered user value rather than task count, and outlines practical control techniques for lean product development.
This article outlines the design and implementation of high‑concurrency systems, covering server architecture, load balancing, database clustering, caching strategies, message‑queue based asynchronous processing, static data handling, and operational best practices such as monitoring, redundancy, and automation.
This article walks readers through a three-step process for designing a team’s Kanban wall and system, teaching how to analyze value streams, select appropriate visual elements, and create a customized board that supports efficient workflow management.
This article outlines a comprehensive PDCA‑based methodology for e‑commerce platforms to proactively prevent issues, quickly detect anomalies, and execute rapid decisions during large‑scale promotions, covering system goal definition, performance evaluation, capacity planning, SLA management, and team/process maturity.
This article explores the challenges of operating Alibaba's large‑scale data platforms, describes the automation platform built to address them, and shares data‑driven, fine‑grained operational practices that enable stable, efficient, and cost‑effective service delivery.
Learn how to achieve near real‑time, time‑series monitoring of Spring Boot applications without writing code by combining Spring Boot Actuator, Jolokia, InfluxDB, Telegraf, and Grafana, while evaluating alternative tools like Prometheus, Graphite, and JMXTrans and understanding their limitations.
This article explores the evolution of log processing in the big‑data era, detailing how ELK Stack, Kafka, and Spark Streaming work together to provide scalable, real‑time log collection, analysis, and visualization for modern cloud‑native operations.
The article examines the conflict and convergence between ITIL and DevOps in modern operations, outlining DevOps principles, automation in deployment and operations, and three integration models that balance management and execution, while highlighting the distinct values and scenarios for each approach.
The addops team shares practical insights from technical exchange sessions with AutoHome and Sina, covering CMDB design, code release, containerization, monitoring, hybrid‑cloud architecture, virtualization, and machine‑learning applications in operations.
This article details Alibaba's mobile app operational practices, covering the challenges of client-side maintenance, their high‑frequency release pipeline, gray‑release mechanisms, monitoring, trace systems, remote logging, and rapid issue resolution to ensure stability and performance at massive scale.
This article examines YouTube's large‑scale architecture, detailing its platform components, web and video services, database evolution, data‑center strategy, and key lessons for building high‑concurrency, fault‑tolerant backend systems.
The article presents a comprehensive, application‑centric approach to automated capacity management that analyzes why server utilization is low, defines safe usage thresholds, describes a load‑balancer‑driven stress‑testing workflow with regression modeling, and explains how this practice improves resource efficiency, cost savings, and developer‑ops collaboration.
This article explains how mobile network characteristics, the one‑second rule, and targeted optimizations in access scheduling, protocols, and business logic can dramatically improve download success, startup speed, and overall user experience for mobile services.
This article explains how to automate application‑centric capacity assessment, identify the safe utilization thresholds, use load‑balancer‑driven stress testing and regression modeling to pinpoint resource bottlenecks, and improve server usage while maintaining service reliability through close DevOps collaboration.
A developer recounts how a Redis connection pool overflow across dozens of clusters was traced to a single misbehaving service, diagnosed with netstat and ps commands, and resolved by adjusting configuration and stopping the offending process, illustrating practical troubleshooting of connection limits.
This article explains how to automate capacity management focused on application performance, identifies common causes of low resource utilization, proposes safe utilization thresholds, describes a testing framework that uses load‑balancer weighting and real‑time monitoring to pinpoint bottlenecks, and outlines how ops and developers can collaborate to improve efficiency.
The article shares Tencent’s practical fault‑tolerance journey for a billion‑scale activity platform, covering retry strategies, automatic removal of faulty nodes, timeout tuning, business‑level safeguards, service degradation, and decoupling techniques that together reduce manual firefighting and improve system resilience.
The article recounts a GitLab production database deletion caused by a mistaken command, analyzes why the backup mechanisms failed, and offers technical and cultural recommendations—including automation, proper replication, and transparent post‑mortems—to build more reliable, high‑availability systems.
This article recounts the GitLab production database deletion incident, analyzes why backup mechanisms failed, shares technical and cultural lessons on operational practices, and offers concrete recommendations for building resilient, high‑availability systems to prevent data loss.
The GitLab production database was mistakenly deleted during a manual fix, exposing gaps in backup strategies, PostgreSQL configuration, and operational practices, and prompting a detailed post‑mortem that highlights the need for automated recovery, proper tooling, and transparent incident handling.
GitLab.com suffered a catastrophic database outage on February 1, 2017 when an exhausted operator mistakenly ran a destructive rm command on the wrong server, wiping most production data; the incident’s timeline, root causes, recovery steps, and lessons learned are detailed in this post‑mortem.
A GitLab.com database accident caused the loss of 310 GB of data, exposing multiple failed backup mechanisms and prompting a detailed analysis of technical, operational, and managerial lessons for reliable data protection.
This guide outlines the critical tasks DBA teams should perform before, during, and after the Chinese New Year holiday, including daily security practices, pre‑holiday inspections, on‑call rotations, post‑holiday reviews, and detailed checklist scripts to ensure database reliability and prevent incidents.
This article explains Puppet's architecture, data flow, and practical examples, showing how automated configuration management can efficiently handle large numbers of servers, reduce manual errors, and improve operational reliability in modern IT environments.
The 2016 Ops Alert Report reveals Zabbix’s dominance, preferred notification channels, monthly and daily alert trends, peak alert times, regional distribution, and quirky usage statistics, offering valuable insights for operations teams to optimize monitoring and incident response.
This article presents a comprehensive case study of a business monitoring system, covering its background, architectural analysis, module design, time‑series database selection, visualization with Grafana, alerting strategies, decision‑making logic, and intelligent monitoring experiments, followed by key takeaways and lessons learned.
This guide introduces Ansible for operations, covering its core features, installation, host preparation, key management, essential modules, playbook structure, YAML syntax, handlers, tags, variables, templates, loops, and conditional execution, with practical command examples and visual illustrations.
This article explains what GSLB (Global Server Load Balancing) is, why achieving high availability, low latency, and accurate traffic distribution is difficult due to DNS limitations, caching, and routing constraints, and explores architectural and network‑level techniques such as feedback loops, anycast, and BGP routing to mitigate these challenges.
The article explains the Third Way of DevOps—continuous learning through Kaizen and the PDSA cycle—showing how Docker serves as laboratory equipment that enables rapid, reproducible experiments, illustrated with examples from a financial institution and a personal baseball‑statistics project.
This article explains what high availability means for distributed systems, outlines common availability tiers, and describes how redundancy, load balancing, and automatic failover across a typical Internet architecture can achieve reliable, scalable services.
This article explains the DevOps “Second Way,” emphasizing faster, bidirectional feedback loops, and shows how Docker’s immutable containers, streamlined packaging, and embedded metadata reduce variation, accelerate defect detection, and shorten lead times in service delivery.
This article introduces the standout operations professionals featured by the High‑Efficiency Operations community in 2016, summarizing each expert’s background, key achievements, and a curated list of their most influential technical articles for readers seeking deep insights into modern ops practices.
This article shares a detailed case study of how a financial services team restructured European application operations, applied lean retrospectives, built a top‑down monitoring system, and introduced systematic stakeholder collaboration to dramatically improve incident response, system robustness, and user satisfaction.
This article details Dangdang's 15‑year evolution from a monolithic system to a distributed, SOA‑based architecture, outlining the challenges of high‑traffic e‑commerce events and the strategies—system grading, decoupling, asynchronous processing, batching, and rate limiting—used to achieve reliable, scalable operations.
Facing massive user growth, Tencent’s social network team redesigned its KV storage architecture—introducing CKV and Grocery, automating capacity planning, data migration, and backup reuse—to dramatically lower costs, improve operational efficiency, and maintain high service quality across millions of devices.
Alibaba’s SunFire platform delivers massive‑scale, real‑time log collection, processing, and visualization for e‑commerce spikes like Double 11, using low‑overhead agents, asynchronous Map/Reduce pipelines, fault‑tolerant task scheduling, and shared inputs to ensure accurate, low‑latency monitoring across billions of transactions.
Learn how to assess your Continuous Delivery maturity using a 47‑item checklist, understand its purpose for aligning goals, improving processes, and boosting value delivery, and calculate your score as a percentage to guide technical and organizational improvements.
This article explains how large internet firms evolve their IT operations from reactive fire‑fighting teams to standardized, model‑driven, automated platforms covering planning, building, management, monitoring, and process‑oriented operations across compute, storage, and network resources.
This article reviews sixteen notable 2016 service outages across finance, cloud, and entertainment, analyzes their causes—ranging from power failures to DDoS attacks—and highlights the critical need for robust disaster‑recovery and information‑security practices.
This article explains the principles of DevOps, its technical, process, and organizational considerations, and introduces Docker as a key tool, detailing its architecture, components, native utilities, suitable scenarios, and how it enables continuous integration, delivery, and efficient operations.
This article explains how 360's Qcmd, a Golang‑based real‑time command execution system, overcomes SaltStack's limitations to reliably manage tens of thousands of servers with high success rates, flexible scripting, detailed monitoring, and efficient message handling.
DevOps is a methodology that emphasizes collaboration between development and operations to accelerate software delivery, and while tools alone don’t constitute DevOps, automation and container technologies are essential prerequisites that reduce manual hand‑offs, enable self‑service, and improve feedback loops.
From its origins at the 2008 Agile conference to the modern CALMS framework, this article traces DevOps’s evolution, compares traditional, DevOps 1.0 and 2.0 approaches, and outlines key Chinese practices such as containers, continuous deployment, micro‑services, and enterprise best‑practice recommendations.
This article introduces CAT, an open‑source Java‑based distributed real‑time monitoring platform, detailing its design goals, architecture, message processing pipeline, logging instrumentation, API, real‑time analysis, report modeling, storage challenges, and key takeaways for building highly available, scalable monitoring solutions.
This article outlines Alibaba researcher Zhang Ming’s presentation on the network automation system that enables Alibaba’s infrastructure to handle the massive traffic and rapid fault recovery required during the Double 11 shopping festival, highlighting the challenges, detection methods, and automated tools used across routers, switches, and L4‑L7 devices.
This article explains how to design and implement a comprehensive operations monitoring platform using Ganglia for data collection and Centreon for alerting, detailing a six‑layer architecture, integration steps, data flow, and practical Q&A for effective fault detection and response.
This article recounts Mogujie's journey from a small PHP‑based LNMP stack to a Java‑driven micro‑service architecture, detailing the operational challenges, standardization efforts, continuous integration pipeline, and full‑link tracing techniques that enabled scalable, reliable e‑commerce services.
This article details Ctrip's evolution from a single‑site call‑center to a fully dual‑active, multi‑region architecture, covering the overall system design, public network, application, and client layers, unified login mechanisms, heartbeat monitoring, and future software‑only and mobile‑first directions.
The article shares practical strategies for preventing service failures by doubting third‑party services, protecting against misuse by consumers, and improving one’s own code and architecture, covering fallback plans, timeout settings, retry policies, API design, traffic control, and resource limits.
This article compiles expert Q&A on operations, covering tool selection, monitoring, Linux version choices, automation platforms, security, Docker, backup strategies, and career advice, offering practical guidance for modern infrastructure management.
The article explores personal heroism in operations, defining it as reliance on individual effort to keep flawed systems appearing normal, examines its short‑term benefits and long‑term drawbacks for companies, teams, and the heroes themselves, and offers practical strategies to eliminate this risky mindset.
This guide walks through an automated, fast‑track deployment of DCOS components—including service selection, Docker‑based containers, host initialization, system checks, Ansible provisioning, Consul service discovery, HAProxy load balancing, MySQL HA, and Zookeeper/Marathon integration—providing concrete commands, configuration snippets, and practical tips.
This article shares Tencent's seven practical bandwidth‑saving techniques—ranging from disabling auto‑play to intelligent pre‑push, file compression, on‑demand usage, segmented download, technical breakthroughs, and content compliance—to dramatically reduce operational costs while maintaining user experience.
The 2016 Global Agile Operations Summit in Shanghai concluded with a series of expert sessions covering agile DevOps trends, cloud‑native automation platforms, database performance tuning, container orchestration, and real‑world case studies from leading companies, followed by the award ceremony honoring ten MVPs who drove innovation across operations and infrastructure.
This article describes Ctrip's large‑scale predictive outbound call platform, covering its underlying algorithms, SoftPBX integration, system architecture, concurrency enhancements, deployment experience, and measurable improvements in call success rates and agent efficiency.
Using the classic tale of Bian Que, this article explains how proactive, mid‑stage, and reactive risk controls in IT operations prevent small issues from becoming catastrophic failures, illustrated with real‑world storage, cloud, and equipment‑selection case studies.
Deploying the Transwarp Manager alert system within Hadoop clusters enables operators to monitor resource shortages, failures, and health issues in real time, offering browsing, configurable thresholds, and instant email or script notifications to quickly identify and resolve problems before they impact services.
In this interview, Lu Pengcheng, a platform architect at Mogu Street, discusses the company’s large‑scale e‑commerce architecture, the evolution of its monitoring platform, design choices for high‑availability distributed systems, and future open‑source plans, providing practical insights for engineers and technical managers.
This article explains the definitions of service, SLI, SLO, and SLA, outlines how to choose and measure appropriate indicators, shares best practices for setting and improving SLOs, and shows how SLAs combine objectives with consequences to manage service reliability.
This article outlines four frequent Node.js operational problems—memory leaks, CPU bottlenecks, back‑pressure, and security risks—and provides practical solutions such as heap‑dump analysis, CPU profiling, APM monitoring, and using private npm registries with tools like Snyk to secure dependencies.
This article walks through a real‑world remote debugging session where a high‑CPU issue in a gateway service was reproduced, analyzed with top, gstack, gcore, strace and gdb, and traced to a buffer overflow causing an infinite loop, then fixed.
This article outlines a systematic approach to onboarding and training new Site Reliability Engineers, covering trust building, readiness assessment, diverse learning methods, structured curricula, on‑call milestones, project‑focused work, reverse‑engineering skills, statistical thinking, and improvisation techniques to develop high‑performing SRE teams.
This article shares a practical template for optimizing web service performance, covering kernel sysctl tweaks, file‑handle limits, and Nginx/PHP‑FPM configuration adjustments to eliminate bottlenecks and maximize throughput under high traffic conditions.
This article details Tencent's multi‑region disaster‑recovery architecture and rapid, user‑transparent scheduling techniques that enable seamless service continuity for QQ and Qzone across billions of daily users, illustrated through real‑world drills and performance metrics.
This article explains how internet startups can build a robust logging system by defining log levels, essential log fields, best‑practice logging principles, and choosing between simple file logs or an ELK‑based collection pipeline for monitoring, troubleshooting, and analytics.
This guide shows how to write Bash functions that retrieve process IDs, CPU, memory, file‑descriptor usage, port status, system load and disk space on a Linux server, and how to combine them with conditional checks to generate alerts when thresholds are exceeded.
The article introduces JEN, JD's extended Nginx platform that centralizes configuration, monitoring, traffic splitting, rate limiting and automated operations through a web console and Ansible integration, addressing the complexity, restart requirements, and scaling challenges of large‑scale Nginx deployments.
This article examines the evolution of Taobao's technical architecture—from a LAMP stack through Oracle‑based mainframes to a cloud‑native platform—highlighting the performance, scalability, and cost challenges of traditional IT and offering best‑practice strategies for migrating enterprise systems to the cloud.
The article explains how unnoticed database issues often cause system slowness, outlines key diagnostic questions for operations teams, and presents a three‑step approach—discover, solve, prevent—to regularly health‑check and optimize databases for reliable performance.
This article outlines the challenges of modern software delivery in a VUCA environment and presents a practical, step‑by‑step approach to designing a visual continuous‑delivery pipeline that balances speed, quality, and reliability through agile, lean, and DevOps practices.
This article explains Google’s SRE postmortem philosophy, the criteria for writing postmortems, best practices for a blame‑free culture, and how collaborative knowledge‑sharing and incentives improve incident handling and overall system reliability.
The article explains the importance of configuration in software, distinguishes static and dynamic configuration, discusses the challenges of managing configuration in large distributed systems, and describes the evolution, design principles, and practical solutions of configuration centers such as Alibaba's Diamond.
This article explains how the internally built Wonder monitoring system, based on Open‑Falcon, tackles large‑scale operational challenges by offering automated agent updates, customizable metrics, log and port monitoring, persistent alarm storage, enhanced alert content, and comprehensive dashboards for thousands of devices.
This article details Shanda Games' journey in designing and implementing a comprehensive automated operations platform—including installation, deployment, security, client and server updates, data analysis, backup, and monitoring—to efficiently manage hundreds of games across diverse hardware and operating systems.
This article explores how Google Site Reliability Engineers manage service reliability by balancing risk, cost, and business goals, using metrics like unplanned downtime, availability formulas, and risk tolerance to set realistic SLOs for both consumer and infrastructure services.
