In this detailed account of a 2018 Linux operations interview, the author shares the job description, required skills, practical preparation tips, and concise answers to seven common interview questions, offering valuable insights for aspiring sysadmin and DevOps professionals.
Tencent's SNG social platform team tackles billion‑scale traffic by integrating AI‑driven anomaly detection, multi‑dimensional monitoring, and decision‑tree based root‑cause analysis, turning complex backend architectures and massive alert volumes into streamlined, actionable insights for faster issue resolution.
After a sudden incident deleted files from /data/webapps/xxxx/upload without backup, this guide walks through installing extundelete, locating the deleted inode data, and using extundelete commands to recover as much of the lost data as possible, including tips on mounting read‑only.
Alibaba senior technologist Houyi explains how the company used AI‑powered network intelligence, automated fault detection, traffic scheduling, and smart routing to dramatically improve stability, reduce costs, and boost efficiency during the massive Double 11 shopping event.
On December 30, 2017, a sudden wave of users uploading and downloading their 18‑year‑old photos caused QQ Space's album service to experience a four‑times spike in download traffic and a twelve‑times surge in post activity, prompting the operations and development teams to employ capacity monitoring, elastic scaling, flexible architecture, and targeted optimizations to maintain service stability and user experience.
The article compiles thirteen post‑mortem case studies of severe system outages—from AIX NTP misconfiguration and backup appliance driver issues to PowerHA node ID conflicts and hardware failures—detailing symptoms, root‑cause analysis, and practical remediation steps for each incident.
This article compiles Meituan's Linux operations engineer interview questions covering job requirements, core responsibilities, essential qualifications, and detailed answers on software installation, networking tools, IP configuration, scripting, iptables, MySQL security, replication, and common sysadmin commands, providing a comprehensive study guide for aspiring Linux ops candidates.
This guide explains how to create and deploy a Bash script that retrieves a WeChat corporate token and sends Zabbix alarm messages to all users through the WeChat API, enabling automated monitoring alerts via the corporate WeChat platform.
This article summarizes Wu Xiaoguang's talk at Gdevops 2017, detailing how Vipshop integrates data processing, analysis, and mining technologies—such as Flume, Kafka, Spark, and custom scheduling—to improve operational decision‑making, performance monitoring, root‑cause analysis, and predictive modeling across its e‑commerce platform.
This article compiles Meituan's Linux operations engineer interview requirements, common questions on system installation, networking, scripting, MySQL security, replication, iptables, and provides detailed command-line solutions and sample scripts to help candidates prepare effectively.
This comprehensive guide walks through the full lifecycle of enterprise system architecture, covering access‑chain analysis, network and hardware foundations, virtualization and container strategies, layered design, load‑balancing, database high‑availability, service segmentation, and operational safeguards such as CMDB, monitoring, and disaster‑recovery.
The article describes how a ticketing system with strict SLA enforcement, automated routing, and team‑based service management enabled rapid, orderly issue handling during the high‑volume Double 11 shopping event, achieving near‑90% resolution within 30 minutes and improving overall business stability.
This article presents a comprehensive bilingual DevOps glossary compiled from the DevOps Handbook, offering standardized English‑Chinese terminology, a change log, and open‑source contribution instructions via GitHub for continuous improvement.
This memoir chronicles a Chinese IT professional’s two‑decade evolution from a university student and Oracle DBA to a DevOps and cloud operations leader, sharing career milestones, technical choices, and practical insights for anyone pursuing a long‑term operations career.
This article, based on the Chinese translation of the DevOps Handbook, explains the Flow Principle, continuous delivery practices, value‑stream mapping, waste elimination, and Goldratt’s five‑step method for improving DevOps pipelines and achieving low‑risk, fast releases.
This article surveys the evolution of operations technology, covering microservices, SRE, DevOps, containerization, orchestration, automation, intelligent monitoring, infrastructure, database and big‑data ops, as well as security, game and fintech operational challenges, highlighting current trends and future directions for 2017.
This article explains the purpose of performance (stress) testing, compares four testing approaches, details the chosen proportional‑deployment strategy, and provides comprehensive preparation steps, script guidelines, metric analysis, and practical tips for ensuring system stability during large‑scale traffic spikes.
Learn a systematic approach for Linux system administrators to diagnose and resolve six typical problems—including filesystem errors, 'argument list too long', inode exhaustion, undeleted file space, too many open files, and read‑only filesystem—using command‑line tools, log analysis, and practical fixes.
The 2017 Double 11 stability monitoring project introduced a four‑layer monitoring architecture—including customer & sentiment, business, system water‑level, and infrastructure monitoring—along with data archiving and system‑level reliability measures to detect, respond to, and mitigate issues far faster than traditional manual processes.
The article describes how a company replaced costly commercial load balancers with an open‑source architecture based on Nginx for layer‑4 traffic and a layer‑7 cluster, detailing project background, technology selection, redundant design, network and Nginx configurations, operational scripts, performance testing, and data analysis.
This article explains how high‑availability (HA) clusters provide redundancy for directors, RS‑servers, databases and storage, covering active‑passive node roles, resource stickiness, constraints, quorum voting, split‑brain avoidance, failure detection methods, and essential configuration tips.
This article introduces Meitu’s self‑developed log collection system Arachnia, explaining why a custom solution was needed for massive server‑side user‑behavior logs, the key requirements such as reliability and real‑time throughput, and the core architectural mechanisms that address those challenges.
This article describes how WiFi 万能钥匙 designed and implemented the Roma monitoring platform to handle billions of daily requests, covering background challenges, architectural principles, component design, data collection, transmission, storage, alerting, and future directions for large‑scale observability.
This case study explains how a combination of kernel‑level NVMe driver congestion control, LVM adjustments, and SSD over‑provisioning was used to suppress severe IO bandwidth drops and jitter, ensuring smooth transaction processing for a high‑traffic Double 11 event.
The article describes Alibaba's end‑to‑end automated fault recovery system for its massive network, covering extensive data collection, Spark‑based event processing, flexible alerting with Siddhi, alert convergence using PageRank, and scripted recovery actions to achieve high availability during the Double Eleven traffic surge.
This article explains how Alibaba’s StarOps platform integrates DevOps automation, CMDB, release management, monitoring, host operations, bastion security and fault handling to enable large‑scale, unmanned, data‑driven operations across hybrid cloud environments.
Drawing on 40 years of IT experience, the speaker explores DevOps as a transformative practice, discusses its strategic business value, outlines four key discussion areas—including principles, practices, selling to executives, and identifying weak points—and offers practical guidance for cultural and organizational change.
This guide explains why many Linux distributions start unused services at boot, shows how to list and inspect them with systemd tools, and provides step‑by‑step methods to safely mask or disable specific services to speed up system startup.
The article explains the role, responsibilities, and career benefits of integration validation (solution testing) engineers at Huawei, offering practical advice and personal perspectives for newcomers transitioning from development to testing positions.
This article explores RabbitMQ’s core features—including message acknowledgment, prefetch count, RPC support, virtual hosts, and its powerful plugin system—explaining how each works, when to enable or disable them, and providing step‑by‑step command‑line examples for configuring users, permissions, and management tools.
This article explains how Alibaba’s Sunfire monitoring platform processes terabytes of logs per minute, uses a pull‑based architecture with Brain‑Reduce‑Map roles, tackles scalability and reliability challenges, and outlines future directions such as MQL standardization and intelligent baselines.
The article explores common conflicts between development, testing, and operations staff, explains why operations are often undervalued, and offers practical steps—such as clear documentation, defined processes, and proactive communication—to improve teamwork and reduce blame‑shifting in software projects.
At the 2017 Global Operations Conference, Tencent Cloud’s award‑winning Developer Lab—an immersive, browser‑based IDE integrating SSH, RDP and tutorial‑driven workflows with automated resource scheduling—and its Distributed Cloud Database (DCDB), a sharded, cluster‑managed MySQL‑compatible system featuring advanced scheduling, routing and configuration services, were recognized for innovation and influence.
Facebook’s journey from early startup ops to supporting over 2 billion monthly users reveals how disaggregated network, storage, and warm‑storage‑enabled Spark architectures overcome scalability bottlenecks, illustrating the operational strategies and design principles that power massive, reliable data‑center services.
In this comprehensive talk, Pei Dan outlines the technical and strategic roadmap for bringing AIOps to production, explains the challenges of anomaly detection, fault localization, root‑cause analysis and prediction, and demonstrates how to decompose complex operations problems into AI‑solvable tasks.
Alibaba's R&D efficiency team explains how intelligent operations—spanning resource planning, change management, monitoring, stability, and one‑click site building—are being transformed from manual tooling to automated, AI‑driven DevOps practices to boost efficiency, cut costs, and ensure high availability at massive scale.
This article introduces a practical LVS traffic anomaly detection method that combines short‑term and long‑term ratio analyses, dynamic thresholds, and periodicity‑aware techniques, providing code examples and a decision flow to help ops teams identify sudden traffic spikes or drops accurately.
This article explains how JD.com’s network team continuously optimizes its large‑scale infrastructure, designs effective monitoring strategies, implements practical monitoring solutions, and outlines future directions to improve network availability, fault detection, and operational efficiency across data centers and the internet backbone.
The article analyzes the rise of unmanned solutions such as supermarkets, fitness pods, and toy cars, explaining how mobile payment, IoT, and AI enable them, while examining the shifting cost structure, operational benefits, and challenges of this emerging business model.
This article examines the future of operations (运维) in the internet industry, comparing small and large companies, highlighting why Ops is technically demanding, and offering practical advice for career growth and skill development.
This article shares a hands‑on guide to using BlueKing's Fault Self‑Healing (FTA) platform with Zabbix, detailing benefits, integration steps, package creation, convergence rules, job‑script linking, and real‑world case studies that dramatically reduce manual alert handling time.
This article explores how rapid growth in network devices drives the need for comprehensive automation—covering script‑based tasks, zero‑touch provisioning, orchestration with OpenStack, device selection criteria, fault diagnosis, and monitoring—to keep operations ahead of business demands.
This article details JD.com's multi-active distributed architecture, covering its evolution from single‑data‑center to multi‑region deployments, network design, leaf‑spine topology, data consistency mechanisms, application scheduling, monitoring, and disaster recovery strategies that enhance high availability and user experience.
Alibaba’s Dragonfly P2P file distribution platform, built to handle massive file and container image delivery during peak events like Double‑11, combines peer‑to‑peer networking, smart compression, flow‑control and security features to achieve billions of transfers, petabyte‑scale traffic, and up to 99.9% reduction in registry outbound bandwidth.
This article explains how 360’s Linux Virtual Server (LVS) platform evolved with the FULLNAT forwarding mode, enhancing cross‑VLAN deployment, simplifying real‑server configuration, adding SYN‑proxy protection, and improving UDP handling, while detailing the new deployment architecture and operational benefits.
This article recounts how the Cruise team (later renamed GoCD) tackled the "last mile" problem by implementing continuous deployment through automated pipelines, extensive testing, risk mitigation, and frequent releases, ultimately reducing deployment time and improving software delivery reliability.
This guide compiles the most frequent problems encountered when installing the TDH Community Edition—such as hostname configuration, logical volume creation errors, service startup failures, firewall settings, and license issues—and provides clear step‑by‑step solutions to help users avoid and resolve these obstacles.
This article describes how the OPS team designed and implemented a private‑cloud Elasticsearch service using Mesos for resource management, Docker containers orchestrated by Marathon, and a suite of monitoring, self‑service configuration, and continuous deployment tools to improve resource utilization and operational efficiency.
This article explains the purpose of each log level, when to write logs, performance impacts, and concrete best‑practice patterns for INFO, DEBUG, WARN and ERROR in Java applications, providing actionable templates and configuration tips to build a robust logging system.
An operations director explains why hiring skilled ops engineers is hard, identifies the technology mismatch in typical stacks, and shares a practical skill‑map approach that lets teams cover most essential tools while giving engineers a clear learning roadmap.
This article presents ten practical Linux system‑administration tricks—from ejecting a stuck DVD drive and resetting a frozen console to sharing screen sessions, creating SSH tunnels for VNC, measuring network bandwidth, and gathering system diagnostics—each designed to save time and improve operational productivity.
This article recounts Ele.me's rapid growth from 2014 onward, detailing the challenges of network and server management, the evolution of their operations through standardization, process automation, and platform building, and how private cloud solutions like ZStack enabled fine‑grained, data‑driven infrastructure management.
The article examines Alibaba's operations team responsibilities, traces its evolution from scripting tools to automated and AI‑enhanced practices, outlines a five‑step intelligent‑ops roadmap, and discusses the major technical and organizational challenges facing future large‑scale, unmanned operations.
This article outlines a systematic methodology for understanding, evaluating, and quickly resolving production system incidents by categorizing system layers, assessing impact, employing Linux diagnostic tools, and designing fault‑tolerant mechanisms to minimize downtime and maintain core functionality.
The article explains how JD’s senior architect prepares for the 11.11 shopping festival by defining high‑availability goals, discussing scalability strategies, disaster‑recovery planning, performance optimization, and system resilience to ensure reliable service under massive traffic spikes.
The article summarizes a DevOps talk, outlining eight guiding principles—configuration over hard‑coding, redundancy over single points, restartability, whole‑stack delivery, statelessness, standardization, automation, and unattended operation—while sharing concrete tools, architectures, and real‑world experiences from a cloud provider.
Meituan‑Dianping’s payment team progressed from manual fault alerts to a fully automated channel management system that detects failures, disables affected banks, conducts controlled ramp‑up tests, and restores service, dramatically cutting response times, manpower costs, and secondary‑failure risks while boosting overall availability.
Discover a comprehensive collection of 80 essential Linux monitoring tools, ranging from system resource visualizers like nmon and Glances to log analyzers such as GoAccess, each described with features, usage tips, and links, helping sysadmins efficiently track performance, diagnose issues, and maintain robust infrastructure.
This article presents a comprehensive overview of cloud‑era network boundary management, detailing security challenges, unified access control concepts, endpoint protection, traffic analysis, and how automated operations and visualization platforms can reduce risk while maintaining efficient network operations.
This article chronicles Pinterest's seven‑year evolution from a single‑machine time‑series monitor to a multi‑component system that integrates metrics, log search, and distributed tracing, sharing architectural choices, scaling challenges, and lessons learned for building reliable, high‑performance operations platforms.
This guide walks you through preparing a Linux host, installing JDK, Maven, MySQL, downloading CacheCloud, configuring its database and properties, building the project, deploying the web service, and using the UI to add machines and create or import Redis clusters.
This article details Bilibili's Billions log platform—from its fragmented origins and design goals to the elastic‑stack‑based architecture, shard management, log sampling, custom Go splitters, and monitoring enhancements—highlighting the challenges faced and the roadmap for future improvements.
The USE Method provides a concise, structured framework for quickly locating resource bottlenecks and errors in complex systems by examining utilization, saturation, and error metrics across hardware, software, and cloud environments, enabling practitioners to prioritize and resolve performance issues efficiently.
This article details Ele.me's rapid expansion challenges and shares a three‑stage technical operations journey—fine‑grained division, stability maintenance, and efficiency gains—highlighting real incidents, monitoring upgrades, capacity testing, and practical insights for reliable large‑scale delivery platforms.
This article explains the fundamentals of CDN, introduces QunarCache (CDN + Cache) architecture, compares cache solutions such as Squid, Varnish, ATS and Nginx, describes deployment patterns, operational monitoring, and future improvements for scalable, resilient content delivery.
This reading note introduces the DevOps three‑step workflow from The Phoenix Project, outlines its three core practices, and summarizes fifteen scenarios illustrating common operational challenges such as untested deployments, missing change records, incident response, and improvement actions.
The article argues that even small internet firms cannot ignore operations as a capability, explaining how testing and ops improve functionality, stability, and business value, and outlining stages for integrating ops as a central control node within fast‑moving development cycles.
The article explains the motivation, design, architecture, and benefits of the Portal platform for application operations automation, detailing its Appcode abstraction, integrated systems, and feature set such as host management, resource requests, monitoring, and future development plans.
This article explains how to classify microservice systems, trace their architectural evolution, and conduct thorough dependency analysis to identify strong and weak couplings, enabling reliable governance and disaster‑recovery planning for high‑traffic events such as 618 and Double‑11.
A sudden surge of traffic triggered by a celebrity relationship announcement caused a Weibo outage, prompting frantic reactions from developers, operations, and management, and leading to an in‑depth analysis of high‑availability architecture, elastic scaling, hybrid‑cloud DCP platforms, and Docker‑based service deployment.
This article explains why service monitoring is essential for large‑scale microservice environments, outlines design principles, core monitoring components, dependency mapping, call‑chain analysis, capacity planning, root‑cause analysis, and presents a practical technical architecture for implementing robust monitoring solutions.
The talk outlines WeChat’s approach to handling rapid user growth through disciplined operational standards, cloud‑native management, precise capacity planning, and automated scaling, detailing configuration file conventions, name‑service design, hardware metric evaluation, stress‑testing methods, and dynamic resource allocation to maintain high efficiency and low cost.
This article explains how Tencent's social platform operations team prepares for massive holiday traffic spikes by following a four‑stage process—business preparation, capacity evaluation, resource provisioning, and scaling with stress testing—while detailing team structures, operational standards, and the supporting tool ecosystem that enable reliable, high‑availability services.
This article walks you through common Linux boot problems—recovering a forgotten root password with single‑user mode, fixing GRUB configuration errors, and using Linux rescue mode to repair critical system files—providing step‑by‑step commands and practical examples for effective troubleshooting.
This article presents a curated collection of fast, interactive, and productivity‑boosting command‑line utilities—including ag, tig, mycli, jq, shellcheck, yapf, mosh, fzf, PathPicker, htop, axel, cloc, ccache, tmux, and many more—along with brief usage examples and screenshots to help engineers streamline development, debugging, and system monitoring tasks.
Alibaba’s Global Operations Center (GOC) showcases a comprehensive business continuity solution that combines fault prevention, AI‑driven detection, rapid recovery, and automated post‑mortem processes, enabling 99.99% service availability and earning the DRI International Annual Best BCM Innovation Practice Award.
The article explores common feelings of fatigue, lack of achievement, and low morale among operations professionals, identifies six root causes such as missing systematic frameworks, unclear positioning, closed mindset, insufficient authority, stagnant improvement, and absent cultural integration, and offers actionable suggestions to transform operations into a strategic, valued function.
This article explains how LogHub enables real-time log collection and unified management for an e‑commerce takeout platform, covering operational challenges, logstore configuration, user promotion tracking, server and client logging methods, and network access options.
This article examines how Meipai, a popular live‑streaming and short‑video platform with over 150 million monthly active users, engineered a comprehensive, three‑dimensional monitoring architecture that spans client to server, integrates unified dashboards, and leverages both private and public cloud resources to ensure reliable, scalable operations.
This guide explains how to manage large Linux disks by using GPT partition tables, choosing XFS or EXT4 filesystems, performing partitioning with parted, formatting with appropriate mkfs commands, handling e2fsprogs version requirements, and considerations for lazy init and snapshot impacts.
A sudden rain‑triggered alert cascade exposed a mis‑tested data‑migration tool that overwrote production routing, causing a DB success‑rate drop to zero; the incident narrative details detection, root‑cause tracing, and a set of DevOps tool‑development standards to prevent future outages.
In programming and operations, every element—from services and dependencies to requests, machines, data centers, power, networks, and humans—can fail unexpectedly, so you must assume distrust and implement defensive measures such as monitoring, redundancy, rate limiting, fallback strategies, backups, and automated deployment.
This article outlines the career paths, essential skills, comprehensive toolsets, infrastructure design, security measures, and management practices required to transition from a Linux engineer to an effective operations manager responsible for high‑availability, scalable, and secure IT services.
The article explains how DevOps applies lean principles to integrate IT operations and software development, describes artifact‑centric workflows, introduces value‑stream mapping and flow measurements, and shows how these practices enable continuous delivery, feedback loops, and systematic improvement across the enterprise.
In today's fast‑moving digital landscape, waiting for perfect products is impossible, so companies like Amazon and HARTING adopt anti‑fragile, error‑embracing approaches—using systematic root‑cause analysis, agile micro‑services, and tools like Chaos Monkey—to transform failures into rapid innovation and competitive advantage.
This article shares practical DevOps tactics—including disaster‑recovery drills, SET architecture, automated self‑healing, and disciplined change management—to help operations teams reduce errors, improve reliability, and free time for strategic work.
This article details Meizu's journey from early internet eras to a mature continuous delivery platform, outlining the operational challenges, platform components, standardization, automation, and future intelligent operations to achieve high quality, efficiency, cost control, and security.
This guide walks you through preparing a Linux disk, installing extundelete, protecting the affected partition, and using extundelete commands to locate and restore both files and directories that were mistakenly removed, ensuring safe data recovery for system administrators.
This article examines why electricity dominates data‑center operating costs, outlines practical green‑IT measures—including enclosure design, airflow management, lighting, and renewable power—and presents three detailed case studies that illustrate how modular design, cold‑aisle containment, and innovative cooling can dramatically reduce PUE and overall energy consumption.
The CNUTCon Global Operations Conference highlighted UCloud's container and Serverless practices, explaining the evolution from virtualization waste to function‑as‑a‑service, showcasing UCloud's universal compute platform and real‑world use cases such as image processing, gene data analysis, and OCR recognition.
This article explores applying big‑data thinking and platforms—such as Flume, Spark Streaming, and HBase—to operations monitoring, detailing data sources, metric categories, architecture design, implementation steps, and the benefits of a scalable, low‑code monitoring platform.
This article explains the complete web request lifecycle, breaks down each latency component, and details how various cache layers—from browser DNS caches to server‑side memory and disk caches—can be leveraged and tuned to dramatically speed up page loads.
This article outlines the fundamentals, objectives, methods, core processes, tool selection, metrics, alerting, and interview tips for constructing a robust monitoring ecosystem, emphasizing Zabbix while comparing various open‑source solutions.
This article introduces the fundamental concepts, methods, types, goals, and product attributes of enterprise monitoring and alerting, explains the perspective differences between users and builders, and outlines a comprehensive monitoring system architecture for large‑scale operations.
The article examines how large financial institutions face technical, procedural, and risk‑control challenges when adopting DevOps, and proposes a three‑stage implementation roadmap—including automation of delivery pipelines, optimization of development models, and continuous‑delivery process refinement—to achieve reliable, rapid software releases.
This talk walks through the transition from classic application operations to a DevOps culture, highlighting common pain points, the need for standardization and automation, and practical steps for engineers to evolve their skills and boost organizational efficiency.
This article presents a step‑by‑step operations‑focused roadmap for evolving a website from a single‑server prototype to a highly available, horizontally scalable architecture using load balancing, caching, database replication, service‑oriented design, DNS round‑robin, CDN, and disaster‑recovery techniques.
The article recounts the three‑month development of the DevOps 5.0 LA release, describing integration pain points, MVP‑focused delivery, micro‑service architecture decisions, and the engineering optimizations required to scale the platform for enterprise‑wide continuous delivery.
This article examines how the Cameron‑Quinn cultural framework can be used to assess and shape corporate culture during a DevOps transformation, presenting a detailed case study of Seqr that links cultural diagnostics to continuous delivery practices and operational improvements.
This guide explains why removing Apache/Tomcat log files while the services are running fails to release disk space, analyzes the inode‑locking behavior, and provides practical solutions such as restarting the server, truncating logs, and using scheduled cleanup commands.
