This article describes how WiFi 万能钥匙 designed and implemented the Roma monitoring platform to handle billions of daily requests, covering background challenges, architectural principles, component design, data collection, transmission, storage, alerting, and future directions for large‑scale observability.
In 2017 Vipshop’s senior big‑data architect shares how the company grew its Hadoop‑based platform from zero to a thousand‑node cluster, detailing cluster health monitoring, multi‑HDFS deployment via Hive, Yarn container allocation improvements, and a hook‑driven Capping resource‑control system to boost stability and efficiency.
This article explains how Alibaba’s StarOps platform integrates DevOps automation, CMDB, release management, monitoring, host operations, bastion security and fault handling to enable large‑scale, unmanned, data‑driven operations across hybrid cloud environments.
This article recounts the technical evolution of 58 Express from its early startup days through rapid growth to an intelligent dispatch era, detailing challenges, database sharding, service decomposition, big‑data analytics, AI‑driven order routing, monitoring, and lessons learned for building a high‑performance backend system.
This article introduces the challenges of multi-dimensional monitoring, explains the limitations of traditional alerting, and presents the MDRCA algorithm—combining K‑means clustering, Explanatory Power, and Surprise metrics—to pinpoint root causes efficiently, while sharing practical AI integration experiences for large‑scale monitoring platforms.
This article explains how 360’s Hulk private cloud platform deploys Elasticsearch with a dedicated master architecture, load‑balancing, per‑business isolated clusters, SearchGuard security, dynamic tokenization, self‑service user features, and advanced monitoring to achieve high‑performance, scalable search services.
This article explains how to combine Node.js's experimental Async Hooks and Performance Timing APIs to create a simple tracing and performance monitoring tool, eliminating manual timing and offering a foundation that can be extended into a custom solution, while also noting an open‑source Pandora.js utility.
This article explains how Alibaba’s Sunfire monitoring platform processes terabytes of logs per minute, uses a pull‑based architecture with Brain‑Reduce‑Map roles, tackles scalability and reliability challenges, and outlines future directions such as MQL standardization and intelligent baselines.
A DBA on call discovers a sudden traffic spike caused by a few massive hot keys on a storage server, quickly isolates the issue, migrates data, applies throttling and caching, and outlines automation ideas to prevent future overloads, illustrating practical database operations and incident response.
This article explains why traditional macro performance metrics are insufficient, introduces ten essential micro‑metrics covering memory, thread, and network aspects, and shows how to capture them using GC logs, thread dumps, and tools like netstat or open‑source APM solutions.
The article describes the requirements, architecture, data‑sync strategy, monitoring, and operational workflow of a search open platform that enables fast, zero‑code creation of searchable interfaces, supporting real‑time indexing, customizable ranking, and extensible backend services.
This article shares practical Go microservice building experiences, covering gRPC-based communication, TLS security, request tracing, and comprehensive monitoring with Prometheus, including metric selection, alerting, and log management using Logrus and Graylog, to help reduce coupling and improve system observability.
The article describes how a startup tackled rapid MVP development and frequent changes by establishing a component‑based frontend platform, a Node.js middleware layer for Java services, and performance and error monitoring to streamline operations and improve development efficiency.
This article outlines a large‑scale monitoring solution for distributed microservice environments, comparing traditional ELK logging with a custom GPE stack (Grafana, Prometheus, Exporter, Consul), detailing architecture, components, workflows, and practical considerations for reliable observability.
This guide presents essential Bash scripting best practices and a collection of practical monitoring scripts—including random string generation, user creation, package checks, service status, host reachability, CPU/memory/disk usage, and website availability—complete with debugging tips and naming conventions for reliable automation.
The article recounts Ele.me's rapid growth and the resulting operational challenges, describing how the company progressed from ad‑hoc 1.0 practices to a standardized, automated 2.0 infrastructure built on ZStack private cloud, fine‑grained operations, and data‑driven management to improve quality, efficiency, and cost.
This tutorial walks through using the Zabbix API with the pyzabbix Python library to retrieve monitoring data, then visualizes it with ECharts, showing step‑by‑step how to create personalized monitoring pages for better operational insight.
The article summarizes a DevOps talk, outlining eight guiding principles—configuration over hard‑coding, redundancy over single points, restartability, whole‑stack delivery, statelessness, standardization, automation, and unattended operation—while sharing concrete tools, architectures, and real‑world experiences from a cloud provider.
Meituan‑Dianping’s payment team progressed from manual fault alerts to a fully automated channel management system that detects failures, disables affected banks, conducts controlled ramp‑up tests, and restores service, dramatically cutting response times, manpower costs, and secondary‑failure risks while boosting overall availability.
Over seven years, Pinterest’s monitoring team built and refined a three‑pronged observability platform—time‑series metrics, log search, and distributed tracing—scaling from a single‑machine system to handling millions of data points per second across tens of thousands of AWS VMs, while addressing reliability, cost, and usability challenges.
Discover a comprehensive collection of over 80 Linux monitoring and debugging utilities—including command‑line, system, network, and log tools—detailing their purpose, key features, and typical use cases to help you efficiently manage and troubleshoot server performance.
This article chronicles Pinterest's seven‑year evolution from a single‑machine time‑series monitor to a multi‑component system that integrates metrics, log search, and distributed tracing, sharing architectural choices, scaling challenges, and lessons learned for building reliable, high‑performance operations platforms.
This article shares a seasoned Linux sysadmin’s ten‑point roadmap—from mastering rsync, network services, and scripting to mastering sed/awk, MySQL, firewalls, monitoring tools, clustering, and backup—plus essential security and operational mindsets for thriving in modern infrastructure.
This article details Bilibili's Billions log platform—from its fragmented origins and design goals to the elastic‑stack‑based architecture, shard management, log sampling, custom Go splitters, and monitoring enhancements—highlighting the challenges faced and the roadmap for future improvements.
This article details Gome Group’s transition to cloud computing and operations automation, describing its corporate background, new operational strategies, the establishment of Gome Cloud, IAAS product architecture, monitoring solutions, automation standards, and deployment practices such as gray releases and Docker integration.
This article walks you through the complete process of setting up Zabbix on a Linux server—including preparing the environment, installing LAMP, configuring the Zabbix server and agent, creating databases, defining templates, items, triggers, graphs, and custom script alerts—to achieve real‑time network traffic monitoring and automated notifications.
This article details Ele.me's rapid expansion challenges and shares a three‑stage technical operations journey—fine‑grained division, stability maintenance, and efficiency gains—highlighting real incidents, monitoring upgrades, capacity testing, and practical insights for reliable large‑scale delivery platforms.
At Gome Group, a traditional retailer with over 30,000 employees, the IT team built a unified cloud platform and automated operations, consolidating resources across dozens of subsidiaries to cut costs, boost efficiency, and enable rapid service delivery through IAAS, standardized processes, and custom monitoring tools.
This article outlines Taobao's learning objectives, traces the evolution of its backend architecture from V1.0 to V3.0, highlights the technical challenges faced at each stage, and explains the architectural decisions—such as modularization, service‑oriented frameworks, distributed storage, and large‑scale monitoring—that enabled massive scalability, reliability, and performance improvements.
The article details 58's private cloud platform built on container technology, explaining the motivations, overall architecture, and core module designs such as container management, network model, image repository, logging, and monitoring, illustrating how Docker and Kubernetes enable efficient resource utilization, rapid scaling, and streamlined deployment.
This guide introduces thirteen essential Linux operation utilities—including Nethogs, IOZone, IOTop, IPtraf, IFTop, HTop, NMON, MultiTail, Fail2ban, Tmux, Agedu, NMap and Httperf—providing brief descriptions, download links and step‑by‑step commands to install and use each tool for monitoring, performance testing, security and session management.
This article explains Kafka end‑to‑end auditing, compares three products (Chaperone, Confluent Control Center, Kafka Monitor), describes timestamp and index embedding techniques, and outlines their architectures, metrics, and implementation details for detecting data loss, duplication, and latency.
The article explores how adopting a "don't trust" mindset—through rigorous input validation, defensive coding, thorough testing, gradual rollouts, and comprehensive monitoring—helps build resilient, high‑performance systems and avoid common pitfalls in development and operations.
This article describes how Tongcheng Travel’s monitoring system evolved from a monolithic design to a distributed and big‑data‑based architecture, introducing real‑time processing with Storm, machine‑learning‑enhanced alerts, and a multivariate linear regression model that dramatically improves traffic anomaly detection accuracy.
Overwatch is an open‑source distributed monitoring platform built by Dada‑Jingdong Home that collects, aggregates, and visualizes RPC traffic across thousands of micro‑services in real time, enabling engineers to quickly pinpoint the root cause of system failures using directed‑graph visualizations and CQRS‑based data queries.
This article examines how Meipai, a popular live‑streaming and short‑video platform with over 150 million monthly active users, engineered a comprehensive, three‑dimensional monitoring architecture that spans client to server, integrates unified dashboards, and leverages both private and public cloud resources to ensure reliable, scalable operations.
The sixth Meitu Internet Technology Salon in Beijing showcased Meipai’s evolution, with senior engineers detailing the platform’s recommendation system, real‑time background segmentation, monitoring framework, live‑streaming and bullet‑screen architecture, offering practical insights and best‑practice lessons for building and optimizing large‑scale video services.
This step‑by‑step guide shows how to install Grafana 4 on a Linux server, configure it to start on boot, add required fonts, install the Grafana‑Zabbix and clock‑panel plugins, and create a dashboard that visualizes Zabbix metrics such as memory usage.
In programming and operations, every element—from services and dependencies to requests, machines, data centers, power, networks, and humans—can fail unexpectedly, so you must assume distrust and implement defensive measures such as monitoring, redundancy, rate limiting, fallback strategies, backups, and automated deployment.
This article details Qunar's journey of automating application operations, covering the evolution of their host‑management system, unified monitoring/alert platform, and data‑interchange mechanisms that enabled the company to grow from a few hundred to over ten thousand servers with a stable six‑person ops team.
This article shares practical DevOps tactics—including disaster‑recovery drills, SET architecture, automated self‑healing, and disciplined change management—to help operations teams reduce errors, improve reliability, and free time for strategic work.
This guide explains how to choose between push and pull monitoring models, introduces Prometheus architecture and metric syntax, shows Node.js client integration with code examples, and covers Alertmanager features and Grafana visualization for effective application monitoring.
This article describes an integrated solution that combines code quality monitoring during development with automated crash issue tracking after deployment, using a custom platform, Jenkins, Gradle plugins, static analysis tools, and rule-based filtering to continuously improve project reliability and performance.
This article outlines the essential backend infrastructure components and best‑practice patterns—such as API gateways, service frameworks, caching, databases, search engines, message queues, authentication, configuration, service governance, scheduling, logging, and monitoring—required to build stable, scalable, and maintainable internet applications.
This article details the challenges of traditional monitoring systems, the design and implementation of a custom high‑availability monitoring platform using a Golang‑based service tree, Raft‑backed storage, InfluxDB for time‑series data, and a modular architecture that supports Windows agents, third‑party reporting, and AI‑driven future enhancements.
Sysadmins need reliable command-line utilities to keep services running 24/7, and this guide compiles the most commonly used networking, security, storage, logging, backup, performance, efficiency, package-management, and hardware inspection tools on Linux, explaining each command’s purpose and typical use cases.
This article outlines the fundamentals, objectives, methods, core processes, tool selection, metrics, alerting, and interview tips for constructing a robust monitoring ecosystem, emphasizing Zabbix while comparing various open‑source solutions.
This article introduces the fundamental concepts, methods, types, goals, and product attributes of enterprise monitoring and alerting, explains the perspective differences between users and builders, and outlines a comprehensive monitoring system architecture for large‑scale operations.
This article shares practical experiences of using Prometheus for monitoring complex services, covering metric types, PromQL query techniques, naming conventions, service discovery with file‑based configs, high‑availability sharding, alerting via Alertmanager, and visualisation with Grafana, providing actionable guidance for reliable observability.
This article explains how Tencent Game operations moved from manual, threshold‑based monitoring to an AI‑powered, data‑driven workflow that automates scaling, improves online‑curve monitoring, enables full‑dimensional analysis, and reduces time, labor, and cost while enhancing player experience.
This article details Qunar's hardware automation operations platform, covering the hardware scope, pain points of manual processes, a five‑stage lifecycle, automated testing, data collection, fault handling, and the underlying Mesos‑Marathon‑Docker infrastructure that together improve efficiency, reliability, and cost control.
This article details the challenges, architectural evolution, key components, implementation experiences, and future directions of Ctrip's operations workflow platform, illustrating how a multi‑stage, layered design and standardized services have transformed manual IT operations into an automated, observable, and scalable system.
This article details Qunar's end‑to‑end hardware automation system, covering background challenges, lifecycle management, automated testing, data collection, fault detection, and visualized monitoring, and explains how the integrated platform reduces manual effort, improves reliability, and cuts operational costs.
This article explains how to design and implement a comprehensive operations monitoring platform by integrating Ganglia for data collection and Centreon for alerting, detailing a six‑layer architecture, data flow, seamless integration, and practical Q&A for real‑world deployment.
This article compiles 22 practical recommendations from an operations manager covering domain management, CDN usage, image servers, data center selection, monitoring, security, redundancy, high‑availability architecture, disaster‑recovery planning, and team coordination to help ensure stable and secure online services.
The article outlines Meituan‑Dianping’s approach to front‑end service availability for its checkout system, defining availability across code, static resources, and network links, measuring failure duration, identifying typical bugs, and implementing a three‑stage assurance strategy using people processes, engineering tools, lightweight technology choices, and concrete practices such as TypeScript adoption, automated testing, health‑checks, DNS protection, and post‑incident monitoring.
This guide highlights the urgent need for automation in modern operations and curates essential books, documentation, and information sources covering Puppet, Nagios, Zabbix, Linux scripting, high‑availability servers, and Python‑based automation to help both seasoned engineers and newcomers alike.
The article reviews the evolution of big‑data architectures from a simple RDB‑centric pipeline to a SaaS‑based solution, highlighting common bottlenecks such as scaling, integration, cost, and operational complexity, and shares practical experiences and best‑practice recommendations for building efficient, maintainable data platforms.
This article explains website availability metrics, fault‑weight scoring, layered high‑availability architecture, session management strategies, reusable service design, data redundancy, quality assurance processes, and monitoring practices essential for maintaining reliable large‑scale web systems.
The article details Meizu's massive real‑time push system handling 25 million online users and 600 million messages per minute, explains its four‑layer architecture, and shares how the team tackled phone power consumption, mobile network instability, massive connections, monitoring, and gray‑release deployment.
This article details how Tencent's SNG operations team leveraged the ZhiYun intelligent operations platform—through standardized processes, massive IaaS provisioning, CMDB management, automated workflows, and real‑time capacity monitoring—to support the high‑traffic “Military Photo” H5 campaign, scaling up to 4,000 servers and 24 GB bandwidth.
Drawing from a year‑and‑a‑half of ops experience, this guide outlines six practical categories—online operation standards, data handling, security, daily monitoring, performance tuning, and mindset—to help engineers avoid costly mistakes and maintain stable, secure systems.
Drawing from a year‑and‑a‑half of sysadmin experience, this guide outlines practical online operation standards, data protection habits, security hardening, daily monitoring, performance tuning, and the right mindset to keep production environments stable and resilient.
This guide shows how to create a Python monitoring tool that queries MySQL error logs for third‑party API failures, triggers email alerts when error counts exceed a threshold, runs as a scheduled task, and is managed with Supervisor for reliable operation.
This guide outlines the core Linux system fundamentals, networking services, scripting languages, text‑processing utilities, database handling, firewall configuration, monitoring solutions, clustering, and backup techniques that form the essential toolkit for aspiring Linux operations engineers.
This article presents a comprehensive collection of Python tutorials and scripts covering file I/O modes, directory traversal, log analysis, simple games, command‑line argument handling, process monitoring, port checking, authentication loops, and SNMP‑based CPU and network traffic monitoring, providing a solid foundation for automation and operations tasks.
This article details the evolution, architecture, deployment, monitoring, and performance optimization of Ultron—360’s internal OpenStack‑based virtualization platform—covering its three development stages, technical stack, automation with Ansible, advanced features like VXLAN and Ceph, and lessons learned from large‑scale operations.
This article examines the unique challenges of monitoring containers, outlines three categories of metrics to collect, compares host‑centric and layered monitoring architectures, provides detailed methods for gathering CPU, memory, I/O and network data via cgroup files and Docker commands, and shares practical insights, tooling recommendations, and a Q&A session for effective container observability.
An in‑depth look at the evolution of game operations—from early PC and web games to today’s mobile and cloud‑based titles—covering architecture, Tcaplus storage, CMDB building, automated deployment, performance monitoring, data warehousing, and the essential skills and challenges faced by game ops engineers.
The article shares practical operations lessons—from periodic permission audits and thorough documentation to capacity monitoring, log rotation, and automation—illustrating how systematic practices and tooling can standardize and streamline IT infrastructure management.
Facing the challenge of storing over 120 billion daily monitoring points from hundreds of thousands of servers, Tencent’s monitoring platform migrated from a custom solution and OpenTSDB to a finely tuned HBase architecture, achieving 3‑5× higher throughput, improved reliability, and significant storage savings.
The article describes how Airbnb's data infrastructure team built a next‑generation log storage and query platform to improve data quality, timeliness, flexibility, and anomaly detection, outlining the system architecture, key requirements, five improvement areas, and the resulting benefits.
The Suning Cloud Commerce IT headquarters hosted a comprehensive Intelligent Operations forum featuring experts from Alibaba, Weibo, Meituan, 360, Meizu and PPD, who shared practical insights on automation, platformization, AI‑driven big‑data analytics, network automation, security, and monitoring across modern IT operations.
This article introduces the fundamentals of time‑series databases, compares them with traditional databases, surveys industry adoption, and details how Tiger Live leverages OpenTSDB, Grafana, and Bosun to build a scalable monitoring system for live‑streaming services.
From early manual deployments to a sophisticated, multi-layered monitoring stack—including ELK, Zabbix, Statsd, Grafana, and Prometheus—Bilibili’s ops team shares the evolution, challenges, and lessons learned in building scalable, automated infrastructure for massive internet traffic.
This article explains how to set up the wechat_sender tool—built on wxpy and tornado—to forward logs, alerts, and scheduled messages from any Python application directly to personal or group WeChat chats, enabling instant monitoring and notification.
This article explains the fundamentals of monitoring, distinguishes traditional OPS from SRE perspectives, defines monitoring objects and metrics, introduces quantitative thinking with SLI/SLO, and presents the USE method with a MySQL example to help engineers detect and prevent failures efficiently.
This article presents a comprehensive redesign of Baidu Waimai’s client‑side activity module, detailing background challenges, design goals, functional and performance specifications, trade‑off analyses of three architectural alternatives, and the chosen parallel HTTP‑request solution with monitoring, degradation, and phased rollout plans.
The article explains how to automate the tedious process of adding multiple hosts to Cacti and Nagios by using shell‑wrapped PHP scripts and custom templates, provides download links, and shares practical tips to avoid common installation pitfalls.
This article describes the background, architecture, encountered performance bottlenecks, and a three‑level caching strategy (Redis‑based) implemented in Python to accelerate a measurement platform that aggregates bug, issue, and code metrics for a rapidly growing development team.
The article presents a comprehensive approach to solving crash and performance issues in large‑scale mobile applications by reconstructing user interaction traces through a no‑track analytics platform, compile‑time AOP instrumentation, and unified data aggregation, ultimately improving debugging efficiency and reducing operational overhead.
This article outlines a comprehensive methodology for handling large‑scale online service incidents, covering goals, the "jump‑fill‑avoid" framework, step‑by‑step processes for detection, diagnosis, remediation, and post‑mortem analysis, as well as essential monitoring, logging, and escalation infrastructure.
This article analyzes how operations engineers in Beijing progress from entry‑level salaries of a few thousand yuan to senior roles earning over thirty thousand, by examining job postings, required skills, and experience levels to map a clear career growth path.
This article explains how to design performance testing scenarios, covering test models, metrics, script preparation, concurrency calculations, pressure strategies, run times, delay settings, user termination, monitoring methods, and various typical scenario types such as baseline, load, mixed, capacity, large‑concurrency, stability and scalability tests.
This guide walks through configuring Zabbix to monitor Nginx and PHP‑FPM status, covering software installation paths, enabling status modules, creating extraction scripts, setting up Zabbix agent userparameters, restarting services, testing data retrieval, and adding server‑side templates for items, triggers, and graphs.
This article details how Tencent's AMS system was analyzed, traffic‑estimated, and redesigned for high‑availability during the QQ Spring Festival Red Packet event, covering architecture mapping, scaling strategies, overload protection, flexible availability, disaster recovery, monitoring, and practical lessons learned.
This article introduces a straightforward DevOps implementation framework that maps eleven essential service capabilities across the software development lifecycle, explains why adopting DevOps is a multi‑year journey, and uses a fitness analogy to illustrate how enterprises can progressively build these capabilities.
This article details Tencent's large‑scale live streaming architecture for NBA games, covering the rapid growth of live video, key technical features, network transmission challenges, multi‑angle production, CDN deployment, monitoring, big‑data processing, and strategies for ensuring low latency and high reliability for millions of concurrent viewers.
This article explains how to analyze and visualize Docker container logs using the ELK stack, covering preparation, parsing tips, Kibana query techniques, and example visualizations to help monitor Dockerized environments effectively in production.
This article explains the challenges of Docker log collection and provides a step‑by‑step guide for installing ELK components, configuring Logstash, Kibana, and various log shippers such as Filebeat, logging drivers, Logspout, and Logz.io to reliably gather and visualize container logs.
This article outlines a practical operations learning path, comparing CentOS 6 and 7, recommending foundational skills across OS, web services, databases, load balancing, caching, NoSQL, storage, version control, monitoring, and scripting to help engineers stay current and effective.
This guide walks through installing pnp4nagios on CentOS 6.8, configuring required packages, compiling the software, testing the installation, understanding its bulk mode with npcd, and adjusting Nagios and pnp4nagios settings to enable dynamic performance graphs.
This guide presents practical Linux and server administration commands, covering log cleanup, nginx IP analysis, tcpdump capture, Python date formatting and string reversal, subprocess execution, multiprocessing, iptables port forwarding, cron scheduling, file relocation, RAID concepts, Oracle backup strategies, port checking, Apache MPM modes, and monitoring tool comparisons.
China Life’s Shanghai Data Center team transformed chaotic, multi‑system operations into a unified, automated platform by standardizing hardware, processes, and tools, leveraging OpenStack, Docker, Zabbix, and custom scripts, ultimately achieving efficient monitoring, change management, and a mobile‑enabled DevOps workflow.
This guide walks through the evolution of a small .NET‑based e‑commerce system, covering its initial LAMP‑style setup, detailed backend architecture, logging and monitoring solutions, master‑slave database design, shared‑storage image server, mobile M‑site construction, capacity estimation methods, and caching strategies.
This comprehensive guide explains why monitoring is vital for operations, outlines clear objectives and methods, compares popular open‑source and commercial tools, details a Zabbix‑based workflow, and covers hardware, system, application, network, security, API, performance, and business metrics with practical alerting strategies.
The Sentinel system was built to provide real‑time, zero‑modification monitoring of airline ticket business services by consuming Tianwang logs through a Storm cluster, offering flexible rule configuration, addressing performance pitfalls, and planning future enhancements such as custom monitoring scripts and visual dashboards.
Ctrip built a private container cloud to handle massive seasonal traffic spikes, enabling rapid, automated scaling and shrinking of resources, improving deployment speed, resource utilization, and operational intelligence across more than 20 business units.
This article details Alibaba Group's journey of transforming its GitLab deployment from a single‑node setup to a distributed, sharded architecture that handles tens of millions of daily requests, achieves near‑perfect reliability, and incorporates performance, monitoring, and disaster‑recovery innovations.
An insightful reflection on Google’s SRE philosophy shows how dialectical thinking—questioning absolute stability, embracing limited toil, prioritizing simple monitoring, recognizing automation’s hidden risks, and practicing real‑world failure drills—can reshape operations, encouraging smarter, more resilient system design.
