Facing the challenge of storing over 120 billion daily monitoring points from hundreds of thousands of servers, Tencent’s monitoring platform migrated from a custom solution and OpenTSDB to a finely tuned HBase architecture, achieving 3‑5× higher throughput, improved reliability, and significant storage savings.
The article describes how Airbnb's data infrastructure team built a next‑generation log storage and query platform to improve data quality, timeliness, flexibility, and anomaly detection, outlining the system architecture, key requirements, five improvement areas, and the resulting benefits.
The Suning Cloud Commerce IT headquarters hosted a comprehensive Intelligent Operations forum featuring experts from Alibaba, Weibo, Meituan, 360, Meizu and PPD, who shared practical insights on automation, platformization, AI‑driven big‑data analytics, network automation, security, and monitoring across modern IT operations.
This article introduces the fundamentals of time‑series databases, compares them with traditional databases, surveys industry adoption, and details how Tiger Live leverages OpenTSDB, Grafana, and Bosun to build a scalable monitoring system for live‑streaming services.
From early manual deployments to a sophisticated, multi-layered monitoring stack—including ELK, Zabbix, Statsd, Grafana, and Prometheus—Bilibili’s ops team shares the evolution, challenges, and lessons learned in building scalable, automated infrastructure for massive internet traffic.
This article explains how to set up the wechat_sender tool—built on wxpy and tornado—to forward logs, alerts, and scheduled messages from any Python application directly to personal or group WeChat chats, enabling instant monitoring and notification.
This article explains the fundamentals of monitoring, distinguishes traditional OPS from SRE perspectives, defines monitoring objects and metrics, introduces quantitative thinking with SLI/SLO, and presents the USE method with a MySQL example to help engineers detect and prevent failures efficiently.
This article presents a comprehensive redesign of Baidu Waimai’s client‑side activity module, detailing background challenges, design goals, functional and performance specifications, trade‑off analyses of three architectural alternatives, and the chosen parallel HTTP‑request solution with monitoring, degradation, and phased rollout plans.
The article explains how to automate the tedious process of adding multiple hosts to Cacti and Nagios by using shell‑wrapped PHP scripts and custom templates, provides download links, and shares practical tips to avoid common installation pitfalls.
This article describes the background, architecture, encountered performance bottlenecks, and a three‑level caching strategy (Redis‑based) implemented in Python to accelerate a measurement platform that aggregates bug, issue, and code metrics for a rapidly growing development team.
The article presents a comprehensive approach to solving crash and performance issues in large‑scale mobile applications by reconstructing user interaction traces through a no‑track analytics platform, compile‑time AOP instrumentation, and unified data aggregation, ultimately improving debugging efficiency and reducing operational overhead.
This article outlines a comprehensive methodology for handling large‑scale online service incidents, covering goals, the "jump‑fill‑avoid" framework, step‑by‑step processes for detection, diagnosis, remediation, and post‑mortem analysis, as well as essential monitoring, logging, and escalation infrastructure.
This article analyzes how operations engineers in Beijing progress from entry‑level salaries of a few thousand yuan to senior roles earning over thirty thousand, by examining job postings, required skills, and experience levels to map a clear career growth path.
This article explains how to design performance testing scenarios, covering test models, metrics, script preparation, concurrency calculations, pressure strategies, run times, delay settings, user termination, monitoring methods, and various typical scenario types such as baseline, load, mixed, capacity, large‑concurrency, stability and scalability tests.
This guide walks through configuring Zabbix to monitor Nginx and PHP‑FPM status, covering software installation paths, enabling status modules, creating extraction scripts, setting up Zabbix agent userparameters, restarting services, testing data retrieval, and adding server‑side templates for items, triggers, and graphs.
This article details how Tencent's AMS system was analyzed, traffic‑estimated, and redesigned for high‑availability during the QQ Spring Festival Red Packet event, covering architecture mapping, scaling strategies, overload protection, flexible availability, disaster recovery, monitoring, and practical lessons learned.
This article introduces a straightforward DevOps implementation framework that maps eleven essential service capabilities across the software development lifecycle, explains why adopting DevOps is a multi‑year journey, and uses a fitness analogy to illustrate how enterprises can progressively build these capabilities.
This article details Tencent's large‑scale live streaming architecture for NBA games, covering the rapid growth of live video, key technical features, network transmission challenges, multi‑angle production, CDN deployment, monitoring, big‑data processing, and strategies for ensuring low latency and high reliability for millions of concurrent viewers.
This article explains how to analyze and visualize Docker container logs using the ELK stack, covering preparation, parsing tips, Kibana query techniques, and example visualizations to help monitor Dockerized environments effectively in production.
This article explains the challenges of Docker log collection and provides a step‑by‑step guide for installing ELK components, configuring Logstash, Kibana, and various log shippers such as Filebeat, logging drivers, Logspout, and Logz.io to reliably gather and visualize container logs.
This article outlines a practical operations learning path, comparing CentOS 6 and 7, recommending foundational skills across OS, web services, databases, load balancing, caching, NoSQL, storage, version control, monitoring, and scripting to help engineers stay current and effective.
This guide walks through installing pnp4nagios on CentOS 6.8, configuring required packages, compiling the software, testing the installation, understanding its bulk mode with npcd, and adjusting Nagios and pnp4nagios settings to enable dynamic performance graphs.
This guide presents practical Linux and server administration commands, covering log cleanup, nginx IP analysis, tcpdump capture, Python date formatting and string reversal, subprocess execution, multiprocessing, iptables port forwarding, cron scheduling, file relocation, RAID concepts, Oracle backup strategies, port checking, Apache MPM modes, and monitoring tool comparisons.
China Life’s Shanghai Data Center team transformed chaotic, multi‑system operations into a unified, automated platform by standardizing hardware, processes, and tools, leveraging OpenStack, Docker, Zabbix, and custom scripts, ultimately achieving efficient monitoring, change management, and a mobile‑enabled DevOps workflow.
This guide walks through the evolution of a small .NET‑based e‑commerce system, covering its initial LAMP‑style setup, detailed backend architecture, logging and monitoring solutions, master‑slave database design, shared‑storage image server, mobile M‑site construction, capacity estimation methods, and caching strategies.
This comprehensive guide explains why monitoring is vital for operations, outlines clear objectives and methods, compares popular open‑source and commercial tools, details a Zabbix‑based workflow, and covers hardware, system, application, network, security, API, performance, and business metrics with practical alerting strategies.
The Sentinel system was built to provide real‑time, zero‑modification monitoring of airline ticket business services by consuming Tianwang logs through a Storm cluster, offering flexible rule configuration, addressing performance pitfalls, and planning future enhancements such as custom monitoring scripts and visual dashboards.
Ctrip built a private container cloud to handle massive seasonal traffic spikes, enabling rapid, automated scaling and shrinking of resources, improving deployment speed, resource utilization, and operational intelligence across more than 20 business units.
This article details Alibaba Group's journey of transforming its GitLab deployment from a single‑node setup to a distributed, sharded architecture that handles tens of millions of daily requests, achieves near‑perfect reliability, and incorporates performance, monitoring, and disaster‑recovery innovations.
An insightful reflection on Google’s SRE philosophy shows how dialectical thinking—questioning absolute stability, embracing limited toil, prioritizing simple monitoring, recognizing automation’s hidden risks, and practicing real‑world failure drills—can reshape operations, encouraging smarter, more resilient system design.
This article outlines the motivation, requirements, modular decomposition, and architecture of a distributed web crawling platform that emphasizes reusability, lightweight modules, real‑time monitoring, and easy configuration for diverse data‑collection tasks.
This article describes the design, architecture, and operational features of Baidu Waimai's unified voucher issuance platform, detailing its four‑layer backend structure, permission and strategy configurations, flow‑control mechanisms, service isolation, monitoring visualizations, and re‑entrancy safeguards to support large‑scale marketing distribution.
The article details Ctrip's rapid business growth driving the need for elastic scaling, the adoption of container technology to achieve second‑level provisioning, the design of their container cloud platform—including deployment principles, network choices, orchestration evaluations, monitoring solutions, and the CDOS overview—providing practical insights for large‑scale cloud‑native operations.
This article presents a comprehensive server‑side monitoring solution covering functional and performance requirements, monitoring objects, design choices between self‑monitoring and centralized reporting, system architecture, API definitions, key challenges such as key collisions, data formats, storage options, and operational considerations.
Learn the essential Linux command-line tools and step-by-step commands you need to run within the first minute of logging into a server to quickly assess process activity, resource usage, and potential bottlenecks, enabling effective performance troubleshooting in production environments.
The article outlines practical engineering guidelines for internet startups, emphasizing simplicity, rapid development, resource efficiency, and the use of cloud servers, MySQL, caching, asynchronous processing, logging, monitoring, documentation, and integrated build‑deploy pipelines to build stable, low‑cost backend systems.
This article details Ctrip's journey in building a private container cloud to support rapid business growth, covering elasticity challenges, container deployment principles, orchestration platform choices, network design, operational issues, custom executors, monitoring solutions, and the overarching CDOS system.
The article describes Dropbox’s Stormcrow system, a configurable feature‑gate platform that enables fast, safe rollout of new functionality across web, desktop, and mobile clients, supports granular A/B testing, leverages custom data fields, and integrates deployment, monitoring, and audit tooling for large‑scale operations.
The article details Vipshop's two‑year PaaS implementation on Kubernetes, covering the CI/CD pipeline, custom networking, Docker registry enhancements, logging and monitoring solutions, as well as the operational challenges and fixes encountered during the migration.
The article outlines a typical six‑segment workday of a Tencent operations engineer, detailing how they review past results, tackle urgent issues, take breaks, perform routine tasks, debug scripts, and handle after‑hours responsibilities, offering practical insight for aspiring sysadmins.
Metrics‑Driven Development (MDD) extends test‑driven principles by embedding real‑time monitoring into design, enabling rapid, precise, and granular software iterations, improving early problem detection, decision support, and aligning development with DevOps culture.
This article outlines a comprehensive PDCA‑based methodology for e‑commerce platforms to proactively prevent issues, quickly detect anomalies, and execute rapid decisions during large‑scale promotions, covering system goal definition, performance evaluation, capacity planning, SLA management, and team/process maturity.
Learn how to achieve near real‑time, time‑series monitoring of Spring Boot applications without writing code by combining Spring Boot Actuator, Jolokia, InfluxDB, Telegraf, and Grafana, while evaluating alternative tools like Prometheus, Graphite, and JMXTrans and understanding their limitations.
Mt‑Falcon, Meituan’s re‑engineered successor to Zabbix, introduces a modular architecture—Agent, Transfer, HBS, Judge, Graph, Alarm, Portal—and extensive refactorings that boost memory efficiency, asynchronous data handling, multi‑condition alerts, and API exposure, enabling over one million QPS, 200 million metrics, and robust, scalable monitoring across the company.
This article details Alibaba's mobile app operational practices, covering the challenges of client-side maintenance, their high‑frequency release pipeline, gray‑release mechanisms, monitoring, trace systems, remote logging, and rapid issue resolution to ensure stability and performance at massive scale.
This article reviews four ELK deployment patterns, compares their strengths and weaknesses, and demonstrates how ELK can be applied to big‑data operations such as log aggregation, system monitoring, fault diagnosis, and performance analysis with practical examples.
A developer recounts how a Redis connection pool overflow across dozens of clusters was traced to a single misbehaving service, diagnosed with netstat and ps commands, and resolved by adjusting configuration and stopping the offending process, illustrating practical troubleshooting of connection limits.
This article shares JD’s large‑scale monitoring system (MDC) design, covering its three‑tier architecture, agent‑based data collection, performance optimizations for SNMP/IPMI, low‑overhead deployment, high‑availability strategies, and practical lessons on scaling monitoring across thousands of physical machines and containers.
This article provides a comprehensive overview of Zabbix monitoring, comparing it with Cacti and Nagios, detailing its features, architecture, and step‑by‑step instructions for installing and configuring the server, database, web interface, and agent components on a Linux environment.
CallGraph, JD.com’s in‑house distributed tracing platform, provides low‑intrusion, high‑performance monitoring for micro‑service ecosystems, enabling real‑time call‑graph analysis, TP metrics, flexible configuration, and future extensions such as deep‑learning‑driven insights.
The article outlines the architecture, distributed design, redundancy, monitoring, automation, and fault‑handling strategies employed in a game company's platform backend to ensure high availability and efficient daily operations.
Effective Node.js monitoring boosts competitiveness, user experience, and cost efficiency, and this guide outlines eleven key recommendations—from tracking downtime and response thresholds to linking performance with business metrics and leveraging third‑party APM tools—ensuring robust, noise‑free alerts and secure, scalable applications.
The 2016 Ops Alert Report reveals Zabbix’s dominance, preferred notification channels, monthly and daily alert trends, peak alert times, regional distribution, and quirky usage statistics, offering valuable insights for operations teams to optimize monitoring and incident response.
This article presents a comprehensive case study of a business monitoring system, covering its background, architectural analysis, module design, time‑series database selection, visualization with Grafana, alerting strategies, decision‑making logic, and intelligent monitoring experiments, followed by key takeaways and lessons learned.
A rare Oracle bug caused object IDs to exceed their maximum value, preventing creation of tables, sequences, and indexes; the article analyzes the error, shows how rapid DDL activity exhausts IDs, demonstrates reproduction steps, and offers short‑ and long‑term mitigation and monitoring recommendations.
This article shares the engineering team’s experience of building a high‑growth, reliable backend for English Fluently, covering inter‑service communication with gRPC, service discovery, Docker‑based deployment, health‑checking, monitoring, autoscaling, Kubernetes orchestration, and multi‑cell availability strategies.
This article explains how to standardize and automate MySQL management at scale, covering dedicated instance deployment, configuration consistency, multi‑instance creation, metadata collection, backup, monitoring, high‑availability with Zookeeper, and task orchestration using DBTask to achieve rapid, reliable database services.
Alibaba’s SunFire platform delivers massive‑scale, real‑time log collection, processing, and visualization for e‑commerce spikes like Double 11, using low‑overhead agents, asynchronous Map/Reduce pipelines, fault‑tolerant task scheduling, and shared inputs to ensure accurate, low‑latency monitoring across billions of transactions.
This article explains how to standardize and automate MySQL operations—including multi‑instance deployment, metadata collection, monitoring, backup, and high‑availability using Zookeeper—so that large‑scale database services can be provisioned, managed, and scaled with minimal human intervention.
This article chronicles Alibaba's ten‑year journey from monolithic Java EE deployments to a cloud‑native microservice ecosystem, detailing the technical challenges, the evolution of its EDAS RPC frameworks, comprehensive monitoring, capacity planning, and the strategies that enabled resilient large‑scale services during massive traffic events.
This article explains how to design and implement a comprehensive operations monitoring platform using Ganglia for data collection and Centreon for alerting, detailing a six‑layer architecture, integration steps, data flow, and practical Q&A for effective fault detection and response.
The article details the technical challenges encountered during the service‑oriented decomposition of Qunar's payment platform, covering Dubbo and HTTP service conventions, database sharding and read/write separation, asynchronous processing, multi‑system management, and comprehensive monitoring and alerting solutions.
This article compiles expert Q&A on operations, covering tool selection, monitoring, Linux version choices, automation platforms, security, Docker, backup strategies, and career advice, offering practical guidance for modern infrastructure management.
This article explains how Taobao achieves horizontal scalability by adopting stateless session handling, efficient client‑side cookie storage, multi‑level caching, service splitting with HSF, database sharding via TDDL, asynchronous messaging, unstructured data storage, and comprehensive monitoring and configuration management.
The article critiques popular but unnecessary log‑analysis features—such as sub‑second alerts, endless pagination, flashy maps, full SQL support, bulk downloads, and live tail—arguing that focusing on practical alert content, efficient querying, and proper architecture yields far more value for IT operations.
The article details Meizu’s real‑time push system that supports 25 million online users and 6 million messages per minute, describing its four‑layer architecture, power‑saving strategies, network‑instability fixes, massive‑connection handling, monitoring practices, and gray‑release deployment techniques.
This article explains how Qunar tackled growing network device management workload, low‑efficiency manual processes, and operational risk by designing an integrated platform that automates common tasks, enforces permission‑based controls, records audits, and provides real‑time monitoring and scalable data collection.
This article details the evolution from manual, standardized operations to an automated, intelligent cloud operations platform for a banking card organization, describing its motivations, core features, key scenarios, technical architecture, scheduling algorithms, data visualization, and real‑world outcomes.
The article examines the technical challenges encountered during the service‑oriented decomposition of Qunar's payment platform—including development efficiency, interface conventions, concurrency, security, monitoring, database sharding, read‑write separation, and asynchronous processing—and presents concrete solutions and best‑practice recommendations.
This guide compiles a comprehensive set of Oracle SQL statements and explanations for detecting fragmented tables, index fragmentation, high clustering factor tables, session and process mapping, DML lock analysis, DDL lock inspection, active SQL tracking, resource usage statistics, and various performance‑related metrics, helping DBAs diagnose and tune database behavior efficiently.
In this interview, Lu Pengcheng, a platform architect at Mogu Street, discusses the company’s large‑scale e‑commerce architecture, the evolution of its monitoring platform, design choices for high‑availability distributed systems, and future open‑source plans, providing practical insights for engineers and technical managers.
The discussion outlines essential service‑monitoring techniques—including health checks, JVM metrics, traffic and payment ring‑ratio analysis, client‑side exception tracking, third‑party CDN monitoring, alert thresholds, instrumentation via AOP or SDKs, and tooling such as Datadog, Zabbix, and the Elastic stack—to reliably detect and respond to incidents in e‑commerce environments.
This article explains how internet startups can build a robust logging system by defining log levels, essential log fields, best‑practice logging principles, and choosing between simple file logs or an ELK‑based collection pipeline for monitoring, troubleshooting, and analytics.
The article explains how unnoticed database issues often cause system slowness, outlines key diagnostic questions for operations teams, and presents a three‑step approach—discover, solve, prevent—to regularly health‑check and optimize databases for reliable performance.
The CAT real‑time monitoring system, open‑sourced in 2014 for Java applications, combines a lightweight ThreadLocal‑based client SDK, Netty‑driven asynchronous transport, and a highly scalable backend that processes ~100 TB of logs daily across 70 machines, using custom binary serialization, in‑memory modeling, segmented storage with 48‑bit indexing, and hourly aggregation to provide near‑full‑volume fault detection, localization, and performance analysis.
This article explains how the internally built Wonder monitoring system, based on Open‑Falcon, tackles large‑scale operational challenges by offering automated agent updates, customizable metrics, log and port monitoring, persistent alarm storage, enhanced alert content, and comprehensive dashboards for thousands of devices.
This article details Shanda Games' journey in designing and implementing a comprehensive automated operations platform—including installation, deployment, security, client and server updates, data analysis, backup, and monitoring—to efficiently manage hundreds of games across diverse hardware and operating systems.
When a Linux server suddenly spikes in load, this guide shows how to pinpoint the root cause within a minute by running ten key commands that reveal CPU, memory, disk I/O, and network metrics.
This discussion compiles real‑world experiences, security concerns, delivery‑rate comparisons, and operational strategies for SMS and voice verification codes, highlighting multi‑provider setups, template handling, monitoring, and mitigation of spam filtering and abuse.
This guide teaches operations engineers a four‑step “Wang‑Wen‑Wen‑Qie” methodology—observing system health, listening via alerts, questioning changes, and pulse‑checking with profiling tools—to rapidly diagnose and resolve high‑impact incidents while maintaining clear communication and post‑mortem learning.
This article presents the design, implementation, and ongoing improvements of Qunar's network device operations platform, detailing its background, optimization strategies, permission model, automated tasks, monitoring capabilities, and how it enhances operational efficiency while reducing risk.
This guide walks you through registering a WeChat public account, downloading and configuring the WeiXin‑Private‑API, creating Zabbix alert scripts, and verifying that WeChat messages are sent correctly when alerts fire.
When a Linux server’s load spikes, you can quickly pinpoint the root cause within a minute by running ten key commands that reveal CPU, memory, I/O, and kernel‑level metrics, enabling fast, data‑driven troubleshooting.
This article explores how historical monitoring alerts can be analyzed and predicted using linear regression, guiding operations engineers to preprocess data, build regression models, and forecast future alert trends to reduce manual alarm handling and improve system stability.
This guide explains how to interpret Redis INFO output, monitor client connections, memory usage, persistence stats, replication CPU metrics, configure eviction policies, understand the LRU algorithm, and perform performance analysis to keep your Redis instance healthy and efficient.
This article introduces three popular open‑source monitoring solutions—Nagios, Graphite, and Icinga—detailing their core features, architecture components, and typical use cases for server, network, and application performance monitoring.
The article examines common pitfalls and best‑practice solutions for traditional enterprises building e‑commerce platforms that must handle traffic spikes, covering large‑scale query optimization, distributed architecture, database design, service degradation strategies, and comprehensive monitoring and operations.
The article details Meizu's Flyme operations platform evolution—from a single‑cabinet setup in 2011 to a multi‑IDC, 6000‑server infrastructure—highlighting challenges, architectural upgrades, monitoring, cost control, automation, and future high‑availability directions for large‑scale internet services.
This article explains what OpenStack is, outlines its core modules, and details the most important monitoring metrics for Nova, Neutron, Keystone, hypervisors, tenants, and RabbitMQ, helping engineers build a robust, scalable OpenStack monitoring solution.
This article explains how enterprises can overcome manual deployment challenges by standardizing their tech stack, defining mutable and immutable deployment modes, designing an XY‑axis model for system components, and implementing continuous integration, delivery, feedback, and monitoring using tools like Jenkins.
This article examines common causes of Redis data loss, walks through a real‑world incident where 90,000 keys vanished, and provides concrete monitoring, configuration, and operational safeguards to detect and avoid such failures.
This article fills the gap of detailed tutorials by chronologically documenting how the author constructed a Django‑powered operations platform, adding monitoring, dashboards, log viewing, task submission, and various management features, each illustrated with screenshots and brief explanations.
This guide compiles a comprehensive set of MySQL learning materials—including introductory and advanced books, online courses, podcasts, and a curated toolbox of performance, monitoring, client, and migration utilities—helping aspiring DBAs build solid SQL skills and master MySQL operations.
This article explains how to design a complete mobile app monitoring framework that covers problem discovery through key quality metrics and user feedback, systematic log instrumentation, effective issue localization methods, and rapid damage‑control strategies such as cloud‑based feature toggles and hot‑fix mechanisms.
This article details Ctrip's journey building a unified real-time data platform—covering business motivations, architectural requirements, technology choices like Kafka and Storm, implementation of Avro schemas, monitoring, alerting, operational lessons, and future explorations such as Streaming CQL and JStorm.
The article outlines typical mobile app quality problems such as adaptation, user experience, and traffic consumption, discusses their characteristics and impact assessment, and emphasizes the need for a comprehensive monitoring system to quickly detect, locate, and mitigate issues in production.
The article describes how Alibaba's game integration platform achieved business‑oriented high availability by abandoning traditional system‑centric designs and implementing a three‑dimensional architecture that combines clear HA goals, multi‑active deployment, client‑side retries, functional isolation, automated monitoring, and rapid fault recovery, ultimately meeting a 3‑minute issue‑location and 5‑minute business‑recovery target.
This article details the evolution of Taobao's home page over a year and a half, covering its background, migration from PHP to Node, modular architecture, performance tuning, stability mechanisms, and agile operational practices that keep a billion‑scale front‑end service reliable and fast.
Zhang Yue, a Qunar operations engineer, discusses the design, selection, architecture, scalability challenges, visualization, alert strategies, and future plans of the company's in‑house monitoring platform Watcher, highlighting lessons learned from migrating from Cacti to a Graphite‑based, Grafana‑enhanced solution.
The article presents a business‑oriented, three‑layer high‑availability architecture for a large‑scale game access platform, detailing measurable goals, client‑side retry with HTTP‑DNS, functional separation and degradation, multi‑region active‑active deployment, and automated, visual monitoring to achieve rapid fault detection, isolation, and recovery.
