Drawing from three and a half years of operations work, this article shares hard‑earned best practices on testing, backups, security, monitoring, performance tuning, and the right mindset to avoid costly mistakes such as data loss, service outages, and security breaches.
The article explains how growing system complexity drives the need for observability, outlines the three pillars of logs, traces, and metrics, compares traditional stability stacks with modern observability, and details OpenTelemetry's design, advantages, and implementation considerations for cloud‑native environments.
Drawing from years of sysadmin experience, this guide lists concrete Linux operational habits—such as rigorous backups, cautious use of rm‑rf, single‑person changes, SSH hardening, firewall rules, monitoring, and disciplined performance tuning—to help teams avoid costly production failures.
The article details how a high‑throughput e‑commerce checkout pipeline achieves stability by combining fine‑grained metrics, custom trace logs, version‑based data validation, and targeted alert rules that detect latency spikes, error‑code surges, and downstream service failures, enabling rapid incident localization and reliable order processing.
This article describes the design, technology choices, architecture, and implementation details of an automated payment channel management system that uses Redis‑based time‑series storage, custom circuit‑breaker logic, and monitoring to achieve fast fault detection, accurate alerting, and future automated failover.
Categraf is an all‑in‑one, Go‑based monitoring agent that consolidates metric, log, and trace collection, offering remote_write support, lightweight deployment, and extensive plugin configurations to replace multiple exporters in Prometheus‑based observability stacks.
The article details the background, architecture, core features, alert governance, trace integration, and cloud‑native evolution of Watcher, Qunar's internally built, highly scalable monitoring platform that unifies application‑level metrics, alerting, and observability across thousands of services and containers.
This tutorial explains how to install the MySQL Exporter, configure Prometheus to scrape MySQL metrics, set up Grafana dashboards for visualization, and define alerting rules, providing a complete end‑to‑end solution for monitoring MySQL databases in production environments.
The article details how a midnight interface timeout in Vivo’s e‑commerce activity system was traced to a logging bottleneck: a synchronous Log4j call blocked all threads while a cron‑driven log‑rotation script copied a 2.6 GB file, and the issue was resolved by switching to asynchronous logging with a non‑blocking appender.
The article details a year‑long refactoring case study of a fast‑iteration web project migrated from Python to Java/Go, describing inherited performance bugs, a prioritized migration plan, monitoring integration, concrete optimizations such as query reduction and cache redesign, and the resulting stability and latency gains, while outlining required developer skills and best‑practice recommendations.
This comprehensive guide explains MongoDB cluster architecture, component roles, common use cases, monitoring commands, essential maintenance operations, data migration steps, troubleshooting of typical production issues, and practical optimization recommendations for high‑performance deployments.
This article introduces Alibaba's Druid database connection pool, explains its core features and filters, shows how to add the Maven starter, configure properties and filters in Spring Boot, demonstrates the built‑in monitoring pages, slow‑SQL logging, Spring AOP integration, and provides methods to remove the default advertisement and retrieve monitoring data via code.
The article details Netflix’s self‑built Telltale monitoring system, explaining how it consolidates data sources, reduces alert fatigue, provides intelligent alerts, and continuously optimizes application health assessment for over 100 production services, ultimately improving operational efficiency and reliability.
This guide explains why Grafana is essential for modern monitoring, outlines its key features, and provides a step‑by‑step tutorial for creating and configuring Grafana dashboards on Alibaba Cloud, including service provisioning, data source integration, and panel customization.
This article walks through a comprehensive front‑end performance optimization process—starting from diagnosing issues with Lighthouse, identifying bottlenecks such as large bundle size and uncompressed assets, applying code splitting, lazy loading, image optimization, CSP, SEO tweaks, and finally setting up continuous monitoring with a custom platform—to achieve a 279% improvement in Lighthouse performance scores and near‑three‑fold speed gains.
This article walks through the challenges of observability for small‑to‑medium companies and provides a detailed, hands‑on guide to installing, configuring, and using the N9e monitoring platform—including architecture options, component setup, and adding data sources—so readers can achieve integrated alerting, metrics, logs, and tracing in a single pane.
This article explains why native Prometheus HA solutions fall short for large, multi‑region clusters and shows how to use Thanos components—including sidecar, query, store gateway, and compactor—to achieve long‑term storage, unlimited scaling, a global view, and non‑intrusive integration with existing Prometheus deployments.
The article examines whether traditional operations positions can become obsolete by analyzing a self‑service platform + COE + Business Partner model, detailing essential monitoring tools, the role of COE specialists, SaaS alternatives, and practical career pathways for newcomers, mid‑level, and senior engineers.
This article, based on a SACC 2022 talk by Huolala architect Cao Wei, explains the principles of bytecode‑enhancement, its practical implementation for large‑scale microservice monitoring, compares enhancement frameworks, shares best‑practice patterns, and explores broader applications such as service‑mesh sidecars.
This article provides a detailed comparison of nine popular log management solutions—Filebeat, Graylog, LogDNA, ELK Stack, Grafana Loki, Datadog, Logstash, Fluentd, and Splunk—covering their core features, pricing models, advantages, and drawbacks to help readers choose the right tool for centralized logging.
This guide explains how to use Linux tools such as htop, top, uptime, sar, strace, free, pidstat, lsof, and Docker commands to monitor CPU, load average, memory, processes, network throughput, socket states, and database connections, helping you diagnose performance issues and map container PIDs to host PIDs.
This article provides a detailed analysis of the OMS community edition 3.3.1 architecture, explains each internal process, describes the migration workflow and monitoring metrics, and offers practical optimization tips to improve data migration performance.
Compass is an open‑source big‑data diagnostic platform developed by OPPO that provides non‑intrusive, real‑time monitoring and root‑cause analysis for offline and streaming tasks on schedulers such as DolphinScheduler and Airflow, covering workflow‑level failures, Spark engine anomalies, resource usage, and offering one‑click reports and extensible rule‑based diagnostics.
This article explains how to correctly obtain CPU utilization inside containers, compares host and container metrics, describes the use of lxcfs and cgroup files (including cgroup V1/V2) for accurate measurement, and clarifies why container statistics omit nice, irq, and softirq fields.
This tutorial shows how to enable Nginx's stub_status module, install the Nginx Prometheus Exporter, configure Prometheus to scrape Nginx metrics, and visualize the data in Grafana, providing a complete end‑to‑end monitoring solution.
This article explains how Python can automate common operations tasks—remote command execution, log parsing, system monitoring with alerts, batch software deployment, and backup/recovery—providing code examples and practical tips to improve efficiency and reduce manual errors.
Drawing from three and a half years of sysadmin experience, this guide shares practical rules for safe online operations, data protection, security hardening, monitoring, performance tuning, and the right mindset to avoid costly outages and maintain stable, secure services.
The article outlines the key competencies DevOps engineers must master—including cloud computing, automation, containerization, CI/CD, monitoring, logging, and infrastructure-as-code—to accelerate, stabilize, and scale software delivery in modern development environments.
This article explains the challenges of monitoring highly distributed applications, why static alert thresholds often fail, and how interval detection using algorithms like Local Outlier Factor can improve alert accuracy while reducing noise across tools such as Grafana, Zabbix, and Open‑Falcon.
This article explains why typical monitoring approaches miss the mark, outlines four root causes of persistent incidents, and introduces the CAR framework—Customer, Application, Resource—to build user‑centric observability that reduces noise, restores trust, and improves reliability.
This article reviews the evolution of monitoring in the cloud‑native era, analyzes Open‑Falcon’s architecture, strengths, and shortcomings, explains why its development hit a bottleneck, and outlines the design principles and features of the Nightingale monitoring system as a modern, open‑source alternative.
This article details Qunar's migration from Prometheus to VictoriaMetrics for large‑scale container monitoring, covering the shortcomings of Prometheus at massive data volumes, the architectural choices made, performance improvements achieved, and future optimization plans.
The article details how Xianyu’s messaging team systematically improved system stability by classifying risks, implementing gray‑release traffic, establishing dedicated monitoring and alerting dashboards, integrating automated regression into CI/CD, and managing strong‑weak dependencies, ultimately reducing online incidents to near zero.
This article explains what logs are, why they are crucial in large‑scale distributed systems, outlines the requirements for effective log operations, reviews common tooling such as ELK, Prometheus and tracing solutions, provides a Go example for batch log retrieval, and shares best‑practice guidelines to achieve observability.
This article explains performance testing fundamentals, why it matters, the specific challenges Yidun faced such as complex execution, human‑dependent monitoring, data isolation, and cost loss, and describes their automated, gradient‑based testing platform with quantified monitoring and future visualisation plans.
This article walks through the evolution of a system’s performance concerns, defines speed and pressure dimensions, explains how to calculate RT, QPS and concurrency, compares QPS with TPS, and provides step‑by‑step methods using tools like Arthas, JMeter and JVM diagnostics to identify and fix CPU, memory and pressure issues before applying layered optimization strategies.
The article outlines a comprehensive high‑availability architecture framework across six layers—development standards, application services, storage, product fallback, operations deployment, and emergency response—detailing design principles such as stateless services, elastic scaling, redundant storage, robust monitoring, gray releases, and chaos engineering to ensure resilient, continuously available systems.
An in‑depth comparison of Prometheus and Zabbix examines their histories, architectures, data storage, scalability, and container support, highlighting Prometheus’s cloud‑native pull model and Go‑based performance versus Zabbix’s mature, relational‑database approach, to help teams choose the right monitoring solution.
This article explains the internal storage layout, on‑disk and in‑memory data structures, and the query execution flow of Prometheus V2, illustrating how blocks, chunks, WAL, indexes and postings are organized and accessed to serve time‑series queries efficiently.
HoloInsight is an open‑source, cloud‑native observability platform derived from Ant Group's AntMonitor, offering integrated log‑based monitoring, business metric analysis, and AI‑driven AIOps capabilities while providing a lightweight, modular architecture and extensive extensibility for modern software stacks.
This article details a comprehensive investigation of a Java backend service suffering from severe OutOfMemoryError due to an unbounded userId list in a count query, describing monitoring findings, heap dump analysis, and practical mitigation steps including request limiting and JVM tuning.
This article explains the architecture of Prometheus V2, detailing its on‑disk block layout, chunk and index formats, the inverted index mechanism, and how queries locate and retrieve time‑series data, while also covering in‑memory structures and practical usage patterns.
The team implemented SLA‑based, node‑level monitoring for mobile push messages—splitting the workflow, measuring latency, blocking volume, and success rates, isolating metrics with Spring AOP, and tracking third‑party vendors—resulting in clear latency standards, doubled peak throughput, faster issue resolution, and improved overall reliability.
Ctrip evaluated and extended OceanBase Migration Assessment tools, built a one‑click migration workflow, implemented comprehensive monitoring and automatic fault‑diagnosis pipelines, and addressed compatibility challenges such as .NET charset issues and Druid parser errors, ultimately achieving a smooth MySQL‑to‑OceanBase transition.
This article explains the principles behind Prometheus alerts, when they trigger, why they sometimes stay silent, and how Alertmanager’s routing tree and notification pipeline work together to manage alert noise, grouping, silencing, and deduplication.
The article examines logging’s critical role in distributed systems, detailing its purpose, severity levels, and value for debugging, performance, security, and auditing, while highlighting challenges of inconsistent formats and traceability, and reviewing observability pillars, ELK and tracing tools, and practical implementation best practices.
This article provides a detailed comparison of nine popular log management tools—including Filebeat, Graylog, LogDNA, ELK, Grafana Loki, Datadog, Logstash, Fluentd, and Splunk—covering their main features, pricing, advantages, and disadvantages to guide readers in selecting the most suitable solution for their needs.
Since launching a systematic stability‑governance program in August 2022, Xianyu’s messaging system has employed gray releases, dedicated monitoring, daily automated regression, dependency reviews and drills, resulting in near‑zero online incidents within six months and demonstrating that continuous, context‑specific measures and vigilant change management are essential for reliable C2C transactions.
This article introduces Uptime Kuma, an open‑source monitoring tool, and provides step‑by‑step instructions for installing it via Docker or manually, configuring monitors, and using its dashboard, highlighting its simplicity, visual appeal, and support for multiple services and notification methods.
This article explains why Kubernetes pods encounter out‑of‑memory errors and CPU throttling, how limits and requests influence resource allocation, and provides practical monitoring techniques using Prometheus and cAdvisor to proactively identify and mitigate these issues before they impact performance or cause pod eviction.
This article explains how to combine Spring Boot with Micrometer, Prometheus, and Grafana for metrics collection and visualization, and provides Maven dependencies, configuration snippets, and Docker commands to deploy a fully monitored backend service using Docker containers.
This article benchmarks Alibaba Cloud Prometheus against the open‑source Prometheus across multiple cluster sizes, churn rates, and query patterns, revealing that while the open‑source version remains stable under light load, its CPU and memory usage grow non‑linearly with high cardinality, whereas Alibaba's managed service delivers higher compatibility, better query performance, and more predictable scaling.
This article provides a comprehensive comparison of nine widely used log management tools, detailing their core features, pricing models, advantages, and drawbacks to help readers make informed decisions when selecting a logging solution for their infrastructure.
This article outlines Vivo's multi‑year journey of designing, evolving, and operating a cloud‑native, AIOps‑enabled monitoring platform that supports tens of thousands of hosts, databases, containers, and services, detailing its architecture, challenges, and future directions for observability and reliability.
This collection presents a range of front‑end development insights, including ByteDance’s monitoring practices, low‑code product considerations, DeepKit’s TypeScript runtime capabilities, emerging 2023 trends, web development forecasts, the need for diverse JavaScript frameworks, markdown‑to‑PPT tools, and Vue 3 table performance optimizations.
This article explains how eBPF technology provides a unified, zero‑intrusion monitoring solution for Serverless applications across any language, detailing its architecture, workflow, and the advantages it brings to cloud‑native operations such as low cost, high performance, and multi‑protocol support.
This article explains the fundamentals of distributed tracing, introduces the Dapper and OpenTracing models, and details SkyWalking's data collection, cross‑process propagation, bytecode enhancement, architecture components, monitoring, alerting, and performance characteristics for microservice environments.
The article outlines a comprehensive stability‑assurance framework for large‑scale promotional events—detailing planning, capacity and pressure‑test rehearsals, strict change‑freeze, internal gray releases, coordinated on‑call response, thorough link and capacity analysis, monitoring, emergency procedures, cross‑team collaboration, external partner coordination, and post‑event review to ensure resilient system performance.
This guide explains essential Redis monitoring metrics, the tools and commands for collecting performance, memory, activity, persistence, and error data, and shows how to use INFO, slowlog, and redis-benchmark to assess and improve database operations.
This article explains how to use Java bytecode instrumentation with ASM and a JavaAgent to automatically monitor, capture, and report exceptions across a backend system, covering exception fundamentals, best‑practice handling, and practical implementation steps.
DynamicTp is a Java library that extends ThreadPoolExecutor to enable dynamic, configuration‑center‑driven adjustment of thread‑pool parameters, real‑time monitoring, alert notifications, and metric collection, providing a lightweight, zero‑intrusion solution for microservice architectures.
This article provides a detailed overview of the essential backend technology stack for startups, covering language choices, components, processes, systems, and cloud services, and offers practical recommendations for selecting databases, messaging, monitoring, CI/CD, and deployment tools to build a robust, scalable infrastructure.
This case study details NetEase Cloud Music's full‑link baseline governance initiative, outlining the challenges of massive data pipelines, the metrics used to measure success, the three‑pronged action plan (infrastructure, task optimization, and standards), and the resulting improvements in availability, resource utilization, and monitoring accuracy.
The article reviews traditional availability metrics such as Success‑Ratio, Error‑Budget, MTTR/MTTF, SLA/SLO, and highlights their limitations, then introduces Google’s User‑Uptime and Windowed User‑Uptime metrics, explains their definitions, challenges, experimental results, and why they provide a more user‑centric view of service reliability.
This article details China Mobile Information's interview about their centralized AIOps platform, the recent excellent‑level assessment by the China Academy of Information and Communications Technology, the system's key modules, future plans, and the broader significance of AI‑driven IT operations.
This article shares practical insights on constructing an O2O industry data platform, detailing user classification, business pain points, and a three‑step strategy—monitoring, analysis, and diagnosis—to extract core metrics, implement tailored reporting, conduct operational and pricing analyses, and drive data‑driven product improvements.
This article provides a detailed comparison of nine popular log management solutions—including Filebeat, Graylog, LogDNA, the ELK stack, Grafana Loki, Datadog, Logstash, Fluentd, and Splunk—covering their main features, pricing models, advantages, and disadvantages to help readers choose the right tool for their needs.
This guide explains why the Alertmanager configuration file in a TiDB cluster is repeatedly overwritten during reloads, analyzes error logs and TiUP documentation, and provides step‑by‑step instructions to edit the topology, set a custom config file, reload the service, and verify the fix.
This article explains how to maintain a DevOps environment by implementing comprehensive monitoring, handling fault detection and performance metrics, automating alerts in a continuously changing cloud landscape, and integrating NOC and MSP practices for 24/7 reliability and efficient incident response.
This article explains how to use MySQL's native SHOW commands and performance_schema tables to collect comprehensive monitoring metrics—including connections, buffer cache, locks, SQL activity, throughput, and slow queries—while minimizing overhead in a single‑node environment.
This article describes Bilibili’s data quality platform, outlining its background, objectives, theoretical models, workflow stages (recording, checking, alerting), DSL for metrics, root‑cause analysis, scheduling strategies, heterogeneous source integration, rule coverage, intelligent monitoring, and future plans to achieve automated, real‑time, high‑reliability data assurance for massive daily workloads.
This article explains what stability assurance is, outlines a systematic workflow—including anomaly identification, monitoring configuration, impact assessment, and solution planning—and provides practical methods such as capacity estimation, traffic limiting, load testing, scaling, and pre‑heating to ensure services remain stable during both daily operations and high‑traffic events.
This article explains how Python can automate common operations tasks—remote command execution, log parsing, system monitoring with alerts, bulk software deployment, and backup/restore—providing code examples for each and highlighting additional tools that help sysadmins improve efficiency and reduce errors.
Loggie is a cloud-native, Go-based log-collection agent that replaces Filebeat and Flume by using a micro-kernel producer-consumer architecture with hot-swappable pipelines, achieving 2 GB/s read speeds, 1.6‑2.6× higher throughput while using only a quarter of the CPU, and providing built-in observability, reliability, and latency monitoring for large-scale enterprise deployments.
This article provides a comprehensive comparison between Zabbix and Prometheus, covering their architecture, data collection, storage, querying, visualization, and alerting capabilities to help enterprises choose the most suitable monitoring solution for their needs.
This article systematically presents the business background, basic monitoring workflow, usage guidelines, OpenTSDB fundamentals, code precision issues, vmonitor collector architecture, old and new system designs, core alerting metrics, demo illustrations, and a comparison with mainstream monitoring solutions, offering insights for technology selection.
This guide explains why MySQL monitoring is critical, defines five key metric dimensions, shows how to collect them with Prometheus and the MySQL Exporter, provides ready‑to‑use alert rules, and walks through the full setup and dashboard creation on Alibaba Cloud.
The article describes a root‑cause localization algorithm implemented in vivo’s monitoring platform that automatically analyzes latency spikes by splitting service timelines, computing variance, clustering results with K‑means, and recursively tracing downstream services, achieving over 85 % accuracy for dependency failures while still requiring human verification and outlining future AI‑driven enhancements.
This article defines enterprise‑grade SaaS, contrasts it with consumer products, and presents a comprehensive framework for product, data, and system stability—including isolation requirements, SLA metrics, risk modeling, mitigation plans, and continuous review—to help SaaS teams deliver dependable services.
The article details the evolution, architecture, capabilities, challenges, and future plans of Vivo's comprehensive monitoring platform, covering its transition from simple Zabbix setups to a cloud‑native, AI‑ops enabled system that ensures service availability across massive infrastructure.
The article outlines vivo’s evolution from simple Zabbix monitoring to a self‑built, unified monitoring platform that now covers infrastructure, containers, databases and user experience at massive scale, integrating AI‑ops, cloud‑native observability and unified alerting to ensure end‑to‑end service reliability and future intelligent, one‑stop monitoring.
This article provides a thorough overview of backend development, covering system design principles such as high cohesion and low coupling, architectural patterns for high concurrency and availability, network communication techniques, common fault scenarios, monitoring and alerting strategies, service governance practices, and deployment workflows.
This article explains how enterprises can design and implement a comprehensive observability platform—covering metrics, logs, tracing, fault response, post‑mortems, testing, capacity planning, and automation—to improve system reliability and user experience.
It introduces a series of front‑end engineering articles covering comprehensive monitoring architecture, CSS‑based click throttling, drag‑and‑drop implementation, 30 essential JavaScript concepts, and a simple responsive data‑dashboard solution, aiming to deepen developers’ skills and showcase resume‑worthy expertise.
This article presents a comprehensive guide to building a front‑end monitoring system—including pain points, error‑reconstruction techniques, data collection methods, performance metrics, user‑behavior tracking, and a modular SDK architecture—illustrated with detailed code examples for Vue, React, XHR, fetch, and cross‑origin handling.
This article systematically explores high‑availability system design from development standards, capacity planning, application services, storage, product strategies, operations deployment, to incident response, presenting key concepts, architectural patterns, and practical guidelines for building resilient services.
Facing hundreds of terabytes of daily monitoring data, Huawei Cloud SRE replaced HBase with the open‑source time‑series database openGemini, conducting extensive write and query performance tests that demonstrated linear scaling, superior query speed, and significant reductions in storage, CPU, and memory usage.
This article details Tencent CDN's challenges and solutions for business continuity, covering bandwidth and device resource constraints, massive request handling, fault‑management lifecycle, automation bottlenecks, and the implementation of AIOps, intelligent alerts, capacity planning, and root‑cause analysis to ensure reliable service.
This article explains how to design effective Prometheus metrics, choose appropriate vectors, labels, buckets, and naming conventions, and offers Grafana usage tricks to help engineers monitor online services, batch jobs, and offline processing systems with clear, actionable insights.
This guide explains Linux performance fundamentals—including throughput, latency, average load, CPU context switching, memory management, swap behavior, and the essential monitoring tools such as vmstat, pidstat, perf, and strace—while providing concrete command examples and troubleshooting steps.
This guide introduces the open‑source nginx‑gui tool, explains its configuration and performance‑monitoring features, and provides step‑by‑step instructions for downloading, setting up, and running the GUI on Linux, including code snippets and screenshots.
This article explains how to replace Zabbix with Prometheus for MySQL monitoring by configuring mysqld_exporter to collect metrics from multiple MySQL instances, details the required user accounts, shows common errors, and provides step‑by‑step solutions including building a newer exporter, adjusting configuration files, and using auth_module for password management.
This article explains how to collect extensive MySQL performance metrics—including connections, buffer cache, locks, SQL status, statement counts, throughput, server configuration, and slow‑query logs—using only MySQL's native SHOW commands and the performance_schema, providing practical code snippets and optimization tips.
After rebuilding its transaction system in 2020, 得物 progressed from the basic CAT monitoring tool to OpenTracing with Prometheus, and finally adopted OpenTelemetry to unify metrics, traces, and logs via a custom vmagent‑Kafka‑Flink pipeline, dynamic sampling, and extensible javaagents, positioning the platform for a performance‑analysis‑driven future.
This article details Qunar's migration from a Prometheus‑based monitoring stack to VictoriaMetrics, describing the limitations they faced, the architectural redesign using vmagent, vmcluster, and vmalert, and the resulting performance improvements and operational benefits for large‑scale Kubernetes environments.
The paper presents the design and deployment of a hundred‑billion‑scale real‑time monitoring platform that meets stringent data‑collection, analysis, storage, alerting and visualization requirements, compares Oceanus + Elastic Stack against a Zabbix‑Prometheus‑Grafana stack, selects the former, and details performance‑and cost‑optimizations that enable massive, low‑latency monitoring while maintaining high availability.
This article provides a comprehensive overview of Zabbix, an open‑source, web‑based monitoring solution, detailing its architecture, key features, monitoring principles, core components, default ports, and step‑by‑step instructions for deploying both the server and agent on Linux systems.
This guide shares practical operations best practices, covering safe online procedures, data protection, security hardening, daily monitoring, performance tuning, and the right mindset to avoid costly mistakes and keep production environments stable and secure.
This article explains how RocketMQ's message lifecycle can be fully observed using OpenTelemetry‑based metrics, covering producer, broker, and consumer stages, and shows practical monitoring, alerting, and troubleshooting practices for cloud‑native deployments.
This article explains how to use the Prometheus HTTP API to query instant and range metrics, interpret the JSON responses, and fetch data programmatically with Python, providing code examples and details on request parameters, error handling, and practical usage.
