This guide walks through manually deploying a highly available Prometheus and Alertmanager stack on Kubernetes using StatefulSets, StorageClasses, and persistent volumes, covering environment setup, RBAC, ConfigMaps, services, node exporters, kube‑state‑metrics, and verification steps.
This article explains the cause of InfluxDB’s max‑value‑per‑tag error when Prometheus remote‑writes high‑cardinality tags, analyzes why the built‑in memory index triggers OOM protection, and presents three practical solutions—including moving indexes to disk, storing high‑cardinality tags as fields, and filtering them before write—to ensure stable monitoring data persistence.
This article introduces a curated list of Kubernetes tools—including cluster deployment solutions, monitoring utilities, CLI helpers, and development aids—explaining how each simplifies container orchestration, enhances DevOps workflows, and empowers engineers to manage, observe, and extend their Kubernetes environments efficiently.
This article shares practical experiences with Prometheus in Kubernetes, covering core principles, limitations, common exporters, metric selection, capacity planning, high‑availability strategies, query optimization, and integration with Grafana, offering actionable guidance for building reliable, scalable monitoring solutions.
This article outlines a comprehensive process for identifying root causes of performance bottlenecks in production business systems, covering hardware, database, middleware, JVM settings, code inefficiencies, and monitoring tools, and provides practical optimization techniques for each layer.
At the 2020 GOPS Global Operations Conference, Tencent’s senior operations engineer Xie Hailin detailed the design and implementation of the Panshi platform—a comprehensive, high‑availability solution that unifies change management, fault handling, continuous operation, and disaster recovery to ensure uninterrupted payment services for billions of daily transactions.
This guide compiles a comprehensive set of Linux shell commands for deleting zero‑byte files, inspecting processes, checking CPU, memory, disk usage, network load, and other system metrics, plus a collection of useful regular expressions for text processing and validation.
This article walks through the evolution of an online supermarket from a simple monolithic website to a fully split microservice architecture, highlighting the pitfalls of ad‑hoc growth, the need for service abstraction, monitoring, tracing, fault tolerance, testing, and the trade‑offs of frameworks versus service mesh.
During JD.com’s 11.11 Global Shopping Festival, the JD Cloud Log Service handled petabyte‑scale log data, delivering real‑time monitoring, cost‑effective storage, high‑availability architecture, circuit‑breaking, rate‑limiting, auto‑scaling and comprehensive dashboards to ensure stable operation of the massive traffic surge.
This article explains how to use Zabbix Agent2 to monitor Redis, covering the plugin's architecture, configuration priority, methods for retrieving INFO, CONFIG, health status, and slow‑query logs, as well as practical steps to set up the Redis template in Zabbix.
The article presents a comprehensive case study of Haodf.com's transition to a micro‑service architecture, detailing the challenges of service stability and observability, the design of a unified governance platform with log‑holographic analysis, real‑time alerts, application profiling, SLO/SLA definition, and future roadmap for capacity and reliability improvements.
This article explains how the JSTracker platform was used to build a comprehensive end‑to‑end front‑end monitoring and data analysis solution that meets the 1‑5‑10 safety production goal—detecting issues within one minute, locating them in five, and fixing them in ten—by improving coverage, subscription, metrics, and gray‑release monitoring for Alibaba’s Double‑11 promotion.
This article outlines how a city‑commercial bank redesigned its monitoring architecture using a closed‑loop “monitor‑manage‑control” strategy, detailing the current challenges, the three‑tier solution, its advantages, and future directions for automated, AI‑enhanced operations.
This guide walks you through installing Prometheus, Grafana, Alertmanager, node_exporter, cadvisor, and blackbox_exporter on CentOS 7, configuring them with Docker or binaries, setting up Prometheus scrape jobs and alert rules, and using a script to add or remove monitored targets.
In this talk, Yun Jin explains how HBase’s petabyte‑scale, horizontally‑scalable architecture—built on Hadoop, HMaster, RegionServers, and Zookeeper—enables e‑commerce platforms to handle extreme promotion‑day traffic by supporting high‑throughput reads/writes, time‑series monitoring, massive order storage, and robust HA, while covering essential table operations, monitoring, and troubleshooting techniques.
The article explains how long‑running database transactions can exhaust connection pools, block threads, and cause widespread service failures, then offers practical strategies—including keeping transactions short, removing RPC calls, enhancing monitoring, and reviewing code—to detect and prevent these high‑risk issues.
This article explains fundamental Elasticsearch concepts such as indexes, index types, and the immutable nature of shard counts, demonstrates how to define custom shard and replica settings when creating or updating indices, and outlines essential monitoring considerations and commands for cluster health.
This article reveals how JD's Zhilian Cloud prepares for the massive 11.11 shopping festival by rapidly mobilizing teams, defining protection scopes, estimating resources, implementing high‑availability across regions and AZs, applying business degradation and elastic scaling, and establishing comprehensive monitoring and rehearsal practices to ensure a smooth, resilient promotion.
This article explores the design and implementation of the Yueying front‑end monitoring SDK, covering its purpose, core design principles, module architecture, reference formats, semantic versioning, key interfaces, testing strategy, and user‑experience enhancements such as quick integration and dynamic sampling.
This article translates and expands on the author’s seven‑year experience with monitoring and alerting, presenting symptom‑based principles, practical guidelines for rule design, incident handling, and operational processes to create a robust, low‑noise alerting system.
This article presents a structured approach to mastering microservices, covering the learning pyramid concept, a detailed learning path with resource collection, and an overview of the six essential components—service description, registry, framework, monitoring, tracing, and governance—along with practical tips and visual diagrams.
This article explains what an SDK is, outlines key design principles such as minimalism, stability, and extensibility, and walks through the practical implementation of Alibaba’s Yueying front‑end monitoring SDK, covering architecture, module division, versioning, core interfaces, testing, and deployment options.
This article explains the typical reasons behind Redis latency spikes—such as complex commands, big keys, concentrated expirations, memory limits, fork overhead, CPU binding, AOF settings, swap usage, and network overload—and provides practical steps and monitoring techniques to identify and resolve each problem.
This article details ZTO Express's journey of adopting Apache Kylin for OLAP, comparing it with Presto, describing platform architecture, performance gains, integration with scheduling and monitoring systems, and the practical optimizations and future plans that enabled sub‑second query responses on massive daily data volumes.
This article explains common JVM problems such as OutOfMemoryError, memory leaks, and thread deadlocks, then introduces core monitoring tools—jps, jstack, jmap/jhat, jstat, and hprof—detailing their syntax, options, and practical examples to help Java developers diagnose and tune production applications.
The article analyzes a persistent active alert observed via Thanos Ruler's HTTP interface, identifies the buffering queue bottleneck as the root cause, and proposes adjusting the maxBatchSize parameter to prevent alert delay and automatic resolution failures.
This guide walks through the inner workings of Zabbix Agent2’s Docker monitoring plugin, detailing how it communicates via the Docker API, the key configuration files, the query mechanism, and how to apply the provided templates to automatically discover and display container and image metrics.
This article systematically explains monitoring fundamentals, the seven core functions of a monitoring system, proper usage practices, common monitoring objects and metrics, the basic data flow, and provides detailed comparisons of three popular open‑source solutions—Zabbix, Open‑Falcon, and Prometheus—to guide informed selection decisions.
This article explores Prometheus as a cloud‑native monitoring solution, covering core principles, limitations, metric selection, exporter consolidation, Kubernetes deployment nuances, memory and storage planning, high‑availability designs, and advanced features like rate calculations, cardinality management, and predictive alerts.
This article explains what Druid Monitor and Druid Admin are, why cluster‑level monitoring is needed in microservice architectures, and provides a complete Spring Cloud starter implementation with configuration examples, code snippets, and usage limitations.
This article explains how Prometheus uses a time‑series database (TSDB) to handle massive monitoring data, detailing its concepts, query examples, storage engine design, indexing mechanisms, and the benefits of pre‑computing expressions for efficient real‑time analysis.
The article presents iQIYI’s service‑maturity model for micro‑services, outlines how scores across development, deployment and operation stages reveal common deficiencies such as code style, testing, gray‑release and alert handling, and recommends concrete optimization practices—including unified coding standards, automated testing, robust rollback, circuit‑breaking, monitoring, and emergency procedures—to raise services to mature, high‑scoring levels.
The article details a real‑world investigation of a Java HTTP service that experienced a sudden QPS increase and response‑time spikes, tracing the issue through database queries, local method latency, CPU load, frequent ParNew GCs, and large response payloads, and presents concrete remediation steps.
This guide surveys over a hundred modern frameworks and tools across fourteen critical backend domains—message queues, caching, sharding, data sync, communication, micro‑services, distributed utilities, monitoring, scheduling, entry proxies, storage, CI/CD, debugging, and local utilities—offering concise recommendations and practical insights for architects and engineers.
Meituan’s AIOps initiative replaces manual rule‑based monitoring with the Horae platform, which automatically classifies time‑series metrics, applies CNN and XGBoost models to detect periodic anomalies, achieves over 90 % precision in production, and paves the way for broader metric types, forecasting, and advanced fault‑localization.
This article explains what etcd is, outlines common failure points, and provides step‑by‑step instructions for collecting etcd metrics via curl, configuring Prometheus scraping, creating alerts, and using Sysdig Monitor to observe key health indicators in a Kubernetes environment.
A concise collection of Linux command‑line snippets helps you query Oracle client IPs, kill specific processes, count connections, summarize traffic, find large files, measure copy time, and monitor CPU and memory usage, all useful for DB and system administrators.
This guide walks through a real‑world call‑center slowdown incident, outlines common fault‑handling techniques, proposes monitoring enhancements, details a comprehensive emergency‑response plan, and introduces intelligent event‑processing concepts to help operations teams resolve outages faster and more reliably.
Performance optimization is a continuous, data‑driven practice that monitors response time and concurrency, applies techniques such as indexing, caching, parallelism, and asynchronous processing, and in Youzan’s live‑streaming product‑detail case reduced bottlenecks by adding multi‑level caches, circuit‑breaker fallbacks, and parallel sub‑task aggregation.
Sampler is a lightweight tool that runs shell commands, visualizes their output, and can trigger alerts; configured via simple YAML, it works on macOS, Linux and Windows, supports various components such as runcharts, sparklines, gauges, and interactive shells for monitoring databases, queues and system metrics.
This guide explains the most common Prometheus Operator metrics for Kubernetes, detailing each metric's purpose, the PromQL expression to monitor it, and the related underlying metrics, helping you fine‑tune alerts and avoid unnecessary noise in your cluster monitoring.
This article explains why Redis can become slow, covering typical latency causes such as high‑complexity commands, large keys, concentrated expirations, memory limits, fork overhead, CPU binding, AOF settings, swap usage, and network saturation, and provides practical troubleshooting steps and best‑practice recommendations.
This article explains Prometheus's built‑in time‑series database (TSDB), covering its concepts, storage configuration, block structure, write‑ahead log, mmap reads, inverted indexing, data compression, and remote storage integration for scalable monitoring.
Version 3.20.09.0 of the open‑source DTLE data‑transfer component for MySQL has been released, introducing replication‑delay and memory‑usage monitoring with Prometheus, providing configuration examples and Docker commands, and fixing incremental serialization, CPU usage, and uppercase‑where clause handling.
The article presents a design for an asynchronous component that is monitorable, fault‑tolerant, and integrates with zero overhead, compares Akka, RxJava, and a custom JUC‑based implementation, and selects the latter—using extended Callables and a CountDownLatch—to track business units, handle timeouts, and provide fallback behavior.
This article explains how to design and implement a Docker container monitoring system using CAdvisor for metric collection, InfluxDB for time‑series storage, and Grafana for visualization, covering deployment, integration, common issues, and practical configuration details.
A comprehensive Q&A with SRE experts explores how Zabbix and Prometheus compare across scalability, storage, alert handling, intelligent monitoring, dashboard design, automation, migration strategies, and performance‑cost trade‑offs for modern infrastructure.
iQIYI’s payment team built a full‑chain load‑testing framework that isolates data, mocks dependencies, constructs realistic multi‑service traffic, and executes protected tests to expose bottlenecks, guide scaling and optimizations, and ultimately ensure reliable payment services during traffic spikes, while planning a unified automation platform.
This article walks through the evolution of an online supermarket from a simple monolithic web application to a fully decomposed microservice architecture, highlighting the challenges of scaling, the need for monitoring, tracing, service discovery, fault tolerance, and the eventual adoption of a service mesh.
The article outlines the complex nature of testing environments, enumerates typical problems such as resource constraints, external dependencies, and service bugs, and presents systematic troubleshooting methods, useful tools, and real‑world case studies to improve reliability and efficiency.
This article examines the complexities of testing environments, outlines typical causes of failures such as resource constraints, external dependencies, and service bugs, and provides systematic troubleshooting methods, useful tools, and real‑world case studies to improve reliability and efficiency.
This article explains how to monitor Kafka consumer groups using the built‑in kafka‑consumer‑groups tool and the Kafka Manager UI, providing commands, field explanations, and setup steps to ensure real‑time data availability for downstream services such as MongoDB or Elasticsearch.
The article explains Linux shared‑memory fundamentals, why it outperforms file‑based IPC, demonstrates the mmap() system call, and walks through a complete Go implementation that creates, synchronizes, reads, and protobuf‑serializes advertising‑tracking metrics in the VCS monitoring platform.
This article introduces JD Cloud’s self‑built time‑series database HoraeDB, explaining its core concepts, typical use cases, architectural layers, high‑performance features, down‑sampling strategies, compression techniques, and stability measures for handling massive, 24‑hour monitoring data at scale.
This article explains ClickHouse's strengths as a high‑performance MPP database, details hardware selection, read/write separation, shard expansion steps, batch‑size tuning, and presents a three‑layer monitoring model, while also describing its practical application in Tencent's game analytics platform.
This article explains why Flink is chosen for online sample generation, describes the end‑to‑end implementation steps—including stream union, state‑timer processing, and output formatting—covers state backend choices, monitoring, validation, fault handling, and platformization for scalable real‑time machine‑learning pipelines.
This article explains the typical reasons Redis latency spikes—such as high‑complexity commands, large keys, concentrated expirations, memory limits, fork overhead, CPU binding, AOF settings, swap usage, and network saturation—and provides practical steps to monitor, identify, and mitigate each issue.
This article details Red Pulse's journey of integrating the NEO blockchain, the security vulnerabilities it faced—from token theft and credential‑stuffing attacks to sophisticated social‑engineering exploits—and the comprehensive technical measures, monitoring tools, and mitigation strategies it implemented to protect its platform and users.
This guide walks through deploying clickhouse-exporter, configuring Prometheus to scrape its metrics, and importing a Grafana dashboard to monitor ClickHouse single‑node or cluster performance, providing a practical monitoring solution for the database.
This article explains the scenarios where MySQL compressed tables are appropriate, describes how to monitor their health using InnoDB CMP tables in information_schema, and provides practical examples of creation, performance comparison, and update/delete operations to illustrate best‑practice usage.
This guide explains the MQTT protocol, shows how to install and run a Mosquitto broker on CentOS, and demonstrates how to collect MQTT messages with a custom Zabbix Agent2 plugin for real‑time monitoring in Zabbix.
This article explains how latency is used as a key indicator for application risk identification, defines slow interfaces, describes why percentile‑based thresholds are preferred over averages, and outlines the architecture, task workflow, and practical optimization strategies for a full‑chain monitoring system in a microservice environment.
This article details the background, performance and stability requirements, strategic approach, and concrete initiatives—including full‑chain load testing, chaos engineering, monitoring, and targeted optimization projects—that were undertaken to boost the performance by over 300% and improve high‑availability of the continuation enrollment platform.
The article outlines G Bank’s transition from a single‑threaded commercial monitoring solution to a self‑developed, open‑source based alert system that leverages Akka for parallel collection, Apache Dubbo for distributed processing, and Apache Ignite for in‑memory storage, achieving million‑level alert capacity, sub‑100 ms latency, and linear scalability.
This guide walks through Prometheus fundamentals, architecture, components, service discovery, Docker-based deployment, exporter integration, Alertmanager configuration, Grafana visualization, PromQL queries, and Consul service discovery, providing a complete end‑to‑end monitoring solution for cloud‑native environments.
This article explains how Suning's big‑data center leverages ClickHouse’s columnar OLAP engine and a full‑chain monitoring platform to achieve real‑time query tracing, slow‑query analysis, cluster health checks, and resource‑level alerts across diverse business scenarios.
From early .Net ESB attempts to a Java‑based CDubbo framework, CTrip details its migration to Dubbo, covering registration, health checks, CAT monitoring, dynamic configuration, SOA compatibility, testing tools, thread‑less execution, performance gains, extensibility, ecosystem integration, and future service‑mesh standardization.
This article explains why consistent Git commit messages matter, presents a detailed commit‑message format with type, scope and subject, shows how to enforce the standard using a webhook that validates messages, monitors large commits, and provides useful statistics for the development team.
The team introduced an Angular‑style Git commit‑message standard—type(scope): subject in Chinese—and built a webhook‑based monitoring service that validates pushes, alerts violations, tracks diff size and deletions, stores metrics, and visualizes compliance, improving traceability, readability, and automated changelog generation.
Grafana’s new Redis Data Source plugin lets DevOps engineers and DBAs seamlessly connect to Redis instances—whether open‑source, Enterprise, or Cloud—visualize time‑series and core data types, run management commands, and build interactive dashboards using Grafana’s transformations and built‑in panels.
This article walks through the transformation of a simple online supermarket from a monolithic design to a fully fledged microservice system, explaining the motivations, architectural changes, component selection, common pitfalls, and best‑practice solutions such as service decomposition, database sharding, monitoring, tracing, service mesh, resilience patterns, and testing strategies.
This article provides a comprehensive guide on using Prometheus for Kubernetes monitoring, covering fundamental principles, exporter selection, Grafana dashboard creation, memory and storage optimization, high‑availability designs, query performance, cardinality management, and integration with alerting and logging systems.
Zhongtong built the ESPaaS platform to automate deployment, unify monitoring, and provide real‑time alerts and diagnostic capabilities for over 40 Elasticsearch clusters, handling petabytes of data with Prometheus, Grafana, and DingTalk integrations while sharing practical lessons learned.
This article introduces Prometheus' query language PromQL, explains instant and range vector selectors, label matching, offset handling, storage design, common functions and aggregation operators, and provides practical advice for efficient querying and avoiding performance issues.
The article details the architecture of 58.com’s SSP advertising platform, identifies three key reliability challenges—data consistency, interface regression, and storage synchronization—and presents a three‑layer testing quality system comprising web‑layer validation, service‑layer automated testing, and data‑layer monitoring with concrete tools and future improvement plans.
This tutorial walks you through installing Zabbix on CentOS 7, covering prerequisite disabling of SELinux and firewalls, adding repositories, installing server, web, and database components, configuring files, securing MariaDB, starting services, and completing the web‑based setup with language customization.
This guide walks through modifying Alertmanager to use a NodePort service, decoding and editing its secret to add custom receivers and a WeChat template, recreating the secret, and extending Prometheus Operator with additional scrape configs for automatic service discovery, including RBAC adjustments and verification steps.
This article explains why Linux's `free` command often reports higher used memory than the RSS values shown by `ps aux`, covering buffer/cache reclaimable memory, slab and page‑table consumption, and provides Bash scripts to accurately calculate total memory usage.
This article traces the evolution of database autonomy from manual, knowledge‑driven operations through tool‑assisted and expert‑level stages to cloud‑native intelligent services, and details Tencent's DBbrain platform, its architecture, performance‑optimization, security, monitoring, cost‑based analysis, and future self‑healing capabilities.
This article introduces four widely used Linux monitoring tools—iotop, htop, IPTraf, and Monit—explaining their features, usage scenarios, and how they help operations engineers diagnose performance issues without a GUI, including real‑time I/O tracking, visual CPU/memory graphs, network traffic analysis, and flexible alerting.
This tutorial walks through installing and configuring Filebeat, Kafka, Logstash, Elasticsearch, and Kibana, detailing version requirements, file permissions, YAML settings, startup commands, topic verification, and how to ingest and visualize log data in Kibana.
An internal Java‑based HTTP service suddenly suffered high latency and timeouts, prompting a systematic investigation that uncovered excessive QPS, frequent ParNew GCs, CPU load spikes, and large response payloads, leading to concrete performance and design improvements.
This tutorial walks you through disabling SELinux and the firewall, adding Zabbix and EPEL repositories, installing Zabbix server, web, and database components, configuring files, securing MariaDB, starting services, and completing the web‑based setup to get a fully functional monitoring system.
This guide walks through installing InfluxDB and Grafana on CentOS, configuring InfluxDB for Flink metrics storage, creating databases and retention policies, integrating the Flink InfluxDB reporter, and building Grafana dashboards to visualize real‑time Flink job metrics.
This article explains how Netflix's performance team uses ten standard Linux command‑line tools to quickly assess system health within the first minute, focusing on error detection, resource saturation, and utilization across CPU, memory, disk, and network to pinpoint performance problems.
iQIYI’s tech product team designed a unified full‑link automated monitoring platform that integrates link, metric, and log collection with deep analysis, enhancing fault localization, performance insight, and scalability across microservices, while addressing limitations of existing tools like ELK, Prometheus, and Dapper.
The article describes the challenges of troubleshooting distributed deployments across many machines and presents a solution built on the ELK stack that centralizes logs from Java and Go services, enabling near‑real‑time search, visualization, and faster issue resolution.
ShenTan is an automated fault‑localization platform for online services that quickly (under five seconds) pinpoints server‑side issues with developer‑level accuracy by aggregating real‑time metrics, applying a decision‑tree model enriched by expert knowledge and dynamic thresholds, and presenting results through an integrated alert and visualization system, while planning broader endpoint coverage and multi‑tenant support.
This article presents ten actionable recommendations—including reverse‑proxy deployment, load balancing, caching, compression, SSL/TLS tuning, HTTP/2 adoption, software upgrades, Linux and web‑server tuning, and real‑time monitoring—to dramatically improve web application performance, often achieving tenfold speed gains.
This guide details how to enable Prometheus metrics on GitLab Runner, configure Prometheus to scrape those metrics, and set up the gitlab-ci-pipelines-exporter with Grafana dashboards to monitor both runner performance and CI/CD pipeline health.
This article chronicles the evolution of the WebMonitor front‑end monitoring system, detailing its three‑tier stack, data pipeline upgrades from raw disk sampling to HDFS and Elasticsearch, extensive collector‑side optimizations, Jetty thread and timeout tuning, and the resulting performance gains that lowered response times from minutes to sub‑second levels.
After two years transitioning from e‑commerce to finance, the team shares practical monitoring strategies for low‑frequency financial services, contrasting e‑commerce traffic‑based methods with finance‑specific challenges, and detailing point‑based metrics, hourly success‑rate alerts, aspect‑oriented exception handling, white‑list filtering, and Sentinel‑based circuit breaking.
This guide explains how to systematically analyze Linux performance problems—including CPU, memory, disk I/O, network, and load—using 5W2H methodology, built‑in monitoring commands, perf, flame‑graph visualizations, and a real‑world Nginx case study to pinpoint and resolve bottlenecks.
This article introduces Qbus, a cloud‑based queue service built on Kafka, covering its architecture, core components such as log collection, SDKs, HDFS persistence, monitoring with Prometheus, business integration methods, use‑case scenarios, and future development directions.
This article describes how our team designed and implemented a comprehensive online monitoring system for construction drawing generation, covering business background, technical architecture analysis, metric definition, monitoring methods, and the resulting dashboards that improve quality, stability, and rapid issue resolution.
This guide walks through installing HAProxy on a Kubernetes master node, compiling and configuring it, adding the HAProxy exporter, creating a ServiceMonitor YAML for the Prometheus Operator, and verifying that metrics are correctly scraped and displayed in the Prometheus UI.
This article shares practical monitoring strategies for financial services with low‑frequency operations, contrasting e‑commerce monitoring methods, outlining the challenges of financial monitoring, and presenting reliable solutions such as success‑rate alerts, aspect‑oriented exception handling with whitelists, and circuit‑breaker degradation using Sentinel.
This article introduces Prometheus, covering its architecture, key features, data model, metric types, installation steps, integration with node_exporter and Grafana, and outlines suitable and unsuitable use cases for this open‑source monitoring system.
This guide walks through creating PrometheusRule resources, deploying Alertmanager via the Prometheus Operator, configuring custom alerting rules, exposing Alertmanager with a Service, and applying custom Prometheus configuration files using Kubernetes secrets and kubectl commands.
This article explains what the Prometheus Operator is, how it extends Kubernetes with custom resources, lists the CRDs it provides, and walks through a complete deployment—including cloning the source, creating a monitoring namespace, applying RBAC, installing the operator, creating a Prometheus instance, configuring ServiceMonitor, and troubleshooting common permission errors—using concrete YAML manifests and kubectl commands.
