This article explains how to programmatically render Prometheus alert charts, upload them to object storage, and embed the images in DingTalk notifications using the Go‑based Promoter tool, including template customization, deployment steps, and core rendering logic.
This article presents a comprehensive overview of StarRocks, covering its key features, deployment challenges, backup and synchronization methods, cluster configuration and upgrade procedures, as well as monitoring and alerting solutions, followed by practical lessons learned from real‑world usage.
This article explains how to detect Redis latency problems, measure baseline performance, monitor slow commands, and address common causes such as network round‑trip delays, fork‑generated RDB snapshots, transparent huge pages, swap usage, AOF settings, key expiration, and big‑key handling, providing practical troubleshooting steps and solutions.
This article details the design, implementation, and performance testing of a scalable red‑envelope service capable of handling up to 100 billion requests, supporting 1 million concurrent users per server, achieving peak QPS of 60 k, and outlines hardware, software, and monitoring strategies.
This guide explains how to create a Spring Boot Admin server, configure a Spring Boot client to register with it, enable Actuator for extended metrics, and view real‑time logs, providing a comprehensive monitoring solution for Java backend services.
To raise front‑end quality, the team implements gray‑release monitoring that triggers log analysis at a 5 % rollout, automatically generates reports within ten minutes, and uses dynamic thresholds and noise‑reduction tactics to detect errors early, enabling rapid rollback or expansion and markedly improving stability and release efficiency.
After a traffic surge exposed severe latency in a 15-year-old multi-service web platform, we used monitoring to discover a DB-connection leak caused by a liveness probe, corrected it, and distilled four practical lessons on latency metrics, tooling, legacy maintenance, and code vigilance.
The article shares a real‑world case study of a legacy multi‑service web platform where traffic spikes exposed DB connection leaks, leading to a 90% response‑time bottleneck, and outlines four key takeaways about tail‑latency metrics, investing in tools and people, actively maintaining legacy systems, and treating every line of code as critical for performance.
The article presents go-monitor, an open‑source Go library that provides lightweight, lock‑free service quality monitoring, automatic analysis, configurable alerts, and flexible reporting for backend applications, complete with installation steps and code examples.
This tutorial walks through SkyWalking, an open‑source APM framework, explaining its features, architecture, how to install and configure the server and agents, persist data with MySQL, enable log collection, perform performance profiling, and set up alerting rules for robust distributed tracing.
This guide provides a comprehensive overview of the Tars microservice framework, covering its core concepts, deployment methods across various environments, configuration management, service discovery, logging, monitoring, and operational features such as gray releases and circuit‑breaker strategies.
This article explains how Zabbix monitoring can be extended with custom shell or Python scripts to gather business-specific metrics, demonstrates a sample script that checks disk usage, and provides three methods to resolve a stale NFS file handle error, including using fuser, process inspection, and forced unmount.
This article explains how to design effective Prometheus metrics for various application types, choose appropriate vectors, labels, and buckets, and offers Grafana tricks for visualizing dimensions and linking tooltips, providing a comprehensive guide for robust observability.
This article presents a concise, step‑by‑step guide to designing clean, consistent RESTful APIs, covering resource naming, URL conventions, HTTP methods, versioning, pagination, field selection, security, monitoring, error handling, and documentation tools, with concrete code examples for each rule.
This article presents a collection of practical Bash scripts for Linux servers, covering DOS attack IP blocking, alert emailing, MySQL backup (single and multi‑loop), Nginx log rotation and analysis, real‑time network traffic monitoring, system initialization, and disk usage checks across multiple hosts.
This article explains how to analyze, diagnose, and optimize performance problems in production business systems, covering the typical causes such as high concurrency, data growth, hardware limits, and environment changes, and detailing practical steps for hardware, OS, database, middleware, JVM tuning, code review, and APM monitoring.
This article presents Meituan's large‑scale Kafka deployment, describing the current state and challenges of massive data ingestion, detailing latency‑reduction techniques, cluster‑level optimizations, SSD‑based caching, isolation strategies, full‑link monitoring, lifecycle management, and future directions for high availability.
The article examines the data‑storage problems caused by rapid microservice growth, explains why traditional Hadoop/Spark stacks were rejected, presents benchmark comparisons that show ClickHouse’s superior performance and compression, and details practical ClickHouse deployment, schema design, sharding, TTL, indexing, and monitoring integrations for real‑time analytics.
This article explains why a well‑designed API error‑code system—using consistent numeric or string codes, clear messages, HTTP‑status‑like segmentation, personalized user messages, and unified handling for monitoring—reduces communication overhead, simplifies maintenance, and improves overall backend reliability.
This article explains how to monitor microservice architectures, describes log, tracing, and metric monitoring, compares open‑source tracing tools, outlines fault‑tolerance strategies such as timeout, rate‑limiting, degradation, async buffering and circuit breaking, and details access‑security mechanisms including gateway authentication, service‑side auth, and OAuth2.0 token flows, while also introducing container technology and its role in microservice deployment.
This article outlines a comprehensive data quality assurance framework for a massive reporting platform, covering the data pipeline architecture, detailed testing methods for timeliness, completeness, and accuracy, as well as application‑level checks, downgrade and backup strategies, and future automation plans.
This article provides a comprehensive guide to microservice monitoring—including log, tracing, and metrics approaches—fault‑tolerance isolation techniques, access‑security mechanisms such as API‑gateway and OAuth2.0, and the role of container technologies like Docker in cloud‑native deployments.
The article explains the concept of white‑screen time, its impact on user experience, and presents multiple detection methods—including Navigation Timing API, MutationObserver, element‑point analysis, and headless‑browser simulation—along with implementation code and a monitoring‑alert architecture for front‑end performance optimization.
This article outlines practical guidelines for designing Prometheus metrics, covering how to define monitoring targets, choose appropriate vectors and labels, name metrics and labels correctly, select histogram buckets, and leverage Grafana features to visualize and troubleshoot data effectively.
This guide walks you through designing a simple front‑back separation microservice architecture, implementing it with Java Spring Boot, deploying multiple instances with Eureka, adding Prometheus‑Grafana monitoring, logging, tracing, flow control, and finally installing Kubernetes using K8seasy and verifying high‑availability across the cluster.
Baidu’s advertising platform computes high‑precision response‑time quantiles at massive scale by intercepting each API call, locally summarizing data with mergeable T‑Digest histograms, periodically uploading compressed, Base64‑encoded summaries to a warehouse where they are merged on demand, enabling low‑latency, cost‑effective percentile analysis with sub‑0.1% error.
This comprehensive guide traces the evolution of open‑source monitoring solutions from the early 2000s to modern cloud‑native tools, comparing their strengths, weaknesses, and ideal deployment scenarios to help IT professionals select the most suitable monitoring product for their infrastructure.
This article introduces essential Redis operational tools, covering the INFO command for monitoring, Prometheus‑based redis‑exporter visualization, the Redis‑shake data migration utility, Redis‑full‑check consistency verification, and the CacheCloud platform for comprehensive cluster management.
This article presents a comprehensive roadmap for adopting microservice architecture, explaining why it is chosen, outlining essential concerns such as Docker, container orchestration, API gateways, load balancing, service discovery, event buses, logging, monitoring, tracing, data persistence, caching, and cloud providers, and recommending popular tools for each component.
This article compares Kubernetes pod monitoring metrics with traditional KVM/VM metrics across CPU, memory, disk, and network, explaining the underlying reasons for the differences and offering guidance on interpreting the data for effective performance troubleshooting.
This guide walks startup leaders through designing and assembling a full backend technology stack—from language and component choices to processes, systems, and deployment tools—providing practical recommendations, diagrams, and best‑practice tips for building scalable, maintainable services.
The article describes how Haodf's SRE team replaced Elasticsearch with ClickHouse to handle massive microservice logs, achieve low‑latency queries, reduce storage costs, and support real‑time monitoring, tracing, and metric analysis through columnar OLAP features, sharding, TTL, and materialized views.
This article presents a comprehensive overview of constructing an intelligent risk control tool system, detailing its evolution from manual processes to automated platforms, describing the core "three‑piece" suite (model, decision, and feature platforms) along with supporting data and monitoring platforms, and explaining the functions and interactions of each module such as data ingestion, feature engineering, automated modeling, decision flow, and real‑time monitoring.
As agile development matures, QA moves beyond functional testing; this article explains test right‑shift in a game project, covering stable release strategies, log‑based monitoring, opinion monitoring, performance monitoring, and risk control mechanisms.
This article describes the background, architecture, and implementation of a centralized database log collection and analysis platform built in 2021, detailing how logs from hosts, containers, and databases are normalized, streamed through Kafka, processed with Flink, stored in Elasticsearch, visualized with Kibana, and extended with alerting and configuration management to improve fault diagnosis and lay the groundwork for future AI‑driven operations.
This guide walks through installing Prometheus and Node Exporter on a Linux server, copying binaries to system paths, configuring Prometheus with scrape jobs for the local node and remote hosts, and running the exporters with specific collector options for system metrics.
This guide outlines why microservice architecture is preferred for large applications, presents a clear learning roadmap, and details each critical concern—such as Docker, orchestration, API gateways, load balancing, service discovery, event buses, logging, monitoring, tracing, persistence, caching, and cloud providers—along with recommended tools.
This article details the background, research, architecture, performance testing, and deployment of a comprehensive monitoring system that leverages Prometheus, Grafana, and M3DB to provide flexible metric collection, automatic dashboard generation, and a custom alerting service for large‑scale business services.
This article explains why microservice architecture is preferred over monolithic applications, outlines a learning roadmap, and details essential components such as Docker, container orchestration, API gateways, load balancing, service discovery, event buses, logging, monitoring, distributed tracing, data persistence, caching, and cloud providers.
This article outlines why microservice architecture is preferred over monolithic designs, presents a clear learning roadmap, and details essential concerns such as Docker, orchestration, API gateways, load balancing, service discovery, logging, monitoring, tracing, persistence, caching, and cloud providers, with recommended tools for each.
This article explains what a distributed database ecosystem is, using TiDB as a case study to detail upstream/downstream tools, backup and monitoring solutions, automation platforms, and emerging projects that together form a comprehensive TiDB ecosystem.
This article explains why alerts and on‑call scheduling are needed, outlines the core principles of an alert scheduling system, describes the architecture evolution from PagerDuty to GoAlert and Notice‑Center, and details the implementation, code snippets, and future outlook for a comprehensive operations monitoring solution.
This article describes how the open‑falcon monitoring system is applied and customized at Automotive Home, covering its architecture, component roles, a comparison with other open‑source solutions, and the enhancements made for service‑tree based dynamic monitoring, alerting, self‑healing, and high‑availability deployment.
This article explains what Spring Boot Admin is, walks through creating a server and client to monitor Spring Boot applications, shows how to configure ports, enable the admin UI, and set up email and custom alert notifications for operational health monitoring.
This article walks through a real‑world live‑streaming service alert where response time and goroutine spikes were traced through Grafana metrics, MySQL/Redis performance, routing logic, and Istio sidecar load, ultimately revealing a mis‑reported Istio metric and a resource‑allocation fix to prevent future jitter.
This article explains how to deploy and configure Filebeat and Graylog for centralized log collection, covering installation methods, configuration files, Docker deployment, input modules, pipelines, and practical examples for efficiently gathering and analyzing logs across multiple environments.
The article shares a senior DBA's journey from early database engine development to modern MySQL operations, outlining practical methods for improving efficiency, automating monitoring, building data‑driven processes, and redefining the DBA role for proactive, high‑impact service delivery.
This article outlines a comprehensive process for analyzing, diagnosing, and optimizing performance issues in production business systems, covering hardware, OS, database, middleware, JVM settings, code inefficiencies, and the role of monitoring tools like APM to pinpoint bottlenecks.
This article describes Beike's real‑time computing team's end‑to‑end practices for collecting and storing Flink metrics, building visual monitoring dashboards, implementing multi‑level alerting, analyzing logs, estimating CPU and memory resources, and deploying Flink on Kubernetes with containerization and storage separation to improve stability, resource utilization, and operational efficiency.
This article traces the 30‑year evolution of system monitoring, explains the differences between monitoring, APM and observability, outlines key practices for building an observability platform, and provides a step‑by‑step guide to implementing Prometheus + Grafana in a cloud‑native environment.
This article outlines a comprehensive approach to diagnosing and optimizing performance problems in production business systems, covering common causes such as concurrency spikes, data growth, and environment changes, and detailing hardware, middleware, database, JVM, code-level analyses, monitoring tools, and APM strategies.
This article explains the architecture and key features of NetEase Cloud's local server‑side recording cluster, detailing how dynamic scaling, multi‑backup high availability, load‑balancing strategies, monitoring, and an embedded registration center enable secure, reliable, and scalable recording for data‑sensitive applications.
This article outlines a comprehensive process for analyzing, diagnosing, and optimizing performance issues in production business systems, covering hardware, database, middleware, JVM tuning, code-level problems, testing limitations, scaling considerations, and the role of APM monitoring.
This article explains Netflix’s home‑grown monitoring system Telltale, detailing its design, multi‑dimensional health‑assessment model, intelligent alerting, integration with Slack, deployment monitoring, and continuous optimization that together keep over a hundred production applications running smoothly.
The article outlines a comprehensive process for diagnosing and optimizing performance problems in production business systems, covering causes such as high concurrency, data growth, hardware constraints, and detailing analysis of hardware, OS, database, middleware, JVM settings, code inefficiencies, and the role of monitoring and APM tools.
This article provides a step‑by‑step tutorial on using Spring Boot Actuator to monitor microservice applications, covering quick setup, essential endpoints such as health, metrics, loggers, info, beans, heapdump, threaddump and shutdown, endpoint exposure configuration, and securing them with Spring Security.
This article details Ctrip's migration from MySQL to TiDB, describing the multi‑data‑center architecture, real‑world use cases such as the international CDP platform and hotel settlement, performance tuning measures, comprehensive monitoring and alerting, auxiliary tools, and future roadmap for the distributed NewSQL database.
This article presents a comprehensive, practical design for an ultra‑large‑scale user center, covering microservice architecture, API separation, token generation with graceful degradation, data‑sharding strategies, password encryption, asynchronous processing, and detailed monitoring to ensure high availability, performance, and security.
This article explains how Shannon's information entropy and NLP are combined in Alibaba Cloud's ARMS intelligent noise reduction to quantify alert uncertainty, filter redundant notifications, and automatically prioritize critical incidents, offering a practical, self‑learning solution for modern monitoring environments.
The article summarizes a GOPS conference presentation by Dingmao Technology on AIOps scenario‑driven construction, detailing challenges, definition of scenarios, technical methods, roadmap planning, and future prospects, while showcasing practical examples and supporting technologies for intelligent IT operations.
This guide explains how to set up a monitoring environment for the DBLE distributed middleware using Zabbix, covering host and software configuration, MySQL master‑slave deployment, DBLE installation, Zabbix script creation, and template configuration with detailed code examples.
This article explains how to use Filebeat together with Graylog to collect, process, and visualize logs from multiple services and environments, covering tool introductions, configuration files, component details, deployment methods, and practical code examples.
This article presents a comprehensive set of design practices for building a highly available, high‑performance, and secure user‑center system that can handle hundreds of millions of users, covering service architecture, API design, token degradation, data sharding, security, asynchronous processing, and monitoring.
This guide explains how to use Filebeat to ship logs to Graylog, covering Filebeat's architecture, configuration files, deployment options with Docker or native packages, Graylog's components and pipelines, and step‑by‑step Docker‑compose setup for a scalable centralized logging solution.
This article explains why slow calls in Kubernetes can jeopardize user experience, project timelines, and system stability, outlines five common causes, introduces the golden‑signal and USE analysis framework, and walks through three practical case studies with step‑by‑step troubleshooting and remediation techniques.
This article describes the architecture, core features, and implementation details of a high‑performance API gateway built with RxNetty, covering request dispatch, conditional routing, API management, rate‑limiting, security policies, monitoring, and future improvement directions.
This article provides a systematic, step‑by‑step methodology for diagnosing Redis latency spikes, covering baseline performance testing, slow‑log analysis, high‑complexity commands, big‑key handling, expiration patterns, memory limits, fork overhead, huge‑page settings, AOF configurations, CPU binding, swap usage, memory fragmentation, network saturation, and practical monitoring tips.
This guide explains why Loki is a lightweight alternative to EFK/ELK, walks through installing Loki and Promtail binaries, configuring them with YAML files, and visualizing logs in Grafana using LogQL, providing a complete end‑to‑end log management solution.
Baidu’s Fengjing monitoring platform tackles the daunting challenge of pinpointing failures in its massive Java‑based microservice ecosystem by employing a non‑intrusive probe that captures log metadata, stores it in a database, and reconstructs full request‑level logs with minimal storage overhead.
This article shares practical experiences and best‑practice guidelines for deploying and operating Prometheus in Kubernetes, covering version selection, inherent limitations, exporter choices, metric design, multi‑cluster scraping, memory and storage planning, GPU monitoring, timezone handling, and alerting considerations.
Qunar’s DBA team overhauled their MySQL monitoring and alert system—originally built on Nagios and NRPE—by integrating a Kafka‑based pipeline, a custom alarm service, and MySQL‑stored alert templates, achieving flexible thresholds, granular silencing, high‑availability processing, and early‑stage intelligent management of alerts, slow queries, and disk space.
The article analyzes the high cost and scalability challenges of traditional log collection pipelines and proposes a streamlined architecture that uses in‑memory buffering, UDP transport, aggressive compression, and ClickHouse storage to achieve massive throughput while drastically reducing hardware and operational expenses.
Learn how to properly configure CPU and Memory requests and limits for Kubernetes Pods, understand QoS classes, manage namespace quotas with LimitRange and ResourceQuota, and monitor resource usage using Prometheus queries and Grafana dashboards to ensure stable, efficient cluster operations.
This article outlines a comprehensive approach to diagnosing and optimizing performance problems in production business systems, covering analysis processes, hardware, OS, database, middleware, JVM tuning, code inefficiencies, and monitoring techniques to identify root causes and improve system reliability.
This guide explains how to collect TiDB metrics via its HTTP monitoring API, preprocess the data into JSON, create master and regular items in Zabbix, and configure triggers using Prometheus‑style expressions to achieve effective TiDB monitoring.
This tutorial explains how to create and configure Grafana Stat panels—including text and background modes, threshold‑based coloring, unit settings, and Markdown/HTML text panels—to visualize metrics such as node uptime, CPU cores, and total memory on a dashboard.
The paper presents a unified alert convergence system that centralizes metric calculation, detection, and alarm handling across monitoring subsystems, employing mechanisms such as convergence, claiming, silencing, escalation, and a Redis‑based delayed queue integrated via Kafka or REST to reduce alarm fatigue, improve MTTA/MTTR, and enable future AI‑driven AIOps.
This article explains why monitoring is essential for production stability, compares white‑box and black‑box approaches, and provides a step‑by‑step guide to deploying Prometheus, configuring scrape targets, using Pushgateway and Alertmanager, and scaling the solution with Thanos in a Kubernetes environment.
This guide demonstrates how to create a Grafana Table panel to monitor server resources, add multiple queries, merge them using the Transform feature, customize fields and units, and organize rows for a comprehensive dashboard view.
This article provides a comprehensive overview of popular Kafka monitoring solutions—including JMX, Kafka Manager (CMAK), Kafka Eagle, and Logi‑KafkaManager—detailing their features, installation steps, configuration examples, and comparative advantages, while also mentioning custom setups using JMXTrans, InfluxDB, and Grafana.
This guide outlines the key Redis monitoring metrics—including performance, memory, basic activity, persistence, and error indicators—explains their meanings, shows how to retrieve them with Redis commands, and provides practical tips for effective performance and health tracking.
This article explains the fundamentals of Site Reliability Engineering, outlines a complete SRE workflow from prevention to post‑mortem, details key availability metrics and golden indicators, examines how technical debt arises and can be mitigated, and describes the tooling and practices needed to keep large‑scale services healthy.
This article explains how to use Zabbix Server 5.4 to monitor TiDB clusters by configuring HTTP agents, converting Prometheus metrics to JSON, creating custom macros, linking TiDB templates, and verifying data collection, while noting version and OS requirements.
The article explains how Alibaba’s QianNiu multi‑platform workbench reduced open‑plugin related user complaints by half through defining open nodes, optimizing long plugin startup chains, building permission‑request loops, establishing comprehensive data‑driven metrics, and creating an open‑experience dashboard that monitors performance, reliability, and user‑perceived issues across mobile and desktop.
This guide walks through manually deploying a Prometheus federation on Kubernetes, covering environment setup with sealos, creating storage classes, persistent volumes, ConfigMaps, StatefulSets, services, applying manifests, and verifying the federation to aggregate metrics across multiple clusters.
This article provides a comprehensive overview of microservices, covering its origin, core architectural principles, key characteristics, advantages and drawbacks, suitable organizational contexts, and essential technical components such as service discovery, gateways, configuration centers, monitoring, circuit breaking, and container orchestration.
This article shares practical Java performance‑optimization techniques, covering UI and non‑UI latency metrics, baseline data collection, bottleneck discovery with tools like Arthas, chronic issue handling, and a comprehensive set of governance measures ranging from network‑level caching to code‑level refactoring, asynchronous processing, and service splitting to achieve stable sub‑200 ms response times.
This guide explains how to define, name, and organize monitoring metrics—covering Google’s four golden indicators, system‑specific measurement objects, vector selection, label conventions, bucket design, and practical Grafana tips—for reliable observability of diverse services.
This article compiles Baidu engineers' QCon 2021 talks, covering micro‑service evolution, large‑scale container elastic scaling, search system elasticity, AI‑chip deployment at massive scale, and cost‑focused monitoring, each with abstracts, outlines and key takeaways for practitioners.
In this detailed account of Huolala's 2021 Cloud Operations Best Practices talk, the company shares its multi‑cloud architecture, service‑oriented governance, capacity‑testing, monitoring, and risk‑prediction techniques that together ensure high‑availability and efficient scaling for its diverse logistics services.
This article provides a detailed analysis of HikariCP’s initialization, core components, startup flow, connection acquisition logic, monitoring metrics, and key configuration parameters, illustrating how Spring Boot 2.x leverages this high‑performance JDBC connection pool and offering guidance for tuning and extending it.
JD.com’s customer service team leverages the open‑source MPP database Doris to power real‑time and offline OLAP dashboards, detailing data ingestion pipelines, full‑link monitoring, dual‑stream high‑availability design, dynamic partition management, multi‑layer caching strategies, and performance optimizations applied during the 2020 11.11 shopping festival.
This article shares a comprehensive guide to building robust logging systems for Node.js backend services, covering log types, storage options, performance considerations, full‑trace design, custom field schemas, integration with cloud log platforms, and practical troubleshooting examples.
This article presents a comprehensive, step‑by‑step guide on improving system reliability through early fault detection, scope reduction, frequency reduction, and rapid incident handling, using real‑world practices from Baidu's commercial hosting platform.
This guide shares practical Linux operations lessons—ranging from cautious command use, rigorous backup habits, and secure SSH configurations to comprehensive monitoring and performance tuning—to help teams avoid costly mistakes and maintain stable, reliable services.
Learn to set up a highly available Alertmanager cluster using the Gossip protocol, covering deduplication, routing, HA architecture, required cluster parameters, systemd service files, and Prometheus integration, with step‑by‑step commands and configuration examples.
This guide walks through the history, core features, installation methods, configuration, PromQL queries, exporter setup, Grafana integration, and alerting with Alertmanager for Prometheus, providing practical commands and examples for building a complete monitoring solution in cloud‑native environments.
The article explains how to properly deal with message‑queue (MQ) outages by first encapsulating MQ operations, then applying degradation tactics such as persisting failed messages to a database, disk, or log, and finally implementing scheduled or manual message‑replay mechanisms while emphasizing monitoring and fallback logic.
This article describes a lightweight, resource‑efficient log collection solution that combines Filebeat agents, optional Logstash aggregation, and Kafka transport, detailing configuration choices, meta‑persistence, back‑pressure mechanisms, monitoring setup, and deployment architecture for reliable at‑least‑once delivery.
After integrating Kubernetes into Probius, this guide shows how to pull pod metrics from Prometheus using the query_range API, process them with a Python client, and visualize CPU, memory, bandwidth, and IOPS data in Probius via ECharts, completing a seamless container‑monitoring feature.
