This guide explains the concept, design, and step‑by‑step implementation of a service tree in a CMDB, showing how to map departments, products, and services to resources for better visibility, cost tracking, and operational control.
On Feb 23, Weimeng suffered a large‑scale system outage caused by a core operations staff mistakenly deleting production databases, prompting a multi‑day recovery effort with Tencent Cloud support; the article examines the incident’s background, historical parallels, crisis response, and broader operational insights for DevOps and reliability engineering.
Youku’s unified playback testing platform consolidates a modular automation framework, a comprehensive service chain, and a layered platform ecosystem to standardize workflows, support multiple device types, and provide transparent, real‑time monitoring, thereby reducing development complexity and paving the way for intelligent case recommendation and dynamic verification.
This article introduces several MySQL performance‑tuning utilities—including mysqltuner.pl, tuning‑primer.sh, pt‑variable‑advisor, and pt‑query‑digest—explains how to download, install, run them, and interpret their reports to identify configuration issues and optimize database performance.
This article introduces the free, open‑source kmdr CLI tool, explains how to install it with Node.js or use the web demo, and demonstrates its ability to break down complex Linux commands into readable modules, covering a wide range of common utilities.
This guide explains typical Linux terminal pitfalls such as incomplete commands, filename typos, and wrong directories, and provides practical shortcuts like tab completion, history navigation, and quick command substitution to boost productivity for developers and system operators.
Facing the explosion of 5G traffic and billions of daily call records, Jiangsu Mobile’s IT operations team adopted Prometheus as the core time‑series database, designing a high‑availability, low‑latency monitoring platform that captures, stores, visualizes and predicts performance metrics across their massive billing system.
The article analyzes the recent Weimob data‑deletion incident, explaining why recovery is complex, comparing on‑premise, hybrid, and full‑cloud database architectures, and outlining the technical steps and obstacles involved in restoring massive lost data.
During the COVID‑19 pandemic, the mall development team adopted a comprehensive DevOps toolchain—including JIRA, GitLab, Jenkins, Sonar, Docker, and Wiki—to enable end‑to‑end remote development, automated pipelines, and continuous delivery, resulting in improved efficiency, reliable releases, and seamless collaboration.
This article introduces Jenkins infrastructure setup, including installation via Ansible, Puppet, Chef or Docker, outlines management tools such as CLI, REST API, python‑jenkins and Jenkins‑client, describes project creation plugins like Job DSL, Job Builder and Jenkinsfile, and explains system configuration using Groovy scripts and the Configuration‑as‑Code plugin.
After a core operations staff accidentally deleted Weimeng’s production database in February, the platform endured a multi‑day outage, prompting a transparent crisis response, extensive Tencent Cloud support, and a deep analysis of recovery challenges, operational best practices, and the broader lessons for modern DevOps teams.
The article analyzes the recent Weimeng database deletion incident, explains why recovery took 36 hours, and provides practical guidance on backup practices, minimal‑privilege management, and cloud‑based disaster recovery to prevent similar data loss in small and large organizations.
A Chinese SaaS provider suffered a catastrophic data loss when a core operations employee maliciously deleted its production databases, prompting emergency repairs, police involvement, and a multi‑day recovery effort that exposed critical gaps in permission management and backup strategies.
This article explains why monitoring is essential for operations, reviews popular monitoring tools such as Cacti, Nagios, Zabbix, Ganglia, Centreon, Prometheus and Grafana, outlines a six‑layer unified monitoring platform architecture, offers selection guidance for different enterprise sizes, and shares evolution lessons from small to large scale deployments.
This article details how JD.com’s order center migrated its Elasticsearch cluster through multiple architectural stages—initial deployment, isolation, replica tuning, master‑slave adjustments, and real‑time dual‑cluster backup—while addressing data synchronization, scaling, and performance pitfalls to achieve high availability and query stability.
The article details how QQ migrated all of its services to Tencent Cloud, describing the business scenarios, migration timeline, technical approaches, challenges such as cost, security, and performance, and the operational and architectural lessons gained from the full cloud transition.
This guide walks through setting up ELK to collect, parse, and visualize Nginx access logs, covering Logstash configuration, Grok patterns, GeoIP enrichment, Elasticsearch indexing, and Nginx proxy with basic authentication, enabling real-time log analysis and dashboard creation.
The author recounts his first year at Alibaba, detailing the transition from a solo newcomer to leading the OceanBase cloud platform, establishing large‑scale real‑time monitoring with JStorm, evolving team processes, and adapting to Alibaba’s fast‑paced, iterative culture.
This article explains why monitoring is essential for operations, outlines the four‑layer monitoring standard (infrastructure, liveliness, performance, business), breaks down functional modules and data flows, and showcases JD Cloud's practical design, alarm‑convergence project, and future AI‑driven observability directions.
This guide walks through setting up Crowd 3.7.1 with Jenkins 2.220 by creating user directories, groups, and applications in Crowd, then configuring Jenkins to use the Crowd2 plugin for global security and login, including screenshots for each step.
The article explains how focusing on a single critical point—such as the deployment stage—can dramatically accelerate an organization’s end‑to‑end DevOps transformation, illustrated with JD.com’s journey from manual releases to an automated, high‑efficiency continuous delivery platform.
Didi's National Carpool Day on Dec 3 2019 attracted 3.1M passengers; stability ensured via six pillars: organized task force, capacity forecasting and rapid container scaling, comprehensive monitoring with fire‑fighting map, robust contingency platform, strict process standards, and coordinated third‑party preparation.
This article chronicles the design, evolution, and performance evaluation of Bond, a distributed‑lock SDK used internally at Youzan, covering its multi‑phase roadmap, storage choices, timeout‑retry strategies, lock‑TTL recommendations, and practical pitfalls for reliable operations.
This article outlines the design, architecture, and key features of an IBPM workflow platform, detailing its background, core concepts, design principles, extensibility, and future direction for creating a configurable, integrated, and intelligent business process management solution.
This article introduces OpenTSDB, a highly scalable time‑series monitoring system built on HBase, explains its architecture, demonstrates how it solves common monitoring challenges, and shows practical usage examples including data modeling, collector integration, and real‑world deployment insights.
During the 2020 Chinese New Year lockdown, a travel platform mobilized its development, product, and operations teams to rapidly build refund systems, coordinate with suppliers, and ensure continuous online services, showcasing a user‑first, cross‑functional emergency strategy that balanced technical delivery with intense customer pressure.
This article examines the inherent tension between operations and development, explains Google’s error‑budget and SLO approach, and shares practical DevOps, on‑call, automation, and talent strategies that help ops teams improve efficiency while maintaining product reliability.
In early 2020, a small team of Chinese developers swiftly created a COVID-19 travel companion query tool—designing, coding, and deploying a searchable web service within a single day, then scaling it to millions of users using CDN, static site generation, and cloud storage, while emphasizing data accuracy and rapid response.
The article explains the origins, metrics, and test suites of TPC and SPEC benchmarks, describes their various models for CPU, web, HPC and storage performance, shows how to query official results, and notes a promotional bundle of technical e‑books.
Tencent’s QQ service migrated over a million servers to public cloud, detailing comprehensive planning, phased execution, and solutions to security, dependency, disaster recovery, and gray‑scale challenges, while highlighting infrastructure upgrades, database migration, cloud‑native tools, and operational transformations that ensured zero user impact.
This article explains IBM WebSphere Application Server's memory architecture, common causes of Java OutOfMemoryError in WAS, and provides a step‑by‑step guide—including log collection, heap analysis, and preventive measures—to diagnose, resolve, and avoid memory overflow incidents in production environments.
The article explains how a strong, transparent enterprise culture—characterized by trust, respect, and accountability—is the foundational prerequisite for successful DevOps transformation, illustrating key concepts, cultural barriers, and real‑world case studies that show why cultural change must precede technical adoption.
This article details how Ping An Bank tackled rapid growth and complex regulatory demands by creating an automated operations middle‑platform, designing a CMDB with data‑closure and subscription mechanisms, and implementing orchestration, gray‑scale deployment, and high‑risk detection to achieve resilient, scalable infrastructure management.
This guide walks you through preparing scripts, uploading them, making them executable, and creating Jenkins jobs for each module of the multi‑module Mall project to achieve fully automated deployment using free‑style projects and SSH execution.
This article outlines how a major state‑owned commercial bank designed and implemented a DevOps framework—including goals, architecture, the three main pillars of tools, processes, and standards—and shares practical insights, maturity assessment methods, and Q&A for large‑scale financial institutions.
This article examines a real‑world group‑buy scenario, explains why timeout‑prone interfaces need robust retry and idempotency handling, distinguishes read and write timeouts, outlines key idempotency practices for services and messages, and introduces Guava‑retrying and Spring‑retry as elegant solutions.
This article outlines a systematic approach to dependency governance in high‑traffic backend services, covering service classification, rate limiting, Dubbo, HTTP, database, and message‑queue management to enhance availability, reduce failure impact, and improve overall system stability.
This article examines why Java applications experience high CPU usage, covering infinite loops, frequent Young GC, thread count and state, user‑space versus kernel‑space metrics, and provides a step‑by‑step troubleshooting guide illustrated with a real incident.
This guide explains Linux package naming conventions, how to inspect binary dependencies and installed libraries, and provides a comprehensive collection of RPM and YUM commands—including installation, query, verification, removal, and repository configuration—to help administrators manage software efficiently.
The QA team delivered a seamless, globally stable Double 11 “Cat Night” live stream across three apps and dozens of devices by applying client‑ and server‑side stability measures, international latency simulation, IPv6 support, cost‑effective CDN strategies, full‑chain monitoring, and automated asset‑loss safeguards, achieving zero financial loss.
This article explains the origins of Site Reliability Engineering (SRE), outlines the responsibilities of SRE teams, and details Alibaba Cloud’s ECS SRE practices—including capacity planning, performance optimization, full‑stack stability governance, automated release pipelines, on‑call processes, and the core principles and mindset that guide modern SRE work.
This article explains why measurement is essential for management, outlines a step‑by‑step process for collecting and analyzing efficiency metrics, and shows how to turn data‑driven insights into concrete conclusions and actionable improvement plans for software projects.
Implementing an effective CI/CD pipeline involves understanding continuous integration, delivery, and deployment, recognizing their benefits such as faster feedback and early error detection, and following key stages—from commit and build to testing and production deployment—while selecting appropriate tools and practices to streamline software delivery.
The 2019 Enterprise IT Operations Whitepaper, released at the national Operations Conference, systematically examines the definition, value, key capabilities, industry applications, challenges, and future trends of IT operations across telecom, finance, Internet, and manufacturing sectors.
This article explains how focusing on three key questions during incident post‑mortems, balancing business speed with system stability, and establishing clear SLOs can turn failures into actionable improvements and better fault‑tolerance strategies.
This article describes Qunar Ticket’s comprehensive test‑environment governance framework, including the “Mirror‑Inspect” monitoring service, configuration and data synchronization strategies, and automated allocation management, highlighting how these practices reduced environment‑related project delays from up to 20% to below 8%.
At the 2019 Operations Conference in Beijing, China Telecom’s R&D leader detailed the company’s transformation journey, the DICT capability center’s strategy, the Biying cloud platform, and the implementation of an integrated DevOps platform that streamlines end‑to‑end software delivery using containers, CI and security automation.
The article recounts Jiangsu Telecom’s successful Level‑3 DevOps continuous delivery assessment at the 2019 Beijing Operations Conference, highlighting the role of standardization and tooling, sharing interview insights on the intelligent pre‑processing system, and outlining the broader DevOps standard ecosystem in China.
CITIC Bank’s software development center shares how three flagship projects passed the level‑3 DevOps continuous‑delivery assessment, revealing the role of standardization, tool empowerment, agile practices, and container‑based pipelines in accelerating delivery and boosting team morale.
This guide explains the background, key features, architecture, and step‑by‑step deployment of Thanos—including Sidecar, Store, Query, Compact, Bucket, Rule, and Check components—to provide a unified, high‑availability Prometheus monitoring view with unlimited historical data storage using object storage.
This project addresses challenges in brand advertising testing by implementing automated testing, monitoring, and data construction solutions, significantly improving efficiency, reducing manual effort, and enhancing product quality through real-time issue detection and resolution.
This article explains the evolution from traditional DevOps to DevTestOps, detailing continuous testing, the benefits of integrating automated testing into DevOps pipelines, practical implementation steps, and why organizations should adopt DevTestOps to enhance software quality and delivery speed.
In 2019, IDCF organized three in‑depth DevOps case study events where participants analyzed the practices of leading companies such as Facebook, Microsoft, Etsy and others, highlighting hacker culture, automated Windows builds, and a five‑step tech transformation roadmap.
This article examines Baidu’s Noah monitoring and alarm platform, detailing its end‑to‑end fault‑handling workflow, the three‑component architecture, and the practical challenges of deploying AIOps—such as long algorithm iteration cycles, complex alarm management, and alarm storms—while highlighting scalability and commercial considerations.
This article chronicles Ctrip's cloud‑infrastructure journey—from the early OpenStack‑based IAAS platform through containerization with Mesos, the migration to large‑scale Kubernetes clusters, and the adoption of cloud‑native practices that improve resource utilization, deployment speed, and application governance.
CITIC Bank’s DevOps lead Li Hongtao explains how the bank’s new organizational‑level agile practice integrates development and data‑center operations, employs a DevOps capability maturity model, and cultivates agile coaches to overcome transformation challenges, offering a practical roadmap for peers in the banking sector.
This article traces the evolution of call center technology across four generations—from early hotlines using PSTN and PBX, through IVR and CTI innovations, to modern multimedia channels—highlighting key concepts, features, and their impact on operational efficiency and customer service.
This article analyzes the core challenges of building flash‑sale (秒杀) systems—high concurrency reads and writes, strict consistency, and ultra‑high availability—and presents a layered architectural approach covering dynamic/static separation, hotspot optimization, database tuning, and comprehensive high‑availability strategies.
This article explains how to use Zabbix for simple service‑port monitoring, covering installation, host and item creation, trigger and graph setup, and email notification configuration, providing a practical guide for developers who need lightweight operational monitoring without writing custom code.
This article examines common mistakes in measuring R&D efficiency, outlines seven guiding principles, presents case studies from leading tech companies, and shares practical steps and lessons for implementing a comprehensive R&D metrics system.
In this interview, Ming Xiyuan, Technical Solutions Director at iKangSheng, shares practical guidance on selecting and deploying high‑availability MySQL solutions for the financial sector, covering planning pitfalls, hardware considerations, key configuration parameters, and the company’s product roadmap.
This guide walks operations engineers through understanding server bottlenecks, setting up JMeter's distributed load testing environment on Windows and Linux, configuring assertions and variables, analyzing performance results, and monitoring both concurrency and stability tests to ensure reliable scalability.
StackStorm is an open‑source automation platform that integrates existing infrastructure and applications, enabling event‑driven workflows, troubleshooting, auto‑remediation, and continuous deployment through modular components such as sensors, triggers, actions, rules, workflows, and packs, all managed via a web UI, CLI, and REST API.
This article explains the importance of operations maintenance, outlines the challenges of traditional rule‑based anomaly detection, and describes how machine‑learning‑driven AIOps—including feature engineering, unsupervised and supervised models—can provide more accurate, scalable, and automated detection of server anomalies.
On December 5th Alipay experienced a brief outage that sent users into panic, prompting a humorous recount of personal losses, meme images, and a reminder of the critical importance of high‑availability architecture and disaster‑recovery planning for large‑scale financial services.
This article explains why many system alerts are poorly designed, describes the true purpose of alerts as actionable notifications, distinguishes business rule monitoring from reliability monitoring, and presents practical metrics, strategies, and simple anomaly‑detection algorithms to create high‑quality, actionable alerts for reliable operations.
Alibaba’s 2019 Double‑11 “Cat Evening” live show combined a unified codebase across Youku, Taobao and Tmall with synchronized clocks, latency‑measurement devices and SEI‑injected messages to guarantee fair, zero‑loss interactions, while employing dynamic routing, pre‑warming, peak‑shaving, downstream protection and rehearsed contingency plans to handle massive concurrency and ensure stable, high‑quality user experience.
This article details how JD.com’s order center migrated its Elasticsearch cluster from a basic, mixed‑cloud deployment to a real‑time dual‑cluster backup solution, covering each architectural stage, scaling decisions, data‑sync strategies, and the performance pitfalls encountered along the way.
This article explains how Alibaba's Elastic Compute Service (ECS) SRE team tackled massive traffic, database bottlenecks, alert overload, and resource inconsistencies by establishing a full‑stack reliability organization, upgrading core components, automating pipelines, and instituting rigorous monitoring, incident response, and change‑management processes.
The article outlines Alibaba Youku’s end‑to‑end resource‑assurance platform for Double‑11 promotions, detailing automated demand collection, business‑to‑technical metric conversion, single‑machine capacity testing, rapid scaling and emergency borrowing, which together cut manual reviews by 80 % and boosted delivery efficiency tenfold.
This article provides comprehensive best‑practice guidelines for using Redis, covering data classification, key naming, size and connection limits, cache TTL, recommended client‑hash sharding, and a strict list of prohibited commands and operations to ensure performance, reliability, and maintainability.
This guide walks you through a systematic, step‑by‑step process for diagnosing and resolving poor‑performance or failure incidents on Linux servers, covering everything from gathering context and checking who is logged in to inspecting processes, network services, hardware, I/O, logs, cron jobs and application‑level diagnostics.
This step‑by‑step guide shows how to install JDK, set up Elasticsearch via Homebrew, verify its operation, optionally adjust the cluster name, install Kibana, and access the Kibana web UI for managing Elasticsearch data.
This article presents a detailed case study of intermittent container timeout problems in a Kubernetes environment, examining kernel upgrades, NUMA configurations, CPU affinity bindings, kubelet behavior, cadvisor overhead, and hardware faults, and outlines the investigative steps and solutions applied.
This article outlines the evolution from manual to automated operations, then presents a comprehensive set of shell‑script guidelines—including header conventions, formatting, safety checks, variable handling, loop pitfalls, logging, concurrency locks, and risk‑avoidance techniques—to help engineers write robust, maintainable automation scripts.
Over eleven years Alipay’s engineers transformed the daunting Double‑11 traffic surges from chaotic outages into a smooth, scalable system through relentless capacity planning, architectural revolutions, rigorous stress testing, and the adoption of the self‑developed OceanBase database, turning “impossible” goals into everyday reality.
The article examines how Alipay’s financial data would survive a physical destruction of its servers by explaining multi‑site data center architectures, hot and cold backups, power redundancy, fire‑suppression systems, and the role of partner banks in data recovery, highlighting the extensive resilience measures in modern financial infrastructures.
This article explains why Jenkins plugin downloads are slow for Chinese users, introduces the TUNA mirror service, details the signed update‑center.json mechanism, and provides a step‑by‑step solution—including configuration changes and open‑source tools—to achieve fast, reliable plugin installation.
The article explains Chaos Engineering as a discipline for deliberately injecting failures into distributed systems to uncover hidden weaknesses, outlines its five core principles, describes practical implementation steps and real‑world examples from Ctrip, and discusses future directions for reliability engineering.
Implementing an API gateway in a microservice architecture addresses key challenges such as external‑internal isolation, backend security, operational cost reduction, streamlined change processes, and client‑service decoupling, while enabling features like service circuit breaking, gray releases, and online testing to simplify development and improve reliability.
The article explains how financial platforms like Alipay protect user funds through multi‑site data centers, hot and cold backups, power redundancy, fire‑suppression systems, and strict location standards, showing why destroying a single server would not erase all stored money.
The BIM construction‑drawing team built an automated monitoring and validation tool using Spring Boot, REST‑Assured and JIRA APIs, turning a tedious manual bug‑fix workflow into a streamlined process that reduced online drawing‑failure rates from 0.01% to virtually zero.
JD.com's Smart Supply Chain Y Business Management team collaborated across divisions to implement AI‑driven demand forecasting, automated replenishment, micro‑service architecture, and real‑time monitoring, enabling precise inventory control, cost reduction, and seamless 11.11 promotion fulfillment through integrated planning, pricing, and fulfillment innovations.
This guide shows how to create a reusable shell function that retrieves a process ID for a given user and program, demonstrates its usage, and explains each command involved so you can reliably detect when a service stops running.
The article details how 7FRESH's R&D, testing, network operations, and product teams coordinated system capacity expansion, degradation mechanisms, emergency response procedures, extensive performance testing, and 24/7 monitoring to ensure stable and scalable service during the high‑traffic 11.11 shopping event.
The article details JD.com's internally developed distributed search engine and recommendation platform, their new architectures, deep‑learning‑driven ranking and recall models, component‑based deployment, extensive performance testing, and coordinated operations that powered the massive 11.11 shopping event.
Jenkins pipelines, defined as Groovy code, enable version‑controlled, reusable CI/CD jobs for complex multi‑service environments, simplifying repetitive tasks such as URL updates, providing change tracking, and automating builds, tests, and releases across branches via Jenkinsfile.
This article presents ten GraphQL principles—grouped into integrity, agility, and operations—that guide the design, evolution, and secure large‑scale deployment of a unified data‑graph layer, emphasizing a single schema, collaborative implementation, schema registries, performance monitoring, and structured logging.
At the 360 Internet Technology Training Camp, experts detailed how AI-driven AIOps can transform large‑scale operations, covering data collection, model‑based anomaly detection, alert correlation, self‑healing workflows, and visual dashboards, and presented a practical end‑to‑end framework that other companies can adopt quickly.
This guide outlines step‑by‑step methods for migrating a production Redis cluster—including using Redis’s native RDB/AOF backups, JSON‑based redis‑dump, the Alibaba‑maintained RedisShake tool, and Docker‑based cluster setups—ensuring data integrity and minimal downtime during the transition.
GitLab announced a "work country/region lock" that bars Chinese and Russian citizens from receiving offers and restricts employees with customer‑data access from relocating to those countries, sparking heated debate over open‑source non‑discrimination and geopolitical pressures.
In an in‑depth interview, Fang Wei, former assistant general manager of Zhejiang Mobile’s network department, shares his 15‑year journey from B‑domain maintenance to leading DevOps, cloud, and AIOps initiatives, detailing the shift from waterfall processes to agile, micro‑services, containerization, and AI‑driven operations that reshaped the company’s IT landscape.
This article summarizes the keynote on intelligent operations presented at the 13th GOPS Global Operations Conference, covering multi‑dimensional anomaly detection, smart alarm aggregation, the construction of an operations knowledge graph, and AI‑driven root‑cause analysis techniques for large‑scale server environments.
Zhejiang Mobile’s IT department chronicles its journey from a 2015 cloud‑native initiative to a cutting‑edge AIOps transformation, detailing a six‑level NoOps roadmap, digital fault‑governance, middle‑platform consolidation, organizational agility, and measurable operational gains that position it as a telecom industry leader.
The article details JD.com's coordinated effort to prepare for the 11.11 shopping festival by expanding load‑testing scale, improving cross‑team collaboration, and enhancing the ForceBot platform with traffic recording and replay capabilities to achieve more realistic and efficient full‑chain performance evaluations.
This article shares practical Prometheus best‑practice tips, covering the accuracy‑reliability trade‑off, self‑monitoring setups, avoiding NFS storage, pruning high‑cardinality metrics, handling rate‑function traps, alert‑graph mismatches, group_interval effects, and the overarching goal of stable, cost‑effective observability.
The article reflects on the challenges of modernizing aviation IT systems, highlighting safety‑first regulations, the lack of plug‑in architecture, the need for robust load‑balancing and fault‑tolerance, and how data‑driven automation can bridge the gap between strict oversight and efficient operations.
This article examines how Google SRE limits weekly alerts to ten, compares it with typical Chinese internet operations teams, and provides practical strategies—including on‑call scheduling, alert escalation, automation, dashboard optimization, and team management—to dramatically reduce alert volume and improve incident response.
Senior database engineer Yang Jianrong recounts his 2,100‑day daily technical‑sharing journey, emphasizing programmer soft‑skills—steady learning, clear communication, and open mindset—while offering four practical tips on planning, avoiding technical silos, using fragmented time, and engaging communities, and outlining future focus on AIOps, modern languages, and advanced database technologies.
JD.com's Retail Technology and Data Platform orchestrated a full‑chain, four‑entry‑point stress test for the 11.11 shopping festival, introduced an upgraded Forcebot traffic‑recording tool, and implemented a "Quality Month" with ten safeguards to ensure system stability and prevent incidents during the massive sales event.
