Tagged articles
reliability engineering
48 articles · Page 1 of 1
DeWu shares a five-gate framework for productionizing financial AI agents, covering evidence traceability, deterministic execution, state management, runtime safeguards, and human accountability, validated through shadow acceptance testing with real data but no real consequences.
AI AgentsDeWuFinancial Technology
0 likes · 5 min read

This article details how to build a proactive fault prevention system for 10M QPS architectures by modeling risk, automating change verification, observing leading signals, continuously validating defenses, and layering independent safeguards—moving beyond reactive incident response to prevent failures before they occur.
SLOcanary deploymentcapacity planning
0 likes · 36 min read

This article details a systematic framework for turning post-mortem action items into verified risk reductions, covering eight required fields, single-owner accountability, a five-state tracking machine, risk-based prioritization, layered governance rhythms, three-evidence acceptance criteria, and cross-team dependency management for 10M QPS systems.
acceptance criteriaaction itemsclosed-loop improvement
0 likes · 32 min read

This article details the six-stage evolution of fault postmortems—from blameless culture and data-driven timelines to structured root cause analysis, prioritized improvement loops, and automated platforms—showing how 10M QPS systems transform incidents into reusable systemic assets rather than blame exercises.
MTTDMTTRRoot Cause Analysis
0 likes · 29 min read

This article uses a service release scenario to distinguish DevOps and SRE, explaining how DevOps emphasizes collaborative delivery pipelines and feedback loops while SRE focuses on service-level objectives, error budgets, and eliminating toil through software engineering, showing both perspectives can apply to the same automation.
DevOpsError BudgetSLI
0 likes · 16 min read

This article details how to evolve ad-hoc incident response into a standing war room mechanism for 10M QPS systems, covering six key roles, a full lifecycle with trigger/exit criteria, fixed 10-minute decision cadences, reversible mitigation constraints, evidence-linked tooling, and a three-phase rollout with drills and templates to build organizational muscle memory.
10M QPShigh-scale systemsincident command
0 likes · 28 min read

This article explains why traditional capacity planning fails under real faults, detailing how to build continuous fault injection practices with hypothesis-driven experiments, blast radius control, automated stop conditions, and fault capacity maps to transform theoretical headroom into verified effective throughput at 10M QPS scale.
blast radiuscapacity mapcapacity planning
0 likes · 32 min read

This article presents a systematic approach to building reliable AI agents by classifying faults into API, tool, context, and control-flow layers, detailing detection mechanisms like repeat-call fingerprints and watchdogs, and outlining recovery strategies including exponential backoff, model cascade degradation, tool-error virtualization, and idempotent pre-check-confirm patterns to prevent death spirals and ensure safe retries.
AI AgentsCircuit BreakerError Recovery
0 likes · 10 min read

Waymo’s co‑CEO Dmitri Dolgov explains that while a 12‑person team proved basic autonomous driving in just 18 months, turning the demo into a reliable, city‑scale product required fifteen years of engineering to meet long‑term safety, hardware aging, sensor redundancy, and rigorous simulation and system‑level validation.
Autonomous DrivingPhysical AIWaymo
0 likes · 7 min read

The article dissects why message loss is an end‑to‑end engineering challenge across production, broker, and consumer stages, presents real‑world failure cases at ten‑million QPS, and outlines concrete strategies—ack loops, outbox tables, broker replication settings, consumer handling rules, observability, reconciliation, and SLA‑driven compensation—to evolve from best‑effort to provable, recoverable reliability.
Message QueueOutbox PatternSLA
0 likes · 17 min read

The article analyzes Google’s SRE whitepaper on AI‑driven operations, detailing how generative AI accelerates code production 4‑10×, introduces five SRE AI autonomy levels, three core AI‑ops components, and a safety architecture that decouples decision‑making from execution to prevent catastrophic failures.
AI OpsGoogleOperations
0 likes · 12 min read

Agent systems often repeat the same failures, like missing historical calendar data or miscalculating time zones, but Garry Tan’s Skillify framework turns each error into a testable skill with a ten‑step checklist—including contracts, deterministic scripts, unit and integration tests, LLM evals, resolver checks, DRY audits, smoke tests, and knowledge‑base filing—to make agents structurally unable to repeat mistakes.
AI AgentsLLM Evaluationcontinuous improvement
0 likes · 22 min read

This article examines Tencent Interactive Entertainment's eight‑year journey from a traditional operations team to a 400‑person SRE organization, detailing timeline milestones, the shift in mindset and practices, management challenges, and the broader industry trends driving reliability engineering adoption.
OperationsOrganizational ChangeSRE
0 likes · 13 min read

This article explains how Baidu's SRE team identified the root causes of avalanche failures in massive microservice architectures, modeled system limits with Little’s Law, and implemented engineering practices such as retry budgets, queue throttling, and global TTL controls to achieve self‑healing and eliminate avalanche incidents.
SREavalanche failuremicroservices
0 likes · 9 min read

This article explains how Netflix’s engineering teams shifted incident handling from a centralized SRE function to a company‑wide, engineer‑driven practice by selecting the right tooling, standardizing processes, and reshaping culture, enabling rapid, reliable responses for hundreds of millions of viewers.
NetflixSREcultural transformation
0 likes · 10 min read

This article outlines comprehensive stability assurance mechanisms—including standards, process workflows, the distinction between developers and SREs, personal responsibilities, and practical construction directions—to guide teams in building resilient, high‑availability systems through proactive, daily, and incident‑response practices.
SREprocessreliability engineering
0 likes · 10 min read

Reviewing the major service disruptions—from Alibaba Cloud to OpenAI—this article extracts key SRE lessons such as early disaster‑recovery planning, regular backups, load balancing, real‑time monitoring, performance tuning, and capacity planning, urging enterprises to adopt resilient practices for a more stable future.
OperationsOutage ManagementSRE
0 likes · 6 min read

This article explains how iLogtail, Alibaba Cloud's high‑performance observability agent, tackles reliability challenges at million‑scale deployments through a comprehensive SRE workflow that spans design, development, testing, gray‑release, operations, and continuous customer support, all while leveraging cloud‑native tools and automation.
DevOpsSREcloud-native
0 likes · 31 min read

This article details how Alibaba Cloud's iLogtail agent, serving tens of thousands of hosts and containers, overcomes unique stability challenges through a comprehensive SRE approach that spans design, development, testing, gray‑release, operations, and customer‑support, ultimately boosting reliability and reducing incident rates.
SREcloud-nativeiLogtail
0 likes · 32 min read

This guide presents comprehensive principles, best practices, and techniques for designing, deploying, and maintaining stable distributed systems, covering fault tolerance, monitoring, capacity planning, incident response, and operational reliability to help engineers achieve high availability.
Operationsdistributed systemsreliability engineering
0 likes · 1 min read

This article explains why server reliability matters, clarifies core reliability concepts, outlines key analysis techniques, and presents practical testing and verification methods to help engineers build more dependable server systems.
Operationsperformance testingreliability engineering
0 likes · 3 min read

This article details Bilibili's practical SLO engineering approach, covering foundational components, SLI selection, application and business level SLIs, alerting strategies, SLO‑driven quality operations, and the GOC framework for rapid fault discovery, localization, and recovery, illustrating how reliability is systematically improved.
OperationsSLOreliability engineering
0 likes · 16 min read

The article details how Beijing Mobile’s SRE Smart Operations team applied SRE principles, automation, and cloud‑native tools to transform traditional DevOps into a reliable, scalable operation, highlighting their fault‑prevention, monitoring, incident response, and continuous improvement practices that earned them the 2023 IT Technology Leadership award.
OperationsSREautomation
0 likes · 7 min read

The August 4 2023 Bilibili outage, triggered by automatic back‑origin and domain‑disable policies that flooded the BFS load balancer with traffic, caused widespread white‑screens, but was mitigated within the 1‑5‑10 framework through rapid CDN switching, rate‑limit enhancements, storage backup, and client‑side fallback, illustrating the need for tighter cloud‑edge coordination.
CDNSLBreliability engineering
0 likes · 19 min read

The article analyzes reliability governance challenges in Meituan's billion‑traffic systems, introduces pattern mining as a way to uncover common reliability issues, and presents three concrete case studies—idempotency, dependency, and over‑privilege governance—demonstrating how large‑scale traffic data and environment isolation enable low‑cost, automated reliability solutions.
Idempotencyaccess controldependency-governance
0 likes · 19 min read

In this detailed case study, HuoLala's stability leader shares how a two‑year, zero‑to‑one stability metric framework was designed, implemented, and iterated—covering the why, the pain points, the metric definition process, data collection platform, cultural adoption, and the resulting 78% fault reduction and SLA improvement from three to four nines.
OperationsPerformance Monitoringcase study
0 likes · 18 min read

The article outlines the Agricultural Bank of China's initiative to adopt chaos engineering, describing the challenges of modern distributed systems, the design and capabilities of their in‑house chaos platform, product research, industry comparisons, practical use cases across development, operations and disaster recovery, and future development directions.
Platform Developmentchaos engineeringcloud-native
0 likes · 14 min read

This article explains the fundamentals of Site Reliability Engineering, detailing how SRE combines development and operations to improve stability through metrics like MTBF and MTTR, the roles of SLI/SLO, the VALET selection method, and the practical use of error budgets for quantifying work and guiding alerts.
Error BudgetMTBFOperations
0 likes · 14 min read

This article examines Tencent's end‑to‑end SRE quality‑operation framework built on Service Level Objectives (SLO) and On‑Call, detailing industry background, problem statements, SLO management, On‑Call benefits, product architecture, large‑scale deployment, and future plans for reliability engineering.
Quality OperationsSLOSRE
0 likes · 11 min read

This article outlines Bilibili's end‑to‑end SLO framework, covering metric selection, SLO definition, error‑budget calculation, alerting strategies, operational workflows, and lessons learned from real‑world deployments.
Error BudgetSLOSRE
0 likes · 28 min read

This article explains the role of Site Reliability Engineering (SRE), its responsibilities, how it complements product development, the software lifecycle perspective, and practical approaches to ensure system stability through controllability, observability, and best‑practice implementation.
OperationsSRESoftware Lifecycle
0 likes · 14 min read

The deck outlines post‑incident stability enhancements and the adoption of Service Level Objectives after the “713” fault, detailing failure analysis, reliability upgrades, monitoring practices, and the definition and operation of SLOs to sustain system quality, illustrated through architecture diagrams and reliability metrics.
SLOreliability engineeringsite reliability
0 likes · 1 min read

Bilibili adopted Google‑SRE SLO practices—selecting SLIs, defining availability and latency targets, grading services, and tracking error budgets—but encountered costly grading inconsistencies, hidden error detection, and inaccurate business‑level metrics, leading them to realize SLOs are chiefly valuable for early alerting rather than exhaustive reporting.
Error BudgetOperationsSLO
0 likes · 21 min read

Industry experts from Zhejiang Mobile, Bilibili, and Xiaomi discuss how SRE has evolved in the cloud‑native era, sharing concrete frameworks, observability practices, and cost‑focused platforms while emphasizing stability, metrics, on‑call processes, and the need to adapt Google’s model to real‑world product and operational contexts.
DevOpsSREcloud-native
0 likes · 31 min read

This article explains how integrating FMEA with knowledge graph and AI technologies can enhance product quality and reliability across high‑end manufacturing sectors such as semiconductors, automotive, and medical devices, presenting case studies, standards, and a platform built by Daguan Data.
FMEAmanufacturingreliability engineering
0 likes · 19 min read

This article explains Site Reliability Engineering (SRE), its core concepts such as SLI, SLO, SLA, error budgets, risk analysis, the four golden metrics, and practical steps for developing, piloting, and operating reliable services with monitoring, automation, and post‑mortem practices.
Error BudgetSLISLO
0 likes · 15 min read

Bilibili’s SRE team, confronting rapid growth and complex systems, built a systematic stability operation that includes emergency response, incident handling, on‑call scheduling, and an Event Operations Center platform, using metrics like MTTR, MTTI and AI‑assisted automation to reduce downtime and improve reliability.
BilibiliMetricsOncall
0 likes · 27 min read

The article explains Haro’s approach to system stability—defining high‑availability, key metrics such as SLA, RPO/RTO, MTTR/MTBF, and the 5‑5‑10 rule—while outlining cultural and technical safeguards, full‑team participation, process integration, and incremental tooling to prevent faults and ensure rapid recovery.
MTTRRPORTO
0 likes · 11 min read

This article summarizes Alibaba Cloud Elastic Compute Service's four‑year SRE journey, covering why ECS created its own SRE organization, the five‑layer SRE framework, standards, automation platforms, empowerment practices, and team‑building insights that can guide large development teams toward reliable, high‑availability operations.
SREreliability engineering
0 likes · 24 min read

This article explores the role of Site Reliability Engineering (SRE) as a bridge between product development and operations, detailing its responsibilities, core principles, lifecycle perspective, stability value, and practical frameworks for controllability, observability, and best‑practice implementation to enhance system reliability.
SRESoftware Lifecycleobservability
0 likes · 13 min read

This article outlines a comprehensive SRE fault‑management framework, covering core responsibilities, stability metrics such as MTBF and MTTR, detailed pre‑, during‑, and post‑incident processes, monitoring, capacity planning, disaster‑recovery, error budgeting, organizational support, and future trends like AIOps and chaos engineering.
Error BudgetMTBFMTTR
0 likes · 30 min read

This article shares practical SRE insights from years of experience at Alibaba, covering the right mindset, team responsibilities, systematic monitoring, alert management, fault‑handling processes, and resource control to build resilient, high‑availability systems.
SREreliability engineeringresource management
0 likes · 34 min read

This article explains why Site Reliability Engineering (SRE) emerged, outlines its core responsibilities, required skill set, and how it addresses reliability challenges through decoupling, SLO‑driven monitoring, and scenario‑based drills, while highlighting key observations and focus areas for modern operations teams.
SLOSREmonitoring
0 likes · 13 min read


21CTONov 15, 2019 · Operations
This article presents Google SRE expert Ramón Medrano Llamas’s comprehensive guide on designing, operating, and maintaining large‑scale, highly available software systems, covering SRE fundamentals, daily workflows, scalability strategies, fault‑tolerant architecture, monitoring, and operational best practices.
SREfault tolerancemonitoring
0 likes · 13 min read

Zhang Guanshi, the operations director at Huya Live, shares how his team designs a hybrid‑cloud architecture, implements a six‑pillar reliability framework, and leverages real‑time monitoring, AIOps, and rapid‑recovery tools to maintain stable, low‑latency live video streams for millions of viewers.
Cloud ArchitectureOperationslive streaming
0 likes · 22 min read

This article recounts a securities firm's intensive one‑day system upgrade, detailing the weeks‑long preparation, the meticulous Saturday‑morning execution, and the post‑upgrade verification that together illustrate the critical role of disciplined operations in maintaining ultra‑reliable trading platforms.
IT OperationsTrading Platformcase study
0 likes · 7 min read

This article outlines Alibaba's Global Operations Center (GOC) practices for ensuring stable, high‑performance online services during massive traffic spikes like Double‑11, covering current challenges, the operational assurance framework, best‑practice implementations, and future directions such as automation and AI‑driven monitoring.
reliability engineering
0 likes · 28 min read

This article explains the responsibilities, requirements, and daily work of Google Site Reliability Engineers, contrasts them with Software Engineers, outlines key internal infrastructure components, and discusses the future direction of operations engineering in the cloud era.
GoogleInfrastructureOperations
0 likes · 11 min read
