How HuoLala Built a Resilient Fault‑Drill Platform to Boost System Reliability

Background

With the widespread adoption of micro‑service architecture and containerization at HuoLala, system complexity and inter‑service dependencies have grown exponentially, making any unexpected change potentially catastrophic. To improve fault tolerance and resilience, a fault‑drill system was built to validate stability, locate failures, and enhance emergency response efficiency.

System Overview

The fault‑drill system consists of three subsystems: management, tooling, and operations.

Management subsystem : Defines SOPs for fault drills to standardize operations, mitigate human‑induced risks, and outline response measures.

Tooling subsystem : Addresses five key aspects—prevention, detection, recovery, post‑mortem, and improvement.

Operations subsystem : Establishes evaluation mechanisms, cultural practices, and organizational structures to create a thriving fault‑drill ecosystem.

Tooling Subsystem

Fault‑Drill Platform Architecture

The platform supports global attack‑defense drills and routine fault drills, offering modules such as Application Management, Fault Center, Machine Management, Experience Library, and Operations Statistics.

Fault‑Drill Capability Panorama

The platform supports three scenario categories—attack‑defense, functional (monitoring, loss, plans), and chaos engineering—as well as fault types such as Java applications, middleware, system resources, and business‑scenario‑based faults. Blast‑radius control is achieved through traffic isolation, environment isolation, business identifiers, and hit‑count limits. Chaos engineering provides strong/weak dependency mapping across services.

Fault Center

The Fault Center handles fault orchestration, injection, and recovery, capable of injecting over 1,000 faults per minute across nodes, meeting company‑wide drill requirements.

Blast‑Radius Control

Three isolation strategies are used to limit impact:

Custom identifier isolation : Targets specific traffic flows for selective fault injection.

Canary isolation : Limits faults to a single canary version, using real user traffic for precise testing.

Multi‑lane isolation : Restricts faults to a particular lane, e.g., certain cities or driver groups.

Automation of Drills

To improve ROI, automation replaces manual effort, expands coverage, and enables periodic, goal‑oriented drills. Key automation factors include strict blast‑radius control, service dependency management, fault orchestration (parallel, serial, manual), circuit‑breaker capability, and traffic verification (natural, test, replay).

Management System

Drill Types

Drills are categorized into fault drills, global attack‑defense drills, and chaos engineering.

Drill Process

A high‑quality drill follows planning, execution, recovery, and analysis phases, covering scenario design, fault injection, observation, issue recording, recovery, and post‑mortem reporting.

Operations System

The rollout is divided into three stages: exploration, trial, and normalization.

Exploration : Small‑scale pilots validate processes and platform capabilities.

Trial : Expand pilots to more teams, build a dedicated drill team, and collect data.

Normalization : Full‑scale adoption across all departments with self‑service capabilities.

Operational Data

To date, the system supports over 10 business units, has processed more than 900 scenarios, runs 800+ drills monthly, uncovered 100+ issues, and continuously improves system stability.

Future Outlook

Future plans focus on expanding fault types (including C++ and Go services), reducing drill costs, and enhancing observability with multi‑dimensional monitoring.