How HuoLala Achieved Zero‑Fault Peaks: A Blueprint for High‑Load System Reliability

Background

Business peaks cause massive traffic spikes that put huge pressure on systems, similar to fire‑risk periods for a fire department. HuoLala needs extra resources during high‑risk periods to identify and resolve problems, as peak events often coincide with key business goals such as breaking single‑day order records.

Since the second half of 2020, HuoLala has been systematically conducting business‑peak assurance. Over three years the technical team launched 34 peak‑assurance events and has maintained a zero‑fault record since 2021.

1. How to Conduct Business‑Peak Assurance? Key Challenges

1.1 Define Assurance Objectives

The primary goal is to ensure no system failures during peaks. Teams must align with specific business targets (e.g., 1 million vs 2 million orders) and evaluate feasibility and cost. Internal metrics such as SLA performance and incident response time are also critical, as is managing cost efficiency.

1.2 Project‑Management Perspective

After setting goals, the work is planned from a project‑management view. A dedicated assurance project team is formed, an organizational matrix is defined, and a reverse‑schedule is created to ensure all milestones are met. Key actions include establishing the project team, clarifying the organization matrix, aligning the assurance rhythm, gathering business inputs, and delivering and accepting tasks through regular kick‑offs and weekly meetings.

1.3 Technical Assurance Perspective

Risk management is the core: identify risks, eliminate them, and devise solutions. Risks are grouped by subject (external customers, internal teams, third‑party dependencies) and type (capacity, change, link, personnel). Specific measures include:

External customers: Set reasonable rate‑limit thresholds based on system capacity.

Internal: Capacity risk – conduct stress tests, expand capacity, and prepare pre‑plans (marketing data pre‑heat, degradation plans). Change risk – enforce network‑freeze periods and pre‑review core changes. Link risk – strengthen service‑risk governance (timeouts, circuit‑breakers, dependency checks) and conduct attack‑defense drills. Personnel risk – schedule on‑call staff, conduct system inspections, and use a “black‑room” on‑call mechanism.

Third‑party dependencies: Focus on capacity risk and degradation. Ensure early resource reservation, prepare one‑click degradation plans, and notify vendors of peak windows.

1.3.1 Cloud‑Provider Heavy Assurance

Key steps with cloud providers:

Information alignment – ensure all parties understand peak‑assurance importance.

Resource stocking – reserve required cloud resources in advance.

Resource pre‑heat – warm up resources before the peak.

Machine inspection – check physical machines for risks and replace if needed.

Aggregation management – monitor and disperse highly aggregated workloads.

Change notification – request advance notice of any provider changes during peaks.

On‑call support – maintain dedicated communication groups and, for major peaks, request on‑site support.

1.4 Assurance Focus Areas

Project organization: Involve PMO, establish department interface mechanisms, and hold regular progress meetings.

System capacity: Build advance buffers, monitor the ramp‑up phase closely, and plan for worst‑case scenarios (e.g., traffic limiting or functional degradation) to avoid total collapse.

2. How HuoLala Implements the Assurance and Its Effects

2.1 Assurance Strategy

Learn from industry giants (e.g., Alibaba Double 11) to avoid reinventing the wheel.

Localize the approach to fit HuoLala’s order‑matching model, focusing on supply‑demand imbalances during peaks.

Continuously optimize through retrospectives and team motivation.

2.2 Strategy Execution

Early on, HuoLala used a simple checklist, which evolved into a comprehensive framework covering risk prevention and rapid recovery. During peaks, the system expands search radius for unmatched orders, but pure capacity scaling is costly, so the focus shifted to stabilizing the dispatch system and preparing degradation plans (e.g., disabling re‑push logic).

2.3 Implementation Details

Macro planning: A yearly calendar lists all expected peak events, enabling early preparation and reminders.

Project management: Clear responsibility matrices, progress boards, and regular meetings ensure coordination across teams.

Operational morale: A “black‑room” war‑room is set up on peak days with amenities; post‑event ceremonies reinforce team cohesion.

Post‑peak review: Comprehensive retrospectives analyze goal achievement, deep‑dive each assurance sub‑item, collect feedback, and track improvement actions.

Lessons learned: Include non‑R&D changes (operations configs, AB tests) in the change‑freeze scope; recognize services with negative traffic correlation to avoid unnecessary scaling; perform cost‑effective downsizing by selecting optimal machines for removal.

3. Assurance Outcomes

Since 2020, HuoLala has conducted 34 peak‑assurance events and maintained zero faults during peaks for three consecutive years. The overall annual incident count has been decreasing year over year.

Conclusion and Outlook

Future goals focus on cost reduction (resource and labor) and making peak assurance a daily practice, turning risk identification and mitigation into everyday activities to further raise yearly system stability.