How Huolala Mastered Hybrid‑Cloud Database Governance: A Year‑Long Platform Journey

Background of Hybrid‑Cloud Database Governance

Huolala, a logistics company operating across many regions, faces a complex technical environment and needed a unified approach to manage databases, message queues, caches, and middleware in a hybrid‑cloud setting.

Challenges

Unfamiliarity with cloud services leading to hidden costs and misconfigurations.

Multiple database and middleware choices without clear management standards.

Difficulty achieving consistent multi‑site, multi‑cloud operations.

Pain Points & Demands

Stability: Frequent incidents (P0‑P3) strained DBA resources.

Developer Efficiency: Manual DBA support slowed development cycles.

Cost: Cloud billing complexity and over‑provisioned resources.

Governance Principles

Do less: eliminate unnecessary database choices.

Define standards: enforce DBA responsibilities and SLA.

Build capabilities: platform‑wide automation and monitoring.

70‑point standard: prioritize core issues, iterate quickly.

Address survival problems first: reduce DBA toil.

Platform Architecture

The platform consists of a micro‑service backend, a unified gateway, and a lightweight frontend for DBA self‑service.

Technology Stack

Languages include Python, Go, and Java; the choice depends on the problem domain rather than superiority.

Key Features

Unified portal for DBA tasks (approval, monitoring, troubleshooting).

Self‑service tools for developers to handle routine operations.

Unified gateway to abstract cloud‑specific differences.

MySQL Platform

Monitoring & Alerts

A health dashboard aggregates ~50 metrics per instance, scoring overall health and providing quick actions (SQL snapshot, kill, throttling).

Operations

Automated task scheduling replaces scattered crontabs, with a central scheduler handling registration, execution, and monitoring of jobs across sites.

DDL & Rollback

Custom integration of gh‑ost with task queues ensures safe online schema changes; a flashback mechanism records binlog positions to generate reverse SQL when needed.

// Default configuration
$redisList = [
    'tcp://127.0.0.1:7000?timeout=3.0',
    'tcp://127.0.0.1:7000?timeout=3.0',
    'tcp://127.0.0.1:7000?timeout=3.0',
    'tcp://127.0.0.1:7000?timeout=3.0',
];
// Bind slots to reduce pressure
$redisList = [
    'tcp://127.0.0.1:7000?timeout=3.0&slots=1-100',
    'tcp://127.0.0.1:7000?timeout=3.0&slots=101-200',
    // ...
];

Redis Platform

Implemented a cluster‑mode deployment with a sidecar Service Mesh proxy to provide unified access, multi‑tenant logical isolation, and real‑time key analytics (big key, hot key, command latency).

Kafka Platform

Built a one‑stop management console covering cluster, topic, and consumer administration, with enhanced lag metrics (time‑based delay) and metadata‑driven resource governance.

Other Middleware (ES, RabbitMQ, Canal)

Unified APIs wrap each component’s native admin interfaces, enabling centralized metadata collection, health monitoring, and self‑service operations.

Developer Self‑Service Portal

Provides query tools for MySQL, Redis, ES, and Kafka, allowing developers to retrieve data without DBA intervention.

From Operations to Operations‑as‑Service

After a year, the platform reduced incident frequency, improved resource utilization (40‑60% cost savings), and transformed DBA work from reactive firefighting to proactive performance and cost optimization.

Reflections on DBA in the Cloud Era

DBAs now focus on business‑level performance, cost, and reliability rather than low‑level infrastructure; cloud services simplify many traditional tasks, but new skills in cloud APIs, automation, and multi‑tenant design are essential.