How Alibaba Cut Costs by 30% with Cloud‑Native Database Scheduling

Introduction

In 2021, Alibaba's Double‑11 shopping festival marked the second year of fully cloud‑native core applications, focusing on cost reduction and resource utilization while ensuring stability. By mixing exclusive and shared instances on the same nodes, they achieved more than 30% cost savings.

Cloud‑Native Database Management

With the rise of cloud‑native technologies, many applications run on Kubernetes, which is becoming the unified delivery platform. Database products are also moving to cloud‑native, aiming for resource pooling and deep integration with cloud infrastructure. DB instance orchestration is now fully based on Kubernetes, requiring a multi‑cluster scheduling system to overcome single‑cluster size limits.

Instance Deployment Modes

Exclusive ECS Node: each instance occupies a dedicated ECS node, offering strong isolation but limited by ECS resource caps.

ECS Resource Pool: a pool of large‑spec ECS machines is maintained, allowing secondary scheduling across pods for better stability and performance.

For the pool mode, cgroup isolation is used, offering two instance types:

Exclusive: CPU binding via cgroup cpu‑set.

Shared: CPU sharing via cgroup cpu‑share, allowing limited over‑commit.

The underlying ECS pool is split into exclusive and shared resource pools; a node cannot host both exclusive and shared instances simultaneously.

CPUSet and CPUShare Mixed Deployment

To mix exclusive and shared instances without impacting exclusive performance, CPU resources are bound separately using cgroup. In Linux, cgroup cpuset can bind processes to specific CPUs, and Docker supports similar parameters.

--cpuset-cpus=""  CPUs in which to allow execution (0-3, 0,1)
--cpuset-mems=""  Memory nodes (MEMs) in which to allow execution (0-3, 0,1). Only effective on NUMA systems.

Kubernetes introduced the CPU Manager feature (beta from v1.8, stable in later versions) to support cpuset binding. Two policies are available:

none: Default value, no cpuset capability; cpu request maps to cpu‑share, cpu limit maps to cpu‑quota.
static: Grants enhanced CPU affinity and exclusivity for qualifying pods; kubelet assigns a bound cpuset before container start, considering CPU topology.

Static policy manages a shared CPU pool, reserving CPUs via --kube-reserved, --system-reserved, or --reserved-cpus. Only pods with Guaranteed QoS and integer CPU requests receive exclusive CPUs.

Scheduler‑Based Mixed Deployment

Because native kubelet binding is limited, a custom scheduler and extension controller were built. Pods receive an annotation indicating the desired CPU allocation, and the scheduler assigns CPUs accordingly.

annotations:
  alloc-spec: |
    {
      "cpu": "Spread"
    }
  alloc-status: '{"cpu":[0,1]}'

The scheduler supports policies such as Spread, SameCoreFirst, and BindAllCPUs, and enforces anti‑affinity so that the same application runs on at most one node.

resourceSet.spec.scheduleConfig.policy.machineMaxList.N.key = ${APP_NAME}
resourceSet.spec.scheduleConfig.policy.machineMaxList.N.value = 1

Controller

The controller abstracts Kubernetes resources from business logic. For exclusive instances, it updates pod annotations with CPU isolation details and provides an API to temporarily lift CPU limits during peak traffic.

# Annotations required for an exclusive instance pod
annotations:
  cpu-isolate-mode: exclusive
  custom-cgroup-info: '{"engine":{"cpuset":"2-3"}}'
  exclude-reserve-cpuset: "true"
  release-isolate-mode: "true"

cgroup‑Controller

The cgroup‑controller runs as a DaemonSet, mounts the host /sys/fs/cgroup/ into containers, watches pod annotations, and directly modifies cgroup configurations to enforce the requested CPU bindings.

volumes:
  - hostPath:
      path: /var/run/
      type: "Directory"
    name: docker-sock
  - hostPath:
      path: /sys/fs/cgroup/
      type: "Directory"
    name: cgroups

It also maintains a shared CPU pool, recalculating available CPUs whenever exclusive pods are added or removed, ensuring dynamic adjustment without modifying Kubernetes core code.

Results

By mixing exclusive and shared instances with custom scheduling and cgroup control, Alibaba reduced the cost of core database clusters during the Double‑11 promotion by over 30%, while preserving performance for exclusive workloads.