kernel overcommit

Bilibili Tech

Jun 4, 2024 · Big Data

Improving Resource Utilization and Isolation in Bilibili Big Data Clusters with the Amiya Over‑commit Component

By deploying the self‑developed Amiya over‑commit component together with kernel‑level cgroup memory isolation, explicit task priorities, OOM‑priority killing, and asynchronous reclamation, Bilibili’s big‑data clusters boosted daily resource utilization by about 15 %, eliminated DataNode OOM kills, cut memory‑reclaim latency to zero, and achieved a further 9 % overall efficiency gain.