Bilibili Tech
Jun 4, 2024 · Big Data
Improving Resource Utilization and Isolation in Bilibili Big Data Clusters with the Amiya Over‑commit Component
By deploying the self‑developed Amiya over‑commit component together with kernel‑level cgroup memory isolation, explicit task priorities, OOM‑priority killing, and asynchronous reclamation, Bilibili’s big‑data clusters boosted daily resource utilization by about 15 %, eliminated DataNode OOM kills, cut memory‑reclaim latency to zero, and achieved a further 9 % overall efficiency gain.
Big DataOOM Prioritycgroup
0 likes · 18 min read