Tagged articles

Cluster Governance

2 articles · Page 1 of 1

Nov 3, 2023 · Operations

Inside Bilibili’s Kafka: Challenges, Guardian Federation, and Future Automation

The article details how Bilibili operates over 1,000 Kafka nodes across 20+ clusters, outlines the scalability and stability challenges they faced, and explains the design and implementation of their self‑built Guardian federation controller, partition‑level throttling, automatic balancing, multi‑tenant isolation, tiered storage, audit, and automated ops workflows.

AutomationCluster GovernanceKafka

0 likes · 21 min read

Inside Bilibili’s Kafka: Challenges, Guardian Federation, and Future Automation

Bilibili Tech

Nov 3, 2023 · Big Data

Comprehensive Governance and Optimization Strategies for Large‑Scale Kafka Clusters

To tame a petabyte‑scale Kafka deployment of over 1,000 brokers, the team built a Raft‑based federation controller (Guardian) that adds per‑partition I/O throttling, disk‑aware automatic balancing, multi‑tenant isolation, cross‑IDC migration, request‑queue splitting, tiered storage, auditing, and fully automated rolling upgrades, enabling stable, self‑healing operations.

Cluster GovernanceKafkaPerformance optimization

0 likes · 21 min read

Comprehensive Governance and Optimization Strategies for Large‑Scale Kafka Clusters