Vivo's Evolution of Large‑Scale Distributed Messaging Middleware Architecture and Practices

Introduction: The talk covers Vivo’s next‑generation data architecture spanning data collection, ingestion, massive computation and distributed storage, focusing on the evolution of its large‑scale distributed messaging middleware.

Business background: Vivo’s data pipeline starts from end‑users, passes through a unified data integration layer, then ETL processing, feeding real‑time and offline warehouses, with Kafka/Pulsar providing high‑throughput, low‑latency, reliable services.

Unified data ingestion evolution: Describes minute‑level failover for HDFS, dual‑link disaster recovery, and the shift from Kafka to the next‑generation Pulsar (Palsar) handling ~2 trillion messages daily.

Kafka deployment analysis: Compares massive single‑cluster versus many small‑cluster deployments, discusses resource isolation, dynamic throttling, traffic and resource balancing, and identifies limitations such as low utilization and slow recovery.

Pulsar core upgrade: Highlights Pulsar’s cloud‑native design with storage‑compute separation, seamless scaling, second‑level failover, intelligent dynamic throttling, and improved client memory isolation.

High‑availability and observability: Details pre‑, during‑, and post‑incident practices, monitoring layers (infrastructure, host, service, user), alert precision, and the observability platform for latency, traffic and GC monitoring.

Future roadmap: Plans for Pulsar’s tiered storage, function computing, SQL integration, full‑link monitoring, and containerized deployment to reduce operational overhead.