How Kuaishou Boosted Data Efficiency with Apache Hudi: Real‑Time + Offline Solutions

1. Pain Points

Data scheduling starts late and has long compute cycles, causing delayed readiness. Data synchronization involves massive incremental data and costly merge calculations, putting pressure on SLA. Repair back‑fills require only partial updates but the offline warehouse forces full batch rewrites, consuming resources and extending cycles.

2. Why Choose Hudi

Hudi offers rich functionality, good integration with Kuaishou’s architecture, high automation, tight Flink integration, and an active community. It enables real‑time computation with Flink/Kafka and supports CRUD on offline tables, meeting the “fast‑ready, fresh‑state” requirement.

3. How to Use Hudi

Key steps include:

Data shuffling to avoid hotspot skew.

Even partition distribution across nodes.

Shuffle to align partitions, preventing write‑lock contention.

Merge strategy to keep only needed records.

Index lookup to map records to files efficiently.

Discard strategy to skip already‑updated records, improving write throughput.

Hudi stores data in Hive‑compatible partitions (Base on HDFS). Proper primary‑key and partition design, merge and index strategies, and concurrency controls (mod‑based partitioning) reduce data volume and maximize write efficiency.

Monitoring uses a six‑stage pipeline to quickly locate latency or accuracy issues, and a probe compares a sample of business data with Row Tables to ensure >95% consistency.