How Kuaishou Built a Scalable Big Data Service Platform to Eliminate Redundant Development

Background

Kuaishou is a data‑driven company where data engineers develop high‑quality structured data tables and stable data services delivered via APIs. They face two main pain points: high barriers to developing data services and repeated development of similar services across business lines.

Pain Point 1: High Development Barrier

Data engineers must consider how data is delivered (API vs. table), how services are built (requiring microservice, service discovery, high‑concurrency knowledge), permission and availability concerns, and operational issues such as scaling, migration, and alerts. This demands not only data modeling and SQL skills but also expertise in building high‑availability, high‑performance services (e.g., Java, microservices).

Pain Point 2: Repeated Service Development

Multiple Kuaishou business lines (payment, live streaming, account, etc.) independently synchronize data to databases and caches and build microservices, leading to duplicated effort, wasted resources, and long delivery cycles.

Big Data Service Platform

The platform is a one‑stop self‑service data platform. Users configure data sources, acceleration targets, API types, and test environments; the platform automatically generates and deploys data services, dramatically improving efficiency.

System Architecture

Raw data resides in a Data Lake, is processed into domain‑organized data assets (typically in a data warehouse), then accelerated to high‑speed storage (Redis, HBase, Druid, etc.) before being exposed via various service interfaces.

Key Technology 1: Configuration‑as‑Development

Platform users are either data service producers or consumers. Producers configure data source, acceleration target, API shape, and isolated test environments. After configuration, the platform automatically creates and deploys the service, after which consumers request access via the platform.

Key Technology 2: Multi‑Mode Service Forms

Three API types are provided:

KV API : Simple key‑value lookups supporting millions of QPS with millisecond latency, returning Protobuf structures for easy ORM mapping.

SQL API : Flexible queries based on OLAP/OLTP engines, supporting complex conditions, aggregations, and pagination.

Union API : Fusion API that composes multiple atomic APIs in serial or parallel, reducing latency by avoiding multiple calls.

Key Technology 3: Efficient Data Acceleration

Data assets are ingested from sources (Kafka, MySQL, logs), modeled, and synchronized to high‑speed stores (Redis, HBase, Druid, ClickHouse). Two acceleration methods are used: full‑data acceleration and multi‑level caching. The platform supports massive daily sync volumes (≈1.2 trillion rows, 20 TB).

Multi‑level caching combines Redis, HBase, Druid, ClickHouse, etc., with hotspot caches and configurable compression (ZSTD, Snappy, GZIP) to reduce storage size dramatically.

Key Technology 4: High Availability Guarantees

High availability is achieved through:

Elastic service framework (container cloud with automatic registration and health checks).

Resource isolation by business and priority levels.

Full‑stack monitoring covering data sync, service stability, and data correctness.

Summary and Outlook

Since 2017, the platform supports diverse online scenarios (live streaming, short video, e‑commerce, monetization) and internal systems, handling up to 10 million QPS with millisecond latency. It offers rich API modes, permission control, and an API marketplace.

Future directions include closer alignment with evolving business needs, deeper data‑asset management (registration, tagging, mapping), and evolving toward a unified OneService framework that supports more data sources (wide tables, files, ML models), varied data retrieval methods (sync, async, push, scheduled), and a unified API gateway with integrated access control, rate limiting, and traffic management.