Building a Unified Big Data Service Platform at Kuaishou: Architecture, Challenges, and Solutions

Kuaishou, a data‑driven company, faces significant challenges in data development, including high barriers to building data services and duplicated effort across business lines.

Pain Points of Data Development

High threshold for developing data services.

Repeated development of similar data services across different business units.

Data engineers must not only create structured tables but also package them into independent, flexible, high‑availability, and secure data services.

Key Issues to Address

How data is delivered to business (API‑based access rather than raw tables).

Service development requirements (micro‑service knowledge, service discovery, high concurrency).

Permission and availability concerns.

Operations such as scaling, migration, deprecation, and alerting.

The solution is a unified data‑service platform that enables "configuration‑as‑development".

Platform Overview

The platform provides a one‑stop self‑service portal where users can configure data sources, acceleration targets, API shapes, and test environments. After configuration, the system automatically generates and deploys the data service.

System Architecture

Raw data resides in a Data Lake, is processed into topic‑based data assets, then accelerated to high‑speed storage (Redis, HBase, Druid, ClickHouse) before being exposed via RPC or HTTP APIs.

Key Technology 1: Configuration‑as‑Development

Define data source.

Specify acceleration target.

Choose API type and access method.

Set up isolated test environments.

After configuration, the platform auto‑generates and deploys the service, after which consumers request access permissions and invoke the service.

Key Technology 2: Multi‑Mode Service Forms

KV API : Simple key‑value lookups, supporting millions of QPS with protobuf responses.

SQL API : Complex, flexible queries built on OLAP/OLTP engines, supporting pagination and aggregation.

Union API : Combines multiple atomic APIs in serial or parallel fashion, reducing latency.

Key Technology 3: Efficient Data Acceleration

Full‑data acceleration: ingest raw data (Kafka, MySQL, logs), model it, and sync to fast stores (Redis, HBase, Druid).

Multi‑level caching: hot‑data caches on top of slower stores, with configurable compression (ZSTD, SNAPPY, GZIP) reducing storage by up to 90%.

Key Technology 4: High Availability Guarantees

Elastic service framework built on Kuaishou's container cloud with automatic service registration and health‑based removal.

Resource isolation by business and priority to limit fault impact.

Full‑link monitoring covering latency, QPS, CPU, memory, and service health.

Summary and Outlook

Since 2017, the platform supports live streaming, short video, e‑commerce, and internal systems, handling up to 10 million QPS with millisecond latency. Future work includes expanding data source support, richer data retrieval modes, and a unified API gateway with integrated permission, rate‑limiting, and traffic management.