Design and Implementation of ByteDance Wallet Asset Platform for the 2022 Spring Festival Activity Across Eight Apps

1. Background & Challenges & Goals

The ByteDance Open Platform – Wallet team was responsible for the reward flow (deposit, display, and usage) of the 2022 Spring Festival activity across eight apps (Douyin, Douyin Volcano, Douyin Lite, Xigua, Toutiao, Toutiao Lite, Tomato Novel, Tomato Listening). The activity involved multiple game modes, diverse reward types, and required high‑throughput processing (up to 3.6 million QPS).

1.1 Business Background

Support for eight apps: Users could participate on any app and redeem rewards on any other.

Varied game modes: Card collection, friend‑page red‑packet rain, red‑packet rain, card‑draw lottery, fireworks gala.

Multiple reward types: Cash red‑packets, video subsidies, ad coupons, e‑commerce coupons, payment vouchers, financial vouchers, insurance coupons, credit‑card discounts, tea coupons, movie tickets, Dou+ coupons, cultural‑creative coupons, avatar accessories, etc.

1.2 Core Challenges

Designing a high‑traffic reward deposit and display solution (estimated 3.6 M QPS).

Handling dozens of reward types and multiple downstream systems.

Ensuring system stability, user experience, fund safety, and operational capabilities.

1.3 Final Goals

Reward Deposit: Provide a unified interface for reward ingestion, hide downstream differences, support idempotent processing, error handling, and budget control.

Reward Display/Usage: Build an activity wallet page that shows rewards, allows cash withdrawal, and supports coupon/asset usage.

Foundational Capabilities: SDK for balance query, cumulative income, and reward‑existence checks. Budget control integrated with upstream allocation algorithms. Graceful withdrawal scaling during peak periods. Operational configuration for announcements, batch re‑issuance, and emergency handling. Stability measures (resource scaling, rate limiting, circuit breaking, degradation, fallback, isolation). Fund safety via idempotency, reconciliation, monitoring, and alerts. Data isolation for internal testing, gray‑scale rollout, and production phases.

2. Product Requirements

Users could earn rewards on any app; the cash‑red‑packet flow example includes login → activity participation → wallet page → withdraw button → withdrawal page → result page.

Core reward scenarios:

Card Collection: Issue various coupons and large cash red‑packets during draws.

Red‑Packet Rain: Issue cash, coupons, and video subsidies (up to 1.8 M QPS each).

Fireworks Gala: Issue cash, coupons, and avatar accessories.

3. Wallet Asset Platform Design & Implementation

The wallet asset platform handles reward issuance and display. It consists of an Asset Order Layer and an Activity Wallet API Layer.

3.1 Overall Architecture

3.2 Asset Order Center Design

Core issuance model:

Key points:

Unique activity ID (actID) generated by the ByteDance account platform, shared across apps.

Scene ID maps one‑to‑one with a reward type, enabling configurable budget control, rate limiting, and reconciliation.

Order number format: ${actID}_${scene_id}_${rain_id}_${award_type}_${stage} ensures idempotent processing.

4. Core Difficulty Solutions

4.1 Multi‑App Reward Data Interoperability

Different apps have separate account systems; a unified actID (phone‑number prioritized) enables reward data binding across all eight apps.

4.2 High‑Traffic Reward Deposit

Cash red‑packet issuance can reach 1.8 M TPS; challenges include fund safety, user experience, and cost control.

Two traditional schemes:

Scheme

Implementation Idea

Advantages

Disadvantages

Synchronous Deposit

Allocate compute and storage equal to estimated traffic.

Simple development; low error rate.

Excessive storage cost; e.g., 180 w QPS would need >1,100 DB instances.

Asynchronous Deposit

Allocate partial resources; actual capacity may lag.

Simple; low error; resource‑efficient.

Significant user‑experience delay (minutes).

The chosen solution combines asynchronous processing with a lightweight token mechanism.

4.2.1 Red‑Packet Token Scheme

Each issued red‑packet generates an encrypted token (asymmetric encryption) containing amount, actID, timestamp, etc., stored both client‑side and server‑side. Tokens are tracked in Redis; the wallet page aggregates settled tokens and pending tokens to present a consistent balance. When a user initiates withdrawal, pending tokens are forced to settle, guaranteeing correct balance.

Token protobuf definition (kept unchanged):

// Red‑Packet token structure
type RedPacketToken struct {
    AppID      int64  `protobuf:"varint,1,opt" json:"AppID,omitempty"` // App ID
    ActID      int64  `protobuf:"varint,2,opt" json:"UserID,omitempty"` // ActID
    ActivityID string `protobuf:"bytes,3,opt" json:"ActivityID,omitempty"`
    SceneID    string `protobuf:"bytes,4,opt" json:"SceneID,omitempty"`
    Amount     int64  `protobuf:"varint,5,opt" json:"Amount,omitempty"`
    OutTradeNo string `protobuf:"bytes,6,opt" json:"OutTradeNo,omitempty"`
    OpenTime   int64  `protobuf:"varint,7,opt" json:"OpenTime,omitempty"`
    RainID     int32  `protobuf:"varint,8,opt,name=rainID" json:"rainID,omitempty"`
    Status     int64  `protobuf:"varint,9,opt,name=status" json:"status,omitempty"`
}

Token lifecycle: created → processing (status 2) → success (status 8). Idempotent order numbers guarantee no double issuance.

4.2.2 Wallet Page Red‑Packet Flow

The wallet page merges three data sources (cash settlement, withdrawal records, C2C red‑packet flow) in reverse chronological order, supports pagination, and degrades gracefully.

4.3 Stability of Multi‑Dependency Reward Chain

26 degradation switches were prepared; the shortest‑path design isolates critical dependencies (e.g., TCC, MySQL) while allowing optional services to be toggled off without breaking the core flow.

4.4 Budget Control for High‑Volume Coupon Issuance

During the fireworks gala, the platform collaborates with algorithmic strategies to enforce inventory limits, refusing issuance when remaining stock falls below 10%.

4.5 Hot‑Key Read/Write in Extreme QPS Scenarios

Two solutions were evaluated for aggregating total issued amount (estimated 180 w QPS reads, 30 w QPS writes).

4.5.1 Solution 1 – Key Sharding

Split the counter into 100 keys; each request increments actID % 100 . Reads aggregate all shards; writes are simple incr operations.

Drawbacks: high storage cost, read amplification, and potential timeout.

4.5.2 Solution 2 – Local Cache + Periodic Sync

Each instance keeps a local atomic counter; a 1‑second timer syncs the delta to a single Redis key. Reads serve from the local cache, falling back to Redis if the cache is empty.

Comparison table shows Solution 2 saves resources and offers simpler disaster recovery at the cost of slightly more complex implementation.

Advantages

Disadvantages

Solution 1

Implementation simple

Wastes storage; hard to provide DR; cannot continuously accumulate.

Solution 2

Resource‑efficient; simple DR

More complex; must handle atomicity.

Solution 2 was selected for production.

4.6 Smooth Switching Between Parent and Child Activities

The platform uses a hierarchical configuration where the parent activity ID determines which child configuration is active based on request time, enabling seamless transition between internal testing, gray rollout, and production phases.

4.7 Large‑Scale Fund Safety Guarantees

Three layers of protection were applied: overall budget interception, per‑transaction limit interception, and real‑time reconciliation. Hourly and near‑real‑time reconciliation dashboards detect anomalies promptly.

5. Generalized Patterns

The experience led to reusable patterns for disaster recovery, rate limiting, circuit breaking, resource isolation, storage sizing, and load testing.

5.1 Disaster‑Recovery & Degradation

Implemented 26 degradation switches, API‑level Nginx rate limiting, distributed rate limiting, and resource isolation to protect core user experience.

5.2 Microservice Considerations

To reduce RPC overhead, the asset platform exposed SDKs for common operations (balance query, reward existence check, forced settlement), saving tens of thousands of CPU cores during peak traffic.

6. Future Evolution

Refine asset platform design, provide one‑stop services for activity developers.

Enhance real‑time and offline dashboards for clearer reward analytics.

Improve configurability and documentation to lower integration cost.