Real-time Multi-system Data Aggregation for Fan Tag System
The Xianyu fan‑tag system solves the challenge of displaying full‑history purchase counts with real‑time updates and low‑latency, high‑throughput queries by daily exporting multi‑system data to a LevelDB‑based KV store, converting schemas, and applying real‑time compensation from transaction and follow‑change messages, merging offline and live data to produce sorted fan lists at ~10 k QPS.
In business development, a common problem is displaying data that spans multiple unrelated systems, requires long‑term historical coverage, real‑time updates, high query QPS and low latency.
The Xianyu fan‑tag system needs full historical purchase counts, real‑time changes, and high‑throughput display across scenarios such as IM chats.
The solution combines daily offline export of multi‑system data with schema conversion to a KV store, and real‑time compensation using business change messages (follow/unfollow, transaction) to keep the data fresh.
Offline data is generated from order and follow records and loaded into a KV store; real‑time data is updated by consuming transaction success and follow‑change messages.
The KV store holds one‑to‑one transaction records and one‑to‑many fan‑list records. Queries merge offline and real‑time data to compute total purchase counts and produce a sorted fan list.
High‑performance KV storage (e.g., LevelDB) and tailored schemas ensure low latency (few ms) while handling ~10k QPS.
The approach satisfies both full‑history and real‑time requirements, though hot data may cause compensation spikes; future work may explore analytic databases for further optimization.
Xianyu Technology
Official account of the Xianyu technology team
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.