Fast Attribution Engine (FAE): High‑Performance Distributed Computing for User Behavior and Advertising Attribution
FAE, Alibaba’s high‑performance distributed MPP engine, stores billions of user‑behavior events in a time‑ordered AFile model and uses stateless masters, importers, mergers and workers with Redis and MySQL metadata to deliver sub‑second, 10‑100× faster ad‑attribution queries across ad‑hoc, offline and near‑real‑time scenarios such as frequency, path and funnel analysis.
FAE (Fast Attribution Engine) is a high‑performance distributed MPP engine developed by Alibaba's advertising data platform, designed for user behavior analysis and advertising attribution.
The talk covers technical challenges in user behavior and attribution, the system’s data model (AFile), architecture, modules (master, importer, merger, worker), storage options, and typical application scenarios such as frequency analysis, crowd flow modeling, path analysis, retention, funnel analysis, and visitor segmentation.
Key points:
Technical challenges: massive data volume (hundreds of billions of events), need for time‑ordered per‑user data, flexible query dimensions, and sub‑second latency.
FAE data model (AFile) stores events pre‑sorted by user ID and timestamp, enabling fast UDAF/MapReduce operations.
Architecture: stateless master nodes, importer for data ingestion, merger for sharding, worker nodes for query execution; relies on Redis for metadata and MySQL for logs.
Supports both local‑disk and NAS‑based distributed storage, with three query modes (ad‑hoc, offline, near‑real‑time).
Application scenarios demonstrate up to 10‑100× speedup over generic OLAP engines.
Q&A clarified incremental vs full import, time ordering guarantees, and CPU vs I/O intensity of queries.
The presentation concludes with an invitation to the upcoming DataFunSummit.
Alimama Tech
Official Alimama tech channel, showcasing all of Alimama's technical innovations.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.