Core Techniques of ID Mapping for Data Integration in Big Data Platforms

ID mapping is a core capability of Alibaba's data middle platform, aimed at solving data island problems by unifying disparate identifiers across systems.

The author shares personal experiences with traditional address cleaning and highlights the limitations of manual rule‑based matching, then introduces the need for ID mapping in internet advertising (DSP) where cookies are used to identify the same user across devices.

Because cookies are domain‑specific, cross‑domain ID mapping is required to link user activity on different sites, enabling personalized recommendations.

In modern big‑data environments, the author advocates using graph databases and graph computation to model data as nodes and edges, allowing automatic identification of the same entity through connected sub‑graph algorithms.

The ID‑mapping workflow consists of six steps: (1) recognize elements and raw IDs from each source, (2) abstract them into nodes and edges with threshold filtering, (3) build a graph model and run connectivity algorithms, (4) assign a new unified ID, (5) deduplicate and merge data, and (6) iterate steps 3‑5 while reusing existing IDs.

The resulting ID‑mapping dictionary acts as a bridge between data islands, enabling the construction of comprehensive user profiles, and can be stored in fast‑query stores such as Elasticsearch for external One‑ID lookup services.

Finally, the article emphasizes the importance of a data warehouse (e.g., Hive on Hadoop) for storing tags and user features, and notes practical considerations such as handling many‑to‑many matches, noise filtering, and the need for ongoing engineering practice.