How Alibaba’s Knowledge Engine Advances AI with Adversarial NER and Graph Embedding

Background

In April 2018 Alibaba Business Platform’s Knowledge Graph team launched the “Cangjingge” (Knowledge Engine) research program together with several universities. The plan defines five technical modules—knowledge acquisition, modeling, reasoning, fusion, and service—and has produced a series of breakthroughs.

Key achievements include automatic ontology construction, attribute discovery, new entity and compact event recognition, relation extraction, deep‑learning based entity and attribute alignment, and powerful inference engines such as CharTransE, XTransE, and a custom reasoning engine.

The resulting knowledge engine is deployed in dozens of Alibaba products (Taobao, Tmall, Hema, Fliggy, Tmall Genie, etc.), handling over 80 million online calls per day and generating 9 billion offline knowledge facts, with vertical graphs for commerce, tourism, and new manufacturing.

Adversarial Learning for NER on Crowdsourced Data

The knowledge‑acquisition module’s core task is named‑entity recognition (NER). To mitigate noisy crowdsourced annotations, the team designed an adversarial network that learns common and private features of annotators, incorporates annotator ID embeddings, and uses a CRF decoder.

Experiments on product title and user query datasets show state‑of‑the‑art performance.

Iterative Rule and Graph‑Embedding Reasoning

Reasoning combines symbolic rules and embedding‑based methods. The proposed framework iteratively learns rules from embeddings, uses rules to predict missing triples for sparse entities, and feeds the predictions back into embedding training, improving link prediction accuracy.

Mathematical formulation and experimental results demonstrate superior performance on benchmark graphs.

These works have been published at AAAI, WWW, EMNLP, and WSDM, and the team continues to advance transferable knowledge‑graph algorithms for Alibaba’s ecosystem.