How Deep Learning Transforms Knowledge Graph Relation Extraction

Deep Learning Model Overview

In the era of intelligent search, engines not only understand user queries but also build comprehensive knowledge graphs. The knowledge‑graph team at Shenma continues to explore this direction.

Previously we introduced a DeepDive‑based relation extraction method. Today we share deep‑learning techniques, focusing on Piecewise Convolutional Neural Networks (PCNNs) and their improvements.

Piecewise Convolutional Neural Networks (PCNNs)

PCNNs, proposed by Zeng et al. (2015), address two problems:

For the wrong‑label problem in distant supervision, the model uses multi‑instance learning to select high‑confidence training examples.

To avoid error propagation from traditional statistical feature extraction, PCNNs automatically learn features with a piecewise CNN, reducing reliance on complex NLP pipelines.

The figure below illustrates the PCNN architecture:

PCNNs + multi‑instance learning improves Top‑N average precision by 5 percentage points over pure MIL.

Attention Mechanism and Other Improvements

PCNNs originally use a single sentence per entity pair, discarding many correctly labeled sentences. Lin et al. (2016) introduced PCNNs+Attention (APCNNs), adding a sentence‑level attention layer before the softmax, as shown below:

Additional auxiliary information such as entity descriptions and external confidence scores further enhance performance.

Progress of Deep Learning Methods in Knowledge‑Graph Construction

Deep learning models are still exploratory for Shenma's knowledge‑graph construction. Key steps include:

Corpus preparation and entity vectorization using Word2Vec trained on full‑scale encyclopedia data.

Model selection and training data preparation, focusing on APCNNs.

Application trials and problem analysis, highlighting challenges such as long training times, domain adaptation from English NYT to Chinese corpora, difficulty of manual intervention, and evaluation complexity.

Experimental results on the NYT dataset show APCNNs outperform CNN+MIL, PCNN+MIL, and CNN‑Attention models.

Training data includes millions of positive examples for 15 core relations (e.g., movie‑actor, book‑author) and over 100 million negative examples.

Summary and Outlook

Both DeepDive and deep‑learning approaches have strengths and weaknesses. DeepDive excels with small, specialized corpora and long‑tail relations, while APCNNs scale to large corpora but require sufficient entity frequency.

Future work includes:

Integrating proven DeepDive algorithms into deep‑learning pipelines.

Visualizing intermediate results to better assess attention decisions.

Extending relation extraction to open domains and discovering new relations.

Exploring multimodal sources such as tables, audio, and images.

References

[1] Lin et al., 2016. Distant Supervision for Relation Extraction with Sentence‑Level Attention. [2] Zeng et al., 2015. Distant Supervision for Relation Extraction via PCNNs. [3] Ji et al., 2017. Distant Supervision with Entity Descriptions. [4] Tang et al., 2017. ENCORE: External Neural Constraints. [5] Riedel et al., 2010. Modeling Relations without Labeled Text. [6] Ce Zhang, 2015. DeepDive: A Data Management System for Automatic Knowledge Base Construction. [7] Mintz et al., 2009. Distant Supervision without Labeled Data. [8] Additional URLs as cited in the original article.