Neural–Symbolic Learning and Multimodal Knowledge Discovery: Recent Advances, Methods, and Challenges

The presentation begins with two illustrative examples: a GPT‑3 generated story that demonstrates erroneous biomedical reasoning, and a multimodal visual‑question‑answering task that shows the necessity of combining multiple data modalities.

It then argues that symbolic knowledge (variables, instances, bindings, deductive/inductive/abductive/analogical reasoning) is essential for complementing neural networks, which excel at perception but lack explicit reasoning and interpretability.

Historical milestones are surveyed, from Valiant’s relational learning and Markov Logic Networks to recent neural‑symbolic systems such as Swift Logic, Logic Tensor Networks, and Deep Learning 2, illustrating how symbolic components have been gradually integrated into deep models.

Three categories of neural‑symbolic integration are described: (1) using neural methods directly for shallow reasoning (e.g., graph neural networks for link prediction); (2) multi‑task learning frameworks that embed heterogeneous knowledge sources; and (3) augmenting neural models with symbolic constraints for tasks like knowledge distillation, distant supervision, and few‑shot vision.

The talk emphasizes the importance of explainability and the limitations of current systems, such as incomplete or noisy knowledge bases and the difficulty of representing innate or commonsense knowledge.

Multimodal knowledge discovery is presented as a large‑scale engineering problem: building multimodal knowledge graphs, aligning text, images, and video, and applying them to recommendation, tourism, software engineering, and personal life‑log scenarios.

Practical challenges are listed, including the availability of truly multimodal datasets, the design of expressive symbolic representations for multimodal data, and the need for powerful multimodal pre‑training and computational resources.

Finally, the speaker announces a new “Resource Track” for the upcoming CCKS conference, inviting the community to contribute multimodal datasets, and concludes with a call for continued research on when multimodal fusion is beneficial versus detrimental.