Why Natural Language Understanding Remains an AI‑Hard Problem and How Deep Learning Tackles It

Introduction

Natural language understanding (NLU) is a subfield of natural language processing in artificial intelligence that deals with machine reading comprehension and is considered an AI‑hard problem.

The term AI‑hard (or AI‑complete) refers to problems whose difficulty is equivalent to solving the central AI problem of building machines as intelligent as humans.

The author previously described an NLU system in 2015; since then the core modules have been upgraded to deep‑learning solutions, focusing on two core algorithms: intent classification and attribute extraction.

Challenges of NLU

NLU faces five major difficulties:

Language diversity – multiple ways to express the same meaning.

Language ambiguity – the same utterance can have different interpretations without context.

Language robustness – speech‑to‑text errors such as misspellings, extra or missing words, noise, and colloquial expressions.

Knowledge dependence – words often refer to world knowledge (e.g., “大鸭梨” can be a fruit or a restaurant name).

Contextual dependence – understanding requires dialogue, device, application, or user‑profile context.

Intent Classification Implementation

Intent classification is a text‑classification task. Common methods include:

Rule‑based approaches (e.g., CFG, JSGF).

Traditional machine‑learning methods (e.g., SVM, ME).

Deep‑learning methods (CNN, RNN/LSTM, etc.).

Rule‑based methods are illustrated with a CFG example for the flight‑ticket domain (see image).

Deep‑learning experiments show that a simple CNN performs best among tested architectures (CNN, LSTM, RCNN, C‑LSTM). However, CNN alone cannot surpass complex feature‑engineered SVMs in knowledge‑heavy domains.

To bridge symbolic and connectionist representations, the system fuses distributed and symbolic features.

Attribute Extraction Implementation

Attribute extraction is treated as a sequence‑labeling problem. Models used include RNN, LSTM, Bi‑LSTM, Bi‑LSTM‑Viterbi, and Bi‑LSTM‑CRF. The production system adopts a Bi‑LSTM‑CRF model.

During inference, the model maps an utterance such as “帮我打开空调” to a path in the grammar graph, allowing the slot “device” to be filled with “空调”.

Conclusion

In practice, rule‑based and deep‑learning methods coexist. Rule‑based techniques are useful for quick fixes and bug handling, while deep‑learning models form the core of the system.

References

[1] https://en.wikipedia.org/wiki/Natural_language_understanding

[2] Ward & Issar, CMU Phoenix System, 1996

[3] Yoon Kim, Neural Networks for Sentence Classification, EMNLP, 2014

[4] Suman Ravuri and Andreas Stolcke, Recurrent Neural Network and LSTM Models for Lexical Utterance Classification, InterSpeech, 2015

[5] Siwei Lai, Liheng Xu, Kang Liu, Jun Zhao, Recurrent Convolutional Neural Networks for Text Classification, AAAI, 2015

[6] Chunting Zhou, Chonglin Sun, Zhiyuan Liu, Francis C.M. Lau, A C‑LSTM Neural Network for Text Classification, arXiv, 2015

[7] Grégoire Mesnil et al., Using Recurrent Neural Networks for Slot Filling in Spoken Language Understanding, TASLP, 2015

[8] Xiaodong Zhang, Houfeng Wang, A Joint Model of Intent Determination and Slot Filling for Spoken Language Understanding, IJCAI, 2016

[9] Bing Liu, Ian Lane, Joint Online Spoken Language Understanding and Language Modeling with Recurrent Neural Networks, arXiv, 2016