From Symbolic Semantics to Vector Representations: Deep Learning for Natural Language Understanding

Before deep learning, meaning in text was conveyed to computers through manually designed symbols and structures; the author previously described this process and now revisits three symbolic resources—WordNet, ConceptNet, and FrameNet—to contrast them with deep learning capabilities.

WordNet, developed at Princeton, groups synonymous words and encodes hierarchical relations, e.g., treating “sedan” and “car” as the same concept within the vehicle category.

ConceptNet, from MIT, captures broader relational knowledge such as the frequent co‑occurrence of “bread” near “toaster,” illustrating the vast and varied connections between words.

FrameNet, a Berkeley project, archives semantic frames that define concepts and their roles (e.g., a birthday party frame includes venue, entertainment, and cake), allowing computers to “understand” text by searching for frame‑triggering keywords, though creating these frames manually is labor‑intensive.

Symbolic language models estimate word probabilities from corpora, but they struggle to exploit similarity between related terms and require prohibitive storage for high‑order n‑grams, highlighting the need for a better approach.

Using Vectors to Represent Semantics

Deep learning encodes meaning as dense vectors (typically ~300 dimensions) where each index corresponds to a learned feature; similarity between vectors reflects semantic closeness, allowing “celebrated baozi” and “dog‑style baozi” to be near each other while remaining distant from unrelated concepts like “car.”

Vectors also exhibit internal structure, enabling analogical reasoning such as:

Italy - Rome = France - Paris

and

King - Queen = Man - Woman

These relationships emerge from training neural networks to predict neighboring words, and pre‑trained embeddings can be downloaded from Google, Stanford, or generated with the Gensim library.

Composing Meaning from Word Vectors

Recursive neural networks (RNNs) can combine word vectors to encode entire sentences; for example, the sentence “The woman ate tacos.” is encoded step‑by‑step into a final hidden vector h4 that represents the whole sentence.

Decoding this vector with another RNN yields a translation, e.g., generating the Spanish sentence word by word, each step conditioned on the previously generated word and the current hidden state.

Encoder‑decoder models require massive parallel corpora and millions of parameters, but once trained they can output parse trees, image captions, or other structured forms, leveraging vector representations of both language and visual data.

From Composed Semantics to Attention, Memory and QA

To answer questions or continue a translation, a system must retain and retrieve past states; Bahdanau et al. introduced attention mechanisms that let the network focus on the most relevant memory vectors at each decision point.

By treating concepts and sentences as vectors, a large set of vector memories can be searched to find the best answer, either via inner‑product similarity or by feeding question and fact vectors through deeper networks trained on QA data.

The Next Frontier: Accurate Semantic Understanding

Current methods capture story‑like information but miss subtle contextual cues (e.g., knowing that moving a table also moves the book on it). Achieving true understanding will require robots to acquire real‑world experience, encode it with deep networks, and integrate that experiential knowledge with abstract reasoning.

Such integration would allow a robot to link the visual event of a box falling with the linguistic expression “the box fell,” enabling correct interpretation of metaphorical uses like “the stock fell 10 points.”

Practical Resources for Getting Started with Deep Learning

Recommended entry points include Stanford’s NLP deep‑learning course, Hinton’s Coursera lectures, and the concise online textbook by Bengio et al.; Python users may start with Theano, while Java developers can explore Deeplearning4j.

Conclusion

The surge in computing power and digital data has driven a deep‑learning revolution; large models with millions of parameters succeed because they can be trained on massive datasets. To achieve genuine intelligence, future algorithms must learn from real‑world experience, conceptualize that experience, and combine it with abstract reasoning.