Understanding Distributional Semantics: How Word Meaning Is Captured by Context

Natural Language Processing (NLP) aims to enable computers to understand, process, and generate human language. Distributional semantics rests on the distributional hypothesis: a word’s meaning can be defined by the contexts in which it appears, so words that share similar contexts have similar meanings.

In neural networks, words must be represented as numeric vectors. During training, each token in the vocabulary is assigned a vector that captures its meaning based on surrounding vectors.

The earliest approach builds a word‑co‑occurrence matrix by counting how often pairs of words appear together within a fixed context window (e.g., ±2 words). For example, using the sentences “The cat chased the mouse in the barn.” and “A cat chased a rat in the attic.”, the matrix records counts such as the word the co‑occurring with cat twice. The matrix is symmetric, has zeros on the diagonal (a word never co‑occurs with itself), and can become extremely large and sparse.

To address sparsity, Singular Value Decomposition (SVD) from linear algebra is applied to the co‑occurrence matrix. This dimensionality‑reduction technique underlies Latent Semantic Analysis (LSA), which projects the high‑dimensional sparse matrix into a smaller, dense vector space, revealing latent semantic structure among words.

Through LSA, each word obtains a compact vector representation that reflects its semantic relationships inferred from massive text corpora, allowing even undefined words to be understood via their associations with other words.