Clever Classical Ideas in Natural Language Processing Tasks

In this article, Fudan University associate professor Qiu Xipeng shares a collection of clever ideas that were widely used in natural language processing (NLP) before the deep‑learning era.

Distributional Hypothesis : The principle that linguistic items occurring in similar contexts tend to have similar meanings, implying that a word’s semantics can be represented by its surrounding context.

Bag‑of‑Words (BoW) : Treats a document as an unordered collection of words, discarding syntax and word order, thereby converting a variable‑length text sequence into a fixed‑length vector. Extensions such as n‑grams and TF‑IDF build on this basic representation.

Latent Semantic Analysis (LSA) : Constructs a term‑document matrix from the BoW representation and applies singular value decomposition (SVD) to obtain dense vector embeddings for words and documents, revealing hidden semantic structures like topics.

Probabilistic Topic Models (PTM) : Introduce a latent “topic” variable between documents and words, modeling the generation process as document → topic → word. The approach is mathematically elegant, with inference often performed via Gibbs sampling, and remains a valuable resource for NLP tasks.

BMES/BIO Tagging Schemes : BMES (Begin/Middle/End/Single) and BIO (Begin/Inside/Outside) label characters or tokens to transform segmentation, named‑entity recognition, and chunking into sequence‑labeling problems that can be tackled with models such as HMMs and CRFs.

TextRank : Adapts the PageRank algorithm to rank words or sentences in a text, enabling applications like keyword extraction and automatic summarization.

The article concludes with reference links to the distributional semantics Wikipedia page, the term‑document matrix page, and the original Zhihu answer.