DataFunTalk
Jul 30, 2021 · Artificial Intelligence
Fundamentals of Natural Language Processing: Language Models, Smoothing, and Basic Tasks
This article provides a comprehensive overview of natural language processing fundamentals, covering the challenges of language modeling, N‑gram and Markov assumptions, smoothing techniques such as discounting and add‑one, evaluation via perplexity, basic tasks like Chinese word segmentation, subword tokenization, POS tagging, syntactic and semantic parsing, and a range of downstream applications including information extraction, sentiment analysis, question answering, machine translation, and dialogue systems.
AINLPlanguage model
0 likes · 29 min read