From Word Embeddings to Large Language Models: A Comprehensive Overview of AI Model Evolution

Introduction

Prompt engineering allows users to converse with large models, but the underlying algorithms have evolved dramatically over the years. The article reviews key milestones that led to today’s generative AI and AGI aspirations.

Word‑Level Embeddings

Word2Vec learns static word vectors from large corpora but ignores word order, resulting in a single representation per token and inability to handle polysemy.

ELMo (2018) introduces contextual embeddings by pre‑training a bidirectional LSTM, enabling different vectors for the same word in different contexts. Its parameter count is ~0.09 B.

Sentence Embeddings and Retrieval‑Augmented Generation

BGE (Beijing Academy) uses a transformer encoder‑decoder architecture to produce high‑quality sentence embeddings for RAG pipelines.

Transformer Model Landscape

The article outlines the progression from encoder‑only BERT, decoder‑only GPT, to encoder‑decoder hybrids (BART, GLM). It highlights the parameter‑size and data‑scale trends that drive performance.

GPT Series

GPT‑1 (2018) demonstrated generative pre‑training on 4.6 GB of BookCorpus followed by supervised fine‑tuning.

GPT‑2 (2019) scaled data to 40 GB (WebText) and model size up to 1.5 B, showing strong zero‑shot capabilities across seven tasks.

GPT‑3 (2020) expanded to 175 B parameters using 570 GB of filtered Common Crawl data, achieving few‑shot performance that rivals fine‑tuned SOTA models.

Alignment and Instruction Tuning

ChatGPT introduced SFT (Supervised Fine‑Tuning) and RLHF (Reinforcement Learning from Human Feedback) to improve helpfulness, honesty, and harmlessness.

Reasoning‑Focused Models

OpenAI’s GPT‑o1 and DeepSeek’s R1 incorporate chain‑of‑thought (CoT) data and reinforcement learning to boost reasoning on code, math, and scientific tasks.

Scaling Laws

Two families of scaling laws are discussed: pre‑train‑time (parameter, data, compute) and test‑time (inference compute). Experiments show that allocating more compute to inference can sometimes outperform larger pre‑trained models.

Multimodal Foundations

ViT adapts the transformer to vision by treating image patches as tokens.

Gemini (Google) and DeepSeek‑Janus extend transformers to jointly process text, images, audio, and video, using modality‑specific encoders followed by a unified transformer.

Conclusion

The rapid evolution from static embeddings to massive multimodal LLMs illustrates how increasing data, model size, and sophisticated training regimes (e.g., RLHF, CoT) continually push AI capabilities forward.