From Symbolic AI to LLMs: A Complete NLP History and Model Guide

NLP Development Stages

Natural Language Processing (NLP) has progressed through distinct eras:

Symbolic (rule‑based) stage : Knowledge is encoded as explicit symbols and logical rules (e.g., early expert systems).

Connectionist & statistical learning stage : Artificial neural networks are introduced and trained with probabilistic learning methods.

Deep Learning stage : Recurrent Neural Networks (RNN), Long Short‑Term Memory (LSTM), and attention mechanisms enable powerful sequence modeling.

Pre‑trained Language Model (PLM) stage : Transformer‑based models such as GPT‑1/2 and BERT are pre‑trained on massive corpora and fine‑tuned for downstream tasks.

Large Language Model (LLM) stage : Scaling of parameters and data, instruction tuning (SFT) and reinforcement learning from human feedback (RLHF) produce emergent abilities like strong instruction following and high‑quality generation.

Sequence Data and Sequence Models

Sequence data (text, speech, video, time‑series) requires models that preserve order and temporal dependencies. Feed‑forward networks and CNNs treat inputs as independent, so they cannot capture sequential context.

RNN – retains a hidden state across time steps.

LSTM – adds a memory cell and gated mechanisms to mitigate vanishing gradients.

GRU – a simplified gated variant of LSTM.

Transformer – uses multi‑head self‑attention to model all pairwise dependencies in parallel.

RNN Architecture

An RNN processes each element sequentially, updating a hidden state: h_t = f(W_hh·h_{t‑1}+W_xh·x_t+b_h) and producing an output: y_t = W_hy·h_t+b_y Because gradients are propagated through many time steps, vanilla RNNs suffer from vanishing or exploding gradients, limiting long‑term dependency learning.

LSTM Details

LSTM introduces three gates (forget, input, output) and a cell state C_t that provides an almost linear gradient path.

f_t = σ(W_f·[h_{t‑1}, x_t] + b_f)</code>
<code>i_t = σ(W_i·[h_{t‑1}, x_t] + b_i)</code>
<code>\tilde{C}_t = tanh(W_c·[h_{t‑1}, x_t] + b_c)</code>
<code>C_t = f_t * C_{t‑1} + i_t * \tilde{C}_t</code>
<code>o_t = σ(W_o·[h_{t‑1}, x_t] + b_o)</code>
<code>h_t = o_t * tanh(C_t)

GRU Details

GRU merges the forget and input gates into an update gate z_t and adds a reset gate r_t:

z_t = σ(W_z·[h_{t‑1}, x_t] + b_z)</code>
<code>r_t = σ(W_r·[h_{t‑1}, x_t] + b_r)</code>
<code>\tilde{h}_t = tanh(W_h·[r_t * h_{t‑1}, x_t] + b_h)</code>
<code>h_t = (1‑z_t) * h_{t‑1} + z_t * \tilde{h}_t

Transformer Architecture

The Transformer replaces recurrence with multi‑head self‑attention, enabling full parallelism and direct long‑range connections.

Embedding + Positional Encoding

Multi‑Head Attention (MHA)

Feed‑Forward Network (FFN)

Layer Normalization & Residual Connections

Self‑attention computes attention weights for each token against all others:

Attention(Q,K,V) = softmax(Q·K^T / √d_k)·V

Key NLP Tasks

Chinese Word Segmentation – splits continuous Chinese text into meaningful words.

Subword Segmentation – breaks rare or unseen words into subword units (BPE, WordPiece, Unigram, SentencePiece).

Part‑of‑Speech Tagging – assigns POS tags to each token.

Text Classification – maps documents to predefined categories.

Named Entity Recognition – extracts entities such as persons, locations, dates.

Relation Extraction – identifies semantic relations between entities.

Summarization – extractive (select sentences) or abstractive (generate new text).

Machine Translation – converts text from one language to another.

Automatic Question Answering – retrieves or generates answers to user queries.

Text Representation Evolution

Vector Space Model (VSM)

VSM represents documents as high‑dimensional sparse vectors (one‑hot encoding). Similarity is measured by cosine, Euclidean distance, etc. The main drawbacks are extreme sparsity and inability to capture semantics or word order.

Word2Vec

Introduced in 2013 by Google, Word2Vec learns dense low‑dimensional embeddings from large corpora using two training objectives:

CBOW – predicts a target word from its surrounding context.

Skip‑Gram – predicts surrounding context words from a target word.

Because the model optimizes the probability of co‑occurring words, the resulting vectors encode semantic relationships (e.g., king - man + woman ≈ queen).

GPT Evolution

GPT‑1 (2018)

First generative pre‑training model focused on language understanding.

GPT‑2 (2019)

Demonstrated that a single large model can perform many downstream tasks without task‑specific fine‑tuning (unsupervised multitask learner).

GPT‑3 (2020)

Scaled to 175 B parameters and introduced few‑shot learning via in‑context examples.

GPT‑4 (2023) and later variants

GPT‑4 (≈1.76 T parameters) adds multimodal capabilities (image + text) and is integrated into productivity tools. Subsequent releases (GPT‑4 Turbo, GPT‑4o) improve latency and add audio‑text interaction.

Practical PyTorch Example: Character‑Level LSTM

import torch
import torch.nn as nn
import numpy as np

# 1. Data preparation
text = """Recurrent Neural Networks (RNNs) are a class of neural networks that are helpful in modeling sequence data.
Derived from feedforward networks, RNNs are similar to human brains in the way they function.
They are designed to recognize patterns in sequences of data, such as text, handwriting, or time series data.
"""
chars = sorted(list(set(text)))
char_to_int = {ch:i for i,ch in enumerate(chars)}
int_to_char = {i:ch for i,ch in enumerate(chars)}

n_chars = len(text)
n_vocab = len(chars)
seq_length = 100
dataX, dataY = [], []
for i in range(0, n_chars - seq_length):
    seq_in = text[i:i+seq_length]
    seq_out = text[i+seq_length]
    dataX.append([char_to_int[c] for c in seq_in])
    dataY.append(char_to_int[seq_out])

X = torch.tensor(dataX, dtype=torch.float32).reshape(len(dataX), seq_length, 1) / float(n_vocab)
y = torch.tensor(dataY)

# 2. Model definition
class CharLSTM(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super(CharLSTM, self).__init__()
        self.lstm = nn.LSTM(input_size, hidden_size, num_layers=2, batch_first=True)
        self.fc = nn.Linear(hidden_size, output_size)
    def forward(self, x):
        h0 = torch.zeros(2, x.size(0), self.lstm.hidden_size)
        c0 = torch.zeros(2, x.size(0), self.lstm.hidden_size)
        out, _ = self.lstm(x, (h0, c0))
        out = self.fc(out[:, -1, :])
        return out

model = CharLSTM(input_size=1, hidden_size=256, output_size=n_vocab)

# 3. Training
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
for epoch in range(20):
    optimizer.zero_grad()
    outputs = model(X)
    loss = criterion(outputs, y)
    loss.backward()
    optimizer.step()
    print(f'Epoch [{epoch+1}/20], Loss: {loss.item():.4f}')

# 4. Text generation
start = np.random.randint(0, len(dataX)-1)
pattern = dataX[start]
generated = ''
with torch.no_grad():
    for _ in range(500):
        x = torch.tensor(pattern, dtype=torch.float32).reshape(1, seq_length, 1) / float(n_vocab)
        pred = model(x)
        idx = torch.argmax(pred).item()
        generated += int_to_char[idx]
        pattern.append(idx)
        pattern = pattern[1:]
print(generated)

RNN vs LSTM Summary

RNN : Simple architecture, few parameters, fast training, but suffers from vanishing/exploding gradients and cannot capture long‑term dependencies.

LSTM : Gated memory solves gradient decay, excels at long sequences, at the cost of higher computational load and more parameters.

Conclusion

The field of NLP has evolved from rule‑based symbolic systems to massive multimodal LLMs capable of understanding and generating text, images, and audio. Core milestones—VSM, Word2Vec, RNN, LSTM, GRU, Transformer, and the GPT series—provide the technical foundation for modern research and practical applications.