What Is Perplexity in Large Language Models?

Perplexity is a metric that measures how well a probabilistic model predicts samples and is widely used to evaluate the performance of large language models.

A language model defines a probability distribution over sentences; a high‑quality sentence should receive a higher probability, resulting in a lower perplexity, while low‑quality text yields higher perplexity.

For illustration, a tiny model with a six‑word vocabulary ("a", "the", "red", "fox", "dog", ".") predicts the sentence "a red fox.". The model assigns probabilities:

P("a") = 0.4

P("red" | "a") = 0.27

P("fox" | "a red") = 0.55

P("." | "a red fox") = 0.79

The sentence probability is the product: P("a red fox.") = 0.4 * 0.27 * 0.55 * 0.79 = 0.0469 Because longer sentences have smaller raw probabilities, the article normalizes by the number of words using the geometric mean: Pnorm(W) = P(W) ^ (1 / n) For the example (n = 4): Pnorm("a red fox.") = 0.0469 ^ (1/4) = 0.465 The perplexity is the inverse of this normalized probability: PP(W) = 1 / Pnorm(W) = 1 / 0.465 ≈ 2.15 Comparing with a uniform model that assigns equal probability (1/6) to each of the six tokens, the sentence probability becomes: P("a red fox.") = (1/6) ^ 4 = 0.00077 Thus Pnorm = 1/6 and PP = 6, a much higher perplexity than the trained model.

The article concludes that lower perplexity scores indicate better models: a perplexity of 1 would be perfect, while values like 50 suggest near‑random predictions. Perplexity therefore serves as a compass for building more accurate, reliable language models.