Understanding LLM Output & Sampling: Temperature, Top-k, Top-p with DeepSeek Examples

From a matrix-operation perspective, the final model output is a vector whose length equals the vocabulary size; after a softmax activation it becomes a probability distribution over every token. Selecting the next token involves a trade-off between determinism and randomness: picking the highest-probability token yields stable, predictable output, while stochastic sampling increases diversity and creativity. This trade-off is controlled by the model's sampling strategy.

1. Sampling Strategies

Three main strategies exist: temperature, top-k, and top-p.

1.1 Temperature

Temperature controls creativity — higher values produce more varied output. The author demonstrates using the DeepSeek API with a prompt asking the model to complete the couplet "天若有情天亦老，" (If heaven has feelings, heaven grows old too).

With temperature: 0 the output is deterministic:

月如无恨月长圆 (The moon, without sorrow, stays forever round)

With temperature: 2 the output varies across runs:

月如无恨月长圆

人间道义尽沧桑 (Worldly morality ends in vicissitudes)

情深偏易断肠多 (Deep affection easily brings heartbreak)

月如无恨月长圆。踏破璀璨星空，为寻前世因缘。清风拂柳千年慢，我倾战意翻云，乾坤内，与谁来争。凝望明月高悬，此意气飘然，今夜群星皆不醒，我本肆意剑出行！啊 举杯与苍天似深情，放声狂笑称心情。剑指长天向侠行！尽我此生豪马踏名城！

酒逢知己醉千杯 (Wine meets a bosom friend, a thousand cups)

水能无怨水当干，然未曾几测高深。 (Water can be without resentment, yet its depth is rarely measured.)

红绡先冷迎风眠 (Red gauze first grows cold facing the wind)

......

The mechanism is simple: each token's logit is divided by the temperature value. A larger temperature shrinks the differences between logits, raising the probability of lower-ranked tokens being selected.

1.2 Top-k

Top-k sampling sorts logits and keeps only the top k values, then runs softmax on that reduced set. This reduces softmax computation. However, it is used less often — DeepSeek does not expose a top-k parameter — because k is an absolute number that must be adjusted for different output lengths.

1.3 Top-p (Nucleus Sampling)

Top-p is currently the more popular strategy. It still computes softmax over the full vocabulary, then sorts probabilities and selects the smallest set of tokens whose cumulative probability exceeds p. The author tests the same couplet prompt with varying top_p: top_p: 0.01 → fixed output: "月如无恨月长圆" top_p: 1 → also fixed: "月如无恨月长圆"

The result diverges from expectation. The author hypothesizes that the top token's probability dominates, and verifies this by requesting logprobs. The returned log-probabilities show:

Token "月" (moon): logprob -0.031558793 → probability ≈ 96.89%

Token "人间" (world): logprob -3.471668 → probability ≈ 3.1%

Token "人" (person): logprob -14.080235 → negligible

Indeed, "月" overwhelmingly leads, so even top_p = 1 effectively samples only that token.

2. Structured Output

In practical scenarios (e.g., generating SQL or regex), structured output is desired. Four approaches exist, none providing absolute guarantees:

Prompt engineering — explicitly specify the output format in the prompt.

Post-processing — validate and parse the output after generation.

Constrained sampling — filter out tokens that violate the format during sampling.

Fine-tuning — train the model to emit structured formats.

Generally, post-processing offers the best cost-effectiveness ratio.

3. Viewing Output Uncertainty

LLMs trained on massive datasets can produce vastly different outputs for identical or similar inputs. This inconsistency is expected. The harder-to-grasp phenomenon is hallucination ; the industry has hypotheses but no fundamental explanation (if the mechanism were understood, it would likely be solved).

After this eleven-article series, the author resonates with Martin Fowler's observation (via Rebecca Parsons):

"My former colleague Rebecca Parsons, has been saying for a long time that hallucinations aren't a bug of LLMs, they are a feature. Indeed they are the feature. All an LLM does is produce hallucinations, it's just that we find some of them useful."

The author offers a balanced phrasing: LLM creativity stems from probability; we must understand and manage this uncertainty, not discard the technology because of its flaws.