Mastering LLM Text Generation: Decoding Methods Explained

Course Review

After studying GLM large models, this week's public class analyzed the core capability of large language models—text generation—introducing various decoding methods with demos using MindSpore NLP.

1. Autoregressive Language Model Principle

Text sequence probability can be factorized as the product of conditional probabilities of each token given the preceding context.

2. Deterministic Decoding (Search Methods)

Greedy Search

Select the token with the highest probability at each step.

Advantage: fastest.

Disadvantage: may miss high‑probability words hidden behind low‑probability tokens.

Beam Search

Keeps the top‑num_beams candidates at each step and selects the highest‑probability sequence, reducing the risk of losing potential high‑probability paths.

Advantage: retains optimal paths to some extent.

Disadvantage: cannot solve repetition issues; performs poorly in open‑domain generation.

3. Stochastic Decoding (Sampling Methods)

Randomly choose the next token according to the current conditional probability distribution.

Temperature

Increases the likelihood of high‑probability words and decreases that of low‑probability ones; larger values yield more randomness.

Top‑K

Select the K most probable tokens, renormalize their probabilities, and sample from this reduced set.

Drawbacks: can produce gibberish when the distribution is sharp; limits creativity when the distribution is flat.

Top‑P (Nucleus Sampling)

Sample from the smallest set of tokens whose cumulative probability exceeds p, then renormalize.

Advantage: the sampling pool dynamically adjusts to the probability distribution.

4. Other Decoding Methods

Constrained Beam Search : Inserts fixed word combinations into beam search candidates to achieve desired content.

Contrastive Search : Adds a penalty term based on similarity with previous context.

Assisted Search : Uses a small assistant model to predict or decide whether to cache text, reducing the number of LLM calls.