How a Simple Prompt Boost Landed a Paper at ICML 2026 and Sparked Online Debate

Problem: Mode collapse in large language models

LLMs frequently generate repetitive, high‑probability answers for jokes, code, or factual queries. Conventional remedies focus on adjusting sampling parameters, changing decoding algorithms, or retraining models.

Hypothesis: Typicality bias in preference data

The authors identify a "typicality bias" in human‑annotated preference data: annotators tend to favor familiar, fluent, and conventional responses, assigning them higher scores. Consequently, even perfect reward models cannot prevent mode collapse if the training preference data are biased.

Proposed method: Verbalized Sampling (VS)

VS asks the model to verbalize its full probability distribution during generation. By prompting the model to output both the generated text and an associated probability value, the approach enables "oral probability sampling" that restores the latent multimodal distribution learned during pre‑training, without any fine‑tuning.

Generate five jokes, each accompanied by a possible probability value.

In this concrete example, the model produces five distinct jokes together with probability estimates, resulting in richer, less repetitive answers.

Experimental setup

Evaluation on five preference datasets covering creative‑writing tasks.

Multiple base LLMs of varying scale were tested.

Metrics included diversity (e.g., distinct‑n), factual accuracy, and safety.

Results

Diversity increased by 1.6–2.1× compared with standard prompts.

Factual accuracy and safety metrics remained unchanged.

Larger models exhibited greater diversity gains.

Analysis

The gains are attributed to exposing the model's full probability distribution at inference time, which mitigates the effect of typicality bias without modifying the model or its training data.

Limitations noted

The prompt‑only technique may not generalize uniformly across all tasks or model families.

Experimental scope was limited to five datasets, leaving broader applicability unverified.

While similar prompt‑engineering tricks exist, the paper provides a systematic empirical evaluation of the proposed approach.

Code example

[1]https://www.reddit.com/r/MachineLearning/comments/1uv1xb3/promptengineering_paper_accepted_to_icml_r/
[2]https://www.linkedin.com/in/jiayizx/
[3]https://simonucl.github.io/
[4]https://www.linkedin.com/in/derekch/