Why LLM‑Generated Research Ideas Differ from Humans: It's About Distribution, Not Quality

Study Design

Researchers collected 11,683 real papers: 5,994 from ICLR, ICML, and NeurIPS (2023‑2026) and 5,689 from Nature Communications (2023‑2025). For each paper they reverse‑engineered a set of "nearest works" that likely inspired the core idea, then supplied the identical reference list to several large language models (LLMs) and asked the models to generate new research ideas from the same starting point.

Research‑Taste Taxonomy

The authors defined a taxonomy with two axes, each containing seven labels:

Opportunity Pattern – the type of gap the idea discovers.

Method Paradigm – how the gap is turned into a contribution.

Distribution of Human vs. Model Ideas

Human‑generated ideas are broadly distributed: only 12.1% of motivations are classified as "bridge opportunities" (connecting fragmented literature or methods) and 5.1% of core methods are "integrate/unify". By contrast, LLM‑generated ideas allocate 47.1%–64.2% to bridge opportunities and 22.5%–38.7% to integrate/unify, a four‑to‑five‑fold increase over humans.

Model Coverage

All nine mainstream models tested (Claude, Gemini, GPT, Qwen, DeepSeek, etc.) exhibited the same bias. Enabling a chain‑of‑thought "thinking mode" amplified the effect: for Qwen‑3‑8B, bridge‑opportunity share rose from 49.7% to 71.1% and integrate/unify share from 38.7% to 52.2%.

Ablation Experiments

Providing the full paper instead of the abstract, and switching to a more neutral prompt, did not reduce the bridge‑opportunity preference, indicating the bias is not caused by prompt wording or context length.

Quantitative Metrics

Normalized entropy on the opportunity axis: humans 0.926; models 0.550–0.758.

Best‑performing model (Claude‑Sonnet‑4.6) total variation distance on the method axis: 0.211, meaning >20% of probability mass would need to shift to match the human distribution.

Claude‑Sonnet‑4.6 scores slightly higher than humans on diagnostic measures (higher bottleneck specificity, lower templating), showing strong writing quality despite distributional skew.

Action‑Verb Analysis

When each idea is abstracted to its primary action verb, the frequency gap is stark:

integrate : model 34.2%, human 2.35%

unify : model 8.2%, human 1.9%

replace : human 9.13%, model 0.92%

decouple : human 2.33%, model 0.21%

Humans tend to modify, replace, or decouple a local mechanism; models preferentially combine two high‑frequency concepts.

Interpretation

The pattern reflects a stable behavioral mode: LLMs excel at stitching together existing papers—producing coherent, well‑written ideas—but they rarely venture beyond mainstream paradigms to design experiments, analyze failures, or propose novel mechanisms.

Implications

For researchers, LLMs can be useful for brainstorming within established paradigms, but reliance on them for breakthrough ideas that require exploring less‑traveled conceptual paths is unwarranted.

Paper: https://arxiv.org/pdf/2607.01233

Code and dataset: https://github.com/ziyuuc/TasteGap