ICLR 2026 Outstanding Papers Reveal the Real Test for LLMs

Hello, I’m PaperAgent (not an agent!). Many of us feel that the longer we chat with large language models, the "dumber" they seem to become.

ICLR 2026 Outstanding Papers

The ICLR 2026 Outstanding Paper award, decided by a 12‑member committee chaired by Gautam Kamath, selected two papers after a five‑week review of 36 long‑list papers, narrowing to five short‑list papers and finally two winners. This year received 19,000 submissions with an overall acceptance rate of about 28.18% .

The two awarded papers cover the full spectrum from pure theory to practical impact: one demonstrates that the Transformer architecture is inherently "succinct," and the other discovers that large models "get lost" in multi‑turn dialogue.

LLMs Get Lost In Multi‑Turn Conversation

LLMs Get Lost In Multi‑Turn Conversation

Authors: Philippe Laban, Hiroaki Hayashi, Yingbo Zhou, Jennifer Neville

arXiv: https://arxiv.org/abs/2505.06120

The core finding is stark: every mainstream LLM tested—including OpenAI (GPT‑4o‑mini, GPT‑4o, o3, GPT‑4.1), Anthropic (Claude 3 Haiku, Claude 3.7 Sonnet), Google Gemini (2.5 Flash, 2.5 Pro), Meta Llama (3.1‑8B‑Instruct, 3.3‑70B‑Instruct, 4 Scout), AI2 OLMo‑2‑13B, Microsoft Phi‑4, Deepseek‑R1, and Cohere Command‑A—shows a **39 % average drop** in performance on six generation tasks when moving from single‑turn QA to multi‑turn dialogue.

The decline is not mainly due to reduced aptitude; rather, the models become far less reliable. The same task can be answered well in one turn and completely off‑track in another, leaving users unable to predict when the output is trustworthy.

Analysis of over 200,000 simulated conversations reveals that models often make early assumptions and emit a premature “final answer.” Once that early answer deviates, the model continues along the wrong path without back‑checking or self‑correction.

The authors quote directly: “When a large model takes a wrong step in a dialogue, it gets lost and cannot recover.” This matters because most existing benchmarks are single‑turn, providing a clear instruction and expecting a single answer, whereas real users interact with models through iterative, incomplete instructions and revisions.

The committee notes that although the evaluated model versions are slightly older, the conclusions and diagnostic framework remain applicable to the latest state‑of‑the‑art models, offering the first systematic tool to measure multi‑turn conversational ability.

Transformers Are Inherently Succinct

Transformers are Inherently Succinct

Authors: Pascal Bergsträßer, Ryan Cotterell, Anthony Widjaja Lin

arXiv: https://arxiv.org/abs/2510.19315

This paper takes a purely theoretical route, proposing that the strength of Transformers stems not from doing more, but from describing the same formal language with far fewer symbols than finite automata or linear‑temporal logic (LTL). The succinctness is measured by formal description length, not aesthetic criteria.

Specifically, the authors prove that Transformers can represent the same formal languages with dramatically shorter descriptions than finite automata and LTL, making verification of Transformer properties computationally infeasible (EXPSPACE‑complete). The committee highlighted the work’s strong conceptual impact, expecting it to inspire further research on the expressive efficiency of different architectures.

https://blog.iclr.cc/2026/04/23/announcing-the-iclr-2026-outstanding-papers/

Read the papers to see how they reshape our understanding of model efficiency and reliability.