Emergent Abilities of Large Language Models: Complex Reasoning, Knowledge Reasoning, and Out‑of‑Distribution Robustness

Introduction

Recent breakthroughs in large language models (LLMs) have revealed capabilities that appear only when model scale reaches a certain threshold, often referred to as emergent abilities. These include complex multi‑step reasoning, knowledge‑based inference, and out‑of‑distribution (OOD) robustness, which challenge traditional fine‑tuning paradigms.

Prerequisites

Readers are expected to be familiar with pre‑training, fine‑tuning, and prompting basics, as well as chain‑of‑thought (CoT) prompting.

Emergent Abilities Unique to Large Models

Empirical evidence shows that small models gain little from scaling, while models above ~60 B parameters exhibit sharp performance jumps on tasks such as GSM8K arithmetic, indicating abilities that do not exist in smaller models.

Three Representative Emergent Abilities

Complex Reasoning

Knowledge Reasoning

Out‑of‑Distribution Robustness

Complex Reasoning

Chain‑of‑thought prompting dramatically improves performance on multi‑step math problems. For example, the following GSM8K prompt achieves 56.6 % accuracy with only eight examples:

问题：
克莱儿每天早饭都用 3 颗蛋做蛋卷，她 4 周会吃掉多少打鸡蛋？

克莱儿每天早饭都用3颗蛋做蛋卷。
一周有 7 天。
所以她一周会吃 3 * 7 = 21 颗蛋。
她 4 周会吃掉 4 * 21 = 84 颗蛋。
一打里面是 12 颗蛋。
所以 84 / 12 = 7。
答案是 7。

Scaling studies (Wei et al., 2022; Wang et al., 2022; Fu et al., 2022) show that larger models (e.g., 540 B PaLM) achieve higher accuracy with far fewer examples than fine‑tuned smaller models.

Knowledge Reasoning

Large models can answer factual questions without external retrieval, unlike many smaller models that rely on knowledge graphs or additional corpora. GPT‑3, for instance, matches or exceeds fine‑tuned baselines on several QA benchmarks while using only internal knowledge.

The trade‑off is that the stored knowledge may be outdated or noisy, but the ability to reason directly from the model’s parameters simplifies system design.

Out‑of‑Distribution Robustness

Studies (Si et al., 2022; Fu et al., 2022) demonstrate that prompting large models yields more stable performance under domain shift, noise, or adversarial perturbations compared with fine‑tuned smaller models.

Scaling Laws vs. Emergent Phase Transitions

Early work suggested a smooth logarithmic‑linear relationship between model size and performance (Kaplan et al., 2020). Later observations of chain‑of‑thought performance reveal a phase‑transition‑like curve: once a model exceeds a critical size, emergent abilities cause a sudden performance boost.

Implications for Paradigm Shift

If prompting large models consistently outperforms fine‑tuning on both in‑distribution and OOD tasks, research may shift toward improving prompt engineering and instruction tuning rather than solely expanding fine‑tuning pipelines.

How Large Must a Model Be?

Empirical thresholds suggest:

≈ 62 B parameters for chain‑of‑thought to surpass standard prompting.

≈ 175 B parameters for chain‑of‑thought to surpass fine‑tuned 11 B models.

However, size alone is insufficient; factors such as instruction fine‑tuning, code‑domain fine‑tuning, and dedicated chain‑of‑thought fine‑tuning also influence emergent behavior.

Beyond Scale

Models of similar size (e.g., OPT‑175B, BLOOM‑176B) sometimes fail to exhibit emergent abilities, indicating that training data, fine‑tuning objectives, and architectural choices matter.

Conclusion

The article surveys the current understanding of emergent abilities in LLMs, highlighting complex reasoning, knowledge reasoning, and OOD robustness as key examples. It discusses scaling thresholds, potential paradigm shifts toward in‑context learning, and open questions about how to reliably induce emergent capabilities through training and fine‑tuning strategies.