How Close Are Agents to AGI? Insights from Experiments and Benchmarks

Overview

This article examines the gap between current AI agents and artificial general intelligence (AGI) by presenting an experimental study, benchmark evaluations, and theoretical reflections on agent design.

Experiment: Sentence Extraction Task

A simple visual task asks participants to extract a syntactically correct English sentence from a string of characters. Colored cues help identify the correct sentence. When the same task is given to large language models (e.g., GPT‑4), they often make subtle errors such as missing letters or splitting words incorrectly.

Benchmark Evaluation

The authors evaluate their system on a three‑level benchmark designed by HuggingFace and Meta (GAIA Benchmark). Humans score around 90, while state‑of‑the‑art systems such as GPT‑4 or search‑engine‑augmented models fall far short. Their own system achieves the highest score on the test set, attracting attention from the Llama team.

Agent Design and Underlying Theories

The system, named Sibyl, uses a global workspace to store retrieved facts and decides when enough information is gathered for multi‑step reasoning. It employs tool‑calling (browser and Python executor) and a multi‑agent debate to reach a final answer.

Two cognitive theories guide the design:

Dual‑Process Theory : fast, automatic (System 1) vs. slow, deliberate (System 2) processing.

Global Workspace Theory : distributed modules communicate via a shared workspace, analogous to the model’s global memory.

Token Management and RAG Limitations

The authors note that their approach consumes fewer than 10 k tokens per query, far less than typical RAG pipelines, which discard sequential information and incur higher cost. They list three main issues with current RAG‑based agents: repetitive searches, lack of self‑learning, and unreliable long reasoning chains.

Reasoning, Chain‑of‑Thought, and Model Scaling

LLMs predict tokens by modeling residual streams; deeper layers add stability and interpretability. Experiments show that many layers can be pruned without major performance loss, suggesting inefficiencies in current model scaling. The authors also discuss in‑context learning, KV‑Database analogies, and the “stochastic parrots” problem.

From Agents to AGI

Using the GAIA benchmark, the best agents score in the 30‑40 range, far from the human 90. The article reviews OpenAI’s AGI definition and five‑level maturity model, then surveys historical perspectives (e.g., AlphaGo, Deep Blue) to illustrate that narrow expertise does not equate to general intelligence.

It introduces the ARC‑AGI benchmark, which tests abstract reasoning without prior knowledge. Current systems achieve only ~50 points, while humans reach ~85.

Q&A Highlights

Q1: How would a powerful agent solve RAG problems? – The answer describes a long, tool‑driven search workflow that balances effectiveness and cost.

Q2: Why do models learn white‑region patterns quickly? – Because those patterns dominate the data distribution and are easy to fit.

Q3: What are the trade‑offs of synthetic training data? – Small models benefit from domain‑specific synthetic data, while large models are more sensitive to noise.

Conclusion

The authors conclude that large language models still illuminate only a small part of the human intelligence spectrum and that many fundamental challenges—such as computational irreducibility and the need for better reasoning architectures—remain before true AGI can be realized.