What’s the Real Difference Between LLMs and Agents? What Does an Agent Add?

Core distinction

Large language models (LLMs) are stateless: each invocation receives a prompt and returns a response, then discards all context. Agents are stateful: they keep track of what has been done, the results obtained, and the remaining goal, enabling multi‑step execution.

Why LLMs cannot handle multi‑step tasks

Because they lack state, an LLM must be fed the result of every previous step manually. In a ticket‑booking scenario the user would have to copy the JSON output of a flight‑search API, paste it into a new prompt to request price comparison, then paste the comparison result to ask for seat selection, and so on. The model never remembers prior actions.

Agent state

Agent state is implemented as three concrete data structures:

Execution history : a log of every action (e.g., "called flight‑search API with Beijing→Shanghai tomorrow"). The log is consulted to avoid repeating failures.

Intermediate results : the data returned by each action (e.g., the list of five flight options) that later steps can reuse without re‑querying.

Goal tracking : the overall objective and current progress (e.g., "book cheapest ticket" – progress: queried and compared prices, still need seat selection and payment).

These layers are usually stored in a Memory component. Simple agents may use a Python dict; more complex agents may use a vector database for long‑term memory and a conversation buffer for short‑term memory.

Tool calling vs. Agent

Prompting an LLM to output a JSON tool‑call does not make it an Agent. An Agent must also:

Execute the generated JSON command.

Feed the execution result back to the LLM.

Let the LLM decide the next step based on the observation.

This closed‑loop of reasoning → action → observation → reasoning is the defining characteristic of an Agent.

Uncertainty and cost

Uncertainty : Each step is generated by a stochastic LLM, so the execution path can differ between runs. For the same ticket‑booking task, one run might query all flights first, another might query price ranges first, and a third might follow a retry path after an API timeout. This variability makes debugging difficult because the natural‑language log may not be reproducible.

Cost : Every step re‑sends the full execution history, causing token usage to grow linearly. A five‑step task may consume ~10,000 tokens (e.g., 1 000 → 1 500 → 2 000 → 2 500 → 3 000 tokens per step) instead of ~5,000 tokens for a single LLM call. In high‑throughput scenarios—e.g., a customer‑service Agent handling 10 000 queries per day—LLM usage can reach thousands of dollars.

Failure case and remediation

When only the current step prompt is sent to the LLM, the model forgets earlier actions, leading to repeated or out‑of‑order tool calls. Adding a Memory object that serializes the complete history into each prompt resolves the issue and enables reliable multi‑step execution.

Comparison

LLM : single‑turn prompt → response; no state; deterministic; low token cost; easy to debug.

Agent : loop of reasoning → action → observation → reasoning; maintains execution history, intermediate results, and goal tracking; nondeterministic; token cost grows with the number of steps; debugging requires inspecting natural‑language reasoning logs.

When to use an Agent

Agents excel at tasks that require multiple dependent steps, dynamic decisions based on intermediate results, and tool usage—e.g., complex data analysis, multi‑round information gathering, or problems that need iterative trial‑and‑error. For tasks that can be expressed as a fixed pipeline, Chains or direct code are cheaper, more deterministic, and easier to maintain.

Illustrative images