AI Agent Architecture: Chain‑of‑Thought, ReAct, and Tool Calls

Part 1: Black Box

When a user sends a message, the agent receives it and either replies or performs an action. In demo settings this simple flow works, but in production the lack of visibility makes debugging and fault isolation impossible.

Without a visible internal process, agents become a black box that cannot be debugged, scoped, or given acceptance criteria.

Part 2: Chain-of-Thought

The simplest architecture is Chain‑of‑Thought (CoT). After receiving input, the model breaks the problem into a series of reasoning steps and executes them sequentially until a conclusion is reached. No external tool calls are involved.

Each step uses the previous step’s output as context, creating a traceable logical chain. This makes the process transparent, fast (no API latency), and cheap (single LLM call with extended output). However, the model cannot access information beyond its training data or the current context.

CoT is optimal when the answer fits entirely within the context window; it fails when real‑time data such as stock prices or database queries are required.

Part 3: ReAct

ReAct (Reasoning + Acting) extends CoT by allowing the model to interact with the external world during reasoning. The loop is: think → act → observe → think again.

This enables an agent to, for example, search for flights, read the results, notice a cheaper one‑stop option, and then re‑plan for a direct flight. Each iteration replaces fabricated content with real data, but more iterations increase latency and the chance of the loop drifting.

The number of iterations is the most critical tuning parameter; too few and the agent quits early, too many and token usage and cost explode. Starting with a maximum of five iterations and adjusting based on online data is recommended.

Part 4: Tool Usage

Tools turn a chat‑bot into an agent by giving the LLM the ability to search the web, query databases, send emails, or run code. The LLM does not execute the operation directly; it generates a structured request (usually JSON) that an external executor carries out.

This separation is crucial: the LLM decides which tool to use and what parameters to pass, while the executor handles authentication, rate‑limiting, and actual execution. Poor tool descriptions lead to incorrect selections (garbage in, garbage out), and each tool call introduces potential failure points such as timeouts or malformed responses.

Tool descriptions are essentially product copy for the AI; they must clearly state what the tool does, when to use it, what it returns, and when not to use it.

CoT and tool usage are not mutually exclusive; CoT provides the reasoning that guides correct tool invocation. Modern models like OpenAI o1, Claude 4.6 sonnet, and Gemini 3.1 pro embed the CoT phase natively, automatically performing hidden reasoning before emitting a tool call.

Part 5: Choosing Architecture

Production agents usually blend elements of the three architectures, but the dominant pattern determines latency, cost, and failure characteristics.

Higher autonomy yields better accuracy but also higher latency, cost, and error surface. A pure CoT agent is fast and cheap for tasks like document summarisation, whereas a tool‑enhanced ReAct agent that calls three APIs for flight, hotel, and car bookings offers richer capabilities at greater expense.

Guideline: start with pure CoT for self‑contained reasoning; add read‑only tools when external data is needed (switch to ReAct); adopt read‑write tools for actions with real‑world impact. Keep the architecture as simple as possible because once users depend on added complexity, it becomes hard to remove.

Choose the simplest architecture that solves the problem; tools can be added later, but complexity is hard to unwind once adopted.

Core Thinking Model

"An agent is a loop with opinions—our job is to decide how many opinions the loop holds and when it should terminate."

Chain‑of‑Thought represents a single‑round loop (think → answer). ReAct adds an action step inside the loop, and tool‑enhanced agents equip the loop with external interfaces. Every architectural decision ultimately controls the loop’s autonomy and termination conditions.