Why Building AI Agents Is Still a Messy Challenge

Building agents remains messy. The author reflects on recent lessons, focusing on the complexities of agent construction and the use of agentic coding tools.

Choosing an Agent SDK

When building a custom agent you can target low‑level SDKs such as the OpenAI or Anthropic SDKs, or higher‑level abstractions like the Vercel AI SDK or Pydantic. The author’s team initially adopted the Vercel AI SDK but only used its provider abstractions, driving the agent loop themselves[1]. They would not repeat that choice because model differences are significant enough to require a bespoke agent abstraction, and existing SDKs do not provide the right abstractions for caching, reinforcement, tool prompts, or provider‑side tools.

Using the Vercel SDK also caused concrete problems: Anthropic’s web‑search tool frequently corrupts the message history, and Anthropic’s cache management is simpler when using its native SDK, with clearer error messages.

At present the team prefers to avoid higher‑level abstractions until they become more stable.

Caching Experience

Different platforms handle caching very differently. Anthropic requires explicit cache points and charges for caching, which forces the author to manage cache manually. Although initially seen as cumbersome, explicit caching improves cost predictability and enables advanced techniques such as splitting a conversation into parallel branches and context editing.

The author’s Anthropic caching layout places one cache point after the system prompt and two at the conversation start, with the latter moving upward as the conversation progresses.

Because system prompts and tool selection must stay mostly static, dynamic messages (e.g., current time) are injected later to avoid invalidating the cache. Reinforcement is also leveraged more heavily in the loop.

Reinforcement in the Agent Loop

Each tool call can return data and also feed additional information back into the loop, such as reminding the agent of the overall objective, task status, or hints for failed tools. Reinforcement can also signal background state changes, especially for parallel processing.

Self‑reinforcement tools like Claude Code’s todo‑write tool simply echo a list of tasks back to the agent, which proves sufficient to keep the loop progressing despite many intervening events.

Reinforcement is also used to inform the system when a failure requires back‑tracking and retrying earlier steps.

Isolating Failures

When many failures are expected, the author isolates them by running flaky tasks in a sub‑agent until they succeed, then reporting only the success (and optionally a brief summary of failed attempts). This helps the main agent learn from what didn’t work without cluttering the context.

Anthropic supports context editing, which can remove unhelpful failure tokens while preserving useful ones, though it automatically invalidates caches, making the trade‑off unclear.

Sub‑Agents and Virtual File System

Most agents rely on code execution and generation, requiring a shared storage location. The author uses a virtual file system so that tools like ExecuteCode and RunInference can read and write the same files via paths.

Designing agents without dead ends is crucial; tools must be able to exchange data (e.g., an image generation tool writing to a location that a code execution tool can later read for archiving).

Using an Output Tool

The agent does not represent a chat session; intermediate messages are hidden. An explicit output tool is used to communicate with humans, currently sending an email. Steering the wording and tone of this tool proves surprisingly hard, likely due to model training quirks.

Attempts to post‑process the output with a fast LLM (e.g., Gemini 2.5 Flash) increased latency and reduced quality, and sometimes leaked unwanted internal steps.

When the loop ends without invoking the output tool, a reinforcement message is injected to encourage its use.

Model Selection

The team finds Haiku and Sonnet to be the best tool callers for the main loop, offering relatively transparent reinforcement behavior. Gemini models are also viable, while GPT‑family models have not performed well for the core loop.

For sub‑tools that need inference, Gemini 2.5 is preferred for document summarization, PDF handling, and image extraction, whereas Sonnet often triggers safety filters.

Token cost alone does not determine agent expense; a more efficient tool caller can reduce overall token usage even if the model itself is not the cheapest.

Testing and Evaluation

Testing and evaluation are the hardest problems. Because agents require extensive context, external evals are insufficient; the author advocates observability‑driven evals based on instrumented test runs, but no satisfactory solution has been found yet.

Coding Agent Updates

The author is trialing Amp more frequently. While not objectively superior, Amp’s design of sub‑agents (e.g., an Oracle) interacting with the main loop is praised for its elegance and helps validate different agent designs. Amp feels similar to Claude Code, built by people who also use their own tools.

Quick Observations

Minimalist browser‑agent setups can avoid over‑engineered MCP servers by using simple CLI tools (start, navigate, evaluate JS, screenshot) executed via Bash, keeping token usage low.

The era of tiny, single‑purpose open‑source libraries may be ending as platform APIs and AI tools generate utilities on demand.

Tmux is recommended for any interactive system an agent should work with.

LLM APIs present a synchronization problem, discussed in a separate article.

References:

[1] https://ai-sdk.dev/cookbook/node/manual-agent-loop

[2] https://earendil.com/

[3] https://ampcode.com/

[4] https://mariozechner.at/posts/2025-11-02-what-if-you-dont-need-mcp/

[5] https://github.com/mitsuhiko/agent-commands/tree/main/skills/web-browser

[6] https://nolanlawson.com/2025/11/16/the-fate-of-small-open-source/

[7] https://lucumr.pocoo.org/2025/1/24/build-it-yourself/

[8] https://x.com/mitsuhiko/status/1991997262810218983

[9] https://github.com/mitsuhiko/agent-commands/tree/main/skills/tmux

[10] https://lucumr.pocoo.org/2025/11/22/llm-apis/