Buy, Customize, or Build? An Architect's Guide to AI Agent Frameworks
This guide helps architects and engineers decide whether to buy, customize, or build AI agent frameworks by evaluating eight key attributes—context management, memory, tool integration, standards compliance, model flexibility, remote access, observability, and hackability—and explores future trends like multi-model routing and performance-oriented serving.
"We shape our tools, and thereafter our tools shape us." — Marshall McLuhan. The next trillion-dollar opportunity lies in choosing the best model for your vehicle and the best vehicle for your model; models and vehicles must co‑optimize and co‑evolve.
This guide helps you think through how to choose your AI vehicle (framework). It starts with a practical question: which vehicle suits your role? Then, from first principles, it discusses the essential building blocks of a useful vehicle and proposes a framework to evaluate whether the simple default option is enough or whether investing in custom capabilities is warranted. Finally, it offers thoughts on where vehicle engineering is headed so you can factor those structures into your builds.
Buy, Customize, or Build?
Non‑engineers (GTM, ops, most knowledge workers): Most of the time you should not customize your own framework. Focus on finding and buying the best AI‑native applications (e.g., Harvey for legal, Clay for GTM, Descript for video). Learn their constraints and strengths, understand how their scaffolding plugs into your workflow, and let them remove a chunk of work. Out‑of‑the‑box, less‑customized experiences provide strong guardrails that prevent degradation in sensitive, mission‑critical use cases. As an end user, continuously curating and maintaining your own eval sets is not feasible. The core job is to populate these existing frameworks with the right context — i.e., become a great context engineer. A previous post (https://x.com/thealexker/status/2045203785304232162?s=46&t=qLTUK63eMdrw2KTz3cCnrg) shares ways to maximize your in‑framework experience.
Engineers (including technical tinkerers from non‑technical backgrounds): The situation is completely different. Bring‑your‑own‑model configurations let you mix frontier open‑source, closed‑source, and self‑hosted models to run frameworks that are both high‑performance and cost‑effective. As we move toward a world of long‑running, self‑managing (auto‑research) agents, you can spin up sub‑agents in parallel to handle both high‑quality deep work and high‑frequency, routine, shallow work.
To maximize framework value, you need to understand how the framework works under the hood so you can take the highest‑leverage actions. Generally, if you spot error cases or experience rough edges, log them to a file to prevent the framework from falling into the same traps and recovery loops. This is also why frequent context compression leads to the "stupid zone" (https://www.aihero.dev/ai-coding-dictionary/smart-zone). Repetitive operations like grep and exploration can be avoided by encoding a large codebase's structure into an a.md file at the repo root and instructing the framework to look there first. Unused tools can be disconnected to save context window. These practices systematically lower the probability of generating garbage and reserve tokens for where compute actually matters.
Going further, if you have a set of user trajectories and an eval suite defining "good," consider post‑training or fine‑tuning the framework together with your model to produce optimal outputs. In coding, teams like Cursor and OpenCode have created model‑specific frameworks optimized around evals — shaping system prompts, tool formats, etc., based on online and offline metrics to maximize user satisfaction and retention. In the enterprise, RL and SFT are the starting points for building differentiated intelligence and moats. But because this is the frontier, most teams are not there and don't need to be. Exhaust the out‑of‑the‑box and customization approaches first.
Attributes of a Useful Framework
Frameworks can be grouped into three categories (from least to most composable):
Frameworks & SDKs: Building blocks for constructing vehicles (e.g., Vercel AI SDK, Anthropic Agent SDK, Mastra).
Extensible: Minimal built‑in functionality beyond the basic tool‑use loop; think vanilla Emacs or Vim. Pi and Deep Agents are prime examples.
Turnkey: Feature‑complete vehicles, often tied to paid or proprietary APIs (e.g., OpenCode, Codex, Claude Code, Cursor Agent).
Choosing a framework reduces to two main components:
Framework‑task fit: Has it been post‑trained for the models and tools you need? Does it provide the right features and abstraction level for what you want to do?
Proficiency: Is this a framework you truly know and can drive effectively?
The following eight attributes diagnose the first question. No prescriptive advice is given because frameworks update weekly; the focus is on timeless principles. Evaluate your candidates against these eight; gaps tell you whether the native solution suffices or you need to build. Read them as questions to ask yourself. They fall into three groups: how the framework manages context, how it connects to the outside world, and how it runs in production.
Context & State (Intra‑task)
Does the runtime maintain stateful context across turns? How does it handle interruption and resumption? Over long horizons, what is its compression mechanism, how much information is lost, and can that loss be mitigated? How does it spin up sub‑agents to combat context corruption? Can it parallelize workflows, whether intertwined or independent?
Memory (Cross‑task)
What persists between sessions and how — local cache, database, or remote store? When are those memories retrieved and used?
MCP / Tool Support
Does the runtime support extensible integration with Model Context Protocol servers? For example, can it be configured to pull Clay data, push to Grafana dashboards, or ingest from Notion pages? What is the fidelity of its interaction with other apps? How does it let the model take actions? Is it compatible with the set of tool calls you expect?
Standards Compliance
To what extent does the runtime follow open specs vs. proprietary structures? OpenCode relies on SQLite storage and open conventions for tools, MCP, and skill installation; Claude Code uses Claude‑specific structures (claude.md, dedicated Claude directory, its own way of organizing session history in files). Open standards mean portability and less lock‑in; proprietary structures may offer tighter integration but make migration harder if you later want to share context across runtimes.
Model Selection Flexibility
How easy is it to try a new model endpoint? Can you override settings, config files, and environment variables to point to non‑native endpoints? With new frontier open‑source models dropping monthly, can you test them in your existing runtime setup, or do you need another runtime? Do all models perform equally well in this runtime?
Remote Access
Can you access sessions remotely? Does the runtime have a built‑in feature (e.g., Claude Code's /remote-control) or require third‑party/community plugins or wrappers? If you close your laptop, does the session remain accessible?
Observability / Debugging Interface
How are logs and traces captured? When things go wrong, can you catch failure modes? Are there early signals when the runtime starts drifting? Does it provide operations to roll back to an earlier state and correct course — or better yet, self‑heal when it enters a bad state?
Hackability
If you are the technical decision‑maker choosing a runtime for a team or company, this is the most critical question.
It can be viewed as a trade‑off between opinionated design and composability. A more opinionated runtime likely has less friction and is easier to adopt in the enterprise — think Copilot. But if it's less opinionated and highly malleable, there are many ways to push it to the limit, yet you'll encounter pitfalls (security holes, extreme token consumption) — think OpenClaw.
In a mature organization, picking an opinionated option with guardrails is the more appropriate path, even though it inevitably suppresses some creativity. The ideal is to find and/or customize a runtime whose failure valleys aren't catastrophic but whose productivity ceiling is as high as possible. A composable, hackable runtime empowers you to smooth out what doesn't work yourself, rather than waiting for the maintainer to maybe make that change upstream.
The Future
As we exit the "prototyping with intelligence" era and enter a multi‑year phase of pushing AI into diverse production scenarios, the hallmark of this shift is runtimes that can support, route, and orchestrate multiple models. Whether you're an individual, a team adopting a standardized runtime, or a company creating a new runtime, it ultimately comes down to two intertwined decisions: how to route, and how to serve.
Routing Across Sub‑Agents
Karpathy notes a peculiarity of LLMs: their intelligence frontier is jagged. They may have superhuman capability in some domains yet fail badly in others. The corollary: specialization will become the dominant paradigm — using multiple models and routing each task to the right model via independent sub‑agents makes sense.
An increasingly common and strategic pattern is dynamic routing inside the runtime: a lightweight classifier decides whether to route to a solid open‑source default to save cost, to a fine‑tuned model that maximizes latency or throughput, or to a frontier closed‑source model for a second opinion. Cache reuse is critical. In multi‑turn conversations, routing to different models can produce cache hits. For optimal speed and cost, the runtime architecture should be cache‑aware so requests hit the hot cache associated with a specific model whenever possible.
For read‑only tasks like survey or research, sub‑agents can use smaller, faster models. These sub‑agents (or multiple replicas) can be spun up quickly to handle focused, parallelizable tasks — e.g., reading from many sources and synthesizing findings on a topic. This sub‑agent layer also aids modularity. Just as OOP organizes around classes and encapsulation, sub‑agents are a pattern that boosts efficiency and isolates work into separate modules. And because context degradation is a primary cause of quality drop and hallucination, running multiple models independently keeps contexts clean.
Serving for Performance
Once routing is in place and every workload is matched to the right kind of intelligence, the model's own performance directly determines user experience — and that experience depends on the infrastructure surrounding inference.
Choosing the right framework matters, but equally critical is selecting the right model for your agentic use case. For instance, GLM 5.2 is an excellent candidate on both quality and performance. Bring‑your‑own‑model frameworks or proxies let you flexibly pick the best model‑framework combination. The LangChain team created Harness Profiles specifically for this (https://x.com/Vtrivedy10/status/2049535740233523600).
Regardless of vendor, the principle worth keeping is: performance is part of framework‑task fit, and BYO‑model is how you control it. If you need guaranteed rate limits, burst handling, and uptime, hosting your own open or custom models is the way to achieve those. Dedicated infrastructure gives you control over caching, batching, and disaggregation to squeeze latency without sacrificing quality. We expect mature builders to serve frameworks with performance at the core and continue finding optimizations in that direction.
Final Thoughts
Frameworks are not one‑size‑fits‑all solutions and should not be treated as such. We've outlined how to handle framework engineering from both technical and non‑technical angles: when to buy, when to customize, when to build from scratch.
As frameworks are constantly added and deprecated, specific recommendations age quickly. That's why we didn't give concrete advice but instead shared a set of timeless questions to help you understand and guide your own decisions. Over time, the best framework attributes will be standardized and commoditized, making all frameworks intuitive to use. My bet is that routing intelligence to the right model and having high‑performance inference will eventually be baked into the framework itself and become table stakes as it evolves.
The core takeaway: pick your non‑negotiables (e.g., strong compression because you have massive context, operability across multiple models, remote access, etc.) and master one specific framework. As proficiency grows, you'll find ways to fork, modify, or orchestrate the framework to do exactly what you need.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
