Self-Built AI Agents: Which Capabilities Are Worth the Engineering Investment?

Start with a Representative Task to Reveal Gaps

The author uses a hypothetical order-system change — releasing reserved inventory after order closure — to illustrate what an AI coding agent must handle. The task requires understanding business constraints: which order states allow release, where reservation records live, whether closure notifications can duplicate, and whether existing consumers tolerate new messages. Without confirmed rules, the model merely encodes its own assumptions.

The author evaluates tools on three criteria: ability to locate valid requirements and design references, ability to modify code along existing module boundaries, and ability to leave verifiable test results. When rules are missing, the tool should ask questions rather than decide a business interpretation. For example, the constraint "same inventory reservation released at most once" demands tests covering duplicate notifications, checking both inventory changes and reservation records — not just a zero exit code. If the agent generates both implementation and tests, they may share the same faulty assumption.

Selection should compare how much human work remains to complete the task, not just how fast code is generated.

Three Paths: Buy, Adapt, or Build

The "Harness" — the tooling, context, permissions, and execution environment supporting the agent — can be acquired in three ways, distinguished by construction scope, not price. Off-the-shelf tools may be open-source; self-built ones may call paid models. Source access does not guarantee low-cost sustained use.

In the order example, if the gap is inventory rules and integration entry points, the priority is curating design references, providing controlled test commands, and hooking into existing CI pipelines — not rewriting the model invocation loop. If the gap is permission isolation, first check whether the current execution environment can restrict credentials and operation scope; writing "no production DB access" into a prompt is not isolation, but isolation does not require rebuilding the entire agent. Only when a critical constraint cannot be met should you compare replacing the runtime, adding external controls, or building from scratch. Learning projects and team delivery tools must be judged by different standards.

Capabilities Worth Owning Are Business-Adjacent

The author describes their own decision: integrating an agent with a proprietary ontology semantic platform. They compared DeepSeek Harness, Codex, and Pi, and chose DeepSeek Harness as the core runtime, then built needed functions and plugins on top. This choice is specific to their use case (ontology platform runtime), not a general ranking.

LLMs accelerate feature implementation but do not eliminate maintenance responsibility for the underlying code. Building on existing foundations makes "what must we build ourselves" more concrete. For the order case, the team must own: who confirms inventory rules, how far interfaces can change, what results count as acceptance evidence, and which operations require approval. These judgments live in design docs, test cases, and platforms. Tools can read and execute, but rule interpretation, change authority, and acceptance accountability cannot become "the model decides."

Worth building: project knowledge that existing tools cannot understand (distinguishing current designs from obsolete material), necessary adapters to internal systems (repeatable inventory integration test entry points, linking test reports to code), and critical control gaps. Generic capabilities — file search, model calls, log storage — can be reused if they meet constraints. Even for permissions and auditing, first check whether organizational platforms can handle them before each team reinvents them.

Mastering key capabilities does not mean owning every layer of code. It means being able to define rules, verify execution, and retain control when tools change.

Controlled Interruption Tests Expose Real Issues

After the happy path works, the author recommends a controlled interruption test. In the inventory scenario: the agent has modified files and started integration tests in an isolated environment, but the session exits before results return. On re-entry, can the tool continue correctly?

Conversation summaries help track progress but cannot replace runtime facts. If code changed after a report, the report must be re-evaluated; if test state is uncertain, the gap must be acknowledged and handled per environment rules — not claimed as success or blindly re-run. Two distinct problem classes exist: duplicate notifications causing double release (business implementation and test issue) vs. session recovery restarting tests or misusing old reports (tooling and environment issue). Business idempotency cannot be entirely delegated to the Harness.

Not every team needs full auto-recovery. Short tasks can be manually verified and restarted, provided they don't overwrite changes, create conflicts, or falsely report completion. The key is knowing the human cost of recovery, not demanding perpetual automation.

Put Subscription Fees and Maintenance Time in the Same Ledger

Tool price is visible; maintenance time is often ignored. The author splits cost into one-time integration/adaptation/migration and ongoing model calls, environment operations, upgrade compatibility, human review, and rework from failures. For individuals, measure how much maintenance crowds out product development. For teams, add: how many people reuse the same extensions, regression effort per upgrade, and whether others can troubleshoot when the primary maintainer is absent.

This does not mean off-the-shelf is always cheaper. Frequently working around product limits can consume massive manual effort; conversely, a well-scoped, long-lived custom adapter may be worth maintaining. For comparison, pick an evaluation period, record representative task outcomes, total time, human interventions, and failure causes. Do not count only successes, exclude failed or abandoned tasks, or extrapolate annual ROI from a single smooth demo.

If comparing whole product experiences, use each vendor's recommended configuration, but conclusions apply only to the full package. To compare Harnesses specifically, fix model, task, environment, and acceptance criteria, and note uncontrolled differences. Otherwise, model-driven gains are easily misattributed to the custom tool.

Summary

The author's preference: let existing solutions complete representative tasks first, log the gaps. Solve those addressable via project knowledge and adapters; only when blocked on core capabilities should you assess whether building is justified. If a custom solution does not reduce human intervention on critical tasks but continuously adds upgrade and debugging burden, reconsider its scope. Building it does not justify maintaining it.

Tools can be self-built or bought. The essential capability to own is judging whether a task is done, whether constraints are enforced, and whether the engineering investment is worthwhile.

Reference

InfoQ, Tina: "Everyone Can Build Their Own 'DeepSeek Harness', So Why Are We Still Paying for Claude Code Subscriptions?" 2026-09-04. https://www.infoq.cn/article/6Jc130IN2OaXqsPDIzmJ

This article uses that interview as a discussion starting point. The ontology semantic platform section reflects the author's current selection and development practice; the order case is a hypothetical example, not a systematic benchmark or ROI data for the three tools.