Harness Is the Product: 12 Design Principles for Production-Grade AI Agents

Why Harness Matters More Than the Model

A production-grade AI agent must run continuously for hours, use dozens or thousands of capabilities, remember organizational practices, operate production systems within safety boundaries, and handle context-length limits. It must also reach users across platforms like Feishu, WeCom, Slack, Teams, SaaS tools, or voice calls while maintaining consistency. The architecture around the model — the Harness — is as important as the model itself.

The classic agent loop (

while not done: response = llm(messages, tools); execute(response.tool_calls)

) remains the core, but by 2026 a mature Harness relies on a full stack of subsystems behind every word in that loop: messages, tools, execution, and the loop itself.

The loop still exists, but it no longer represents the whole system architecture.

Analogy: a strong horse (model) needs a harness (framework) to be controlled, directed, and safely reused. The Harness encodes rules, boundaries, processes, and the definition of "done" — where engineering discipline meets model capability.

Industry Convergence on Harness

Products like Viktor (Slack/Teams AI assistant, $75M Series A, 3000+ tool connections), OpenAI's dots (persistent agents with cloud computers, cross-app memory, approval controls), and Meta's Muse (personal agent on a VM, callable from WhatsApp) all share a similar shape: multiple entry points (Slack, Teams, WhatsApp, ChatGPT, voice) backed by a single, portable Harness that holds identity, memory, permissions, sandbox, and approval rules. The agent can switch entry points because its Harness moves with it.

Core thesis: the model is the engine, the entry point is just a door, the real product lives in the Harness.

1. Don't Dump All Tools on the Model: Let It Discover Capabilities by Task

An enterprise agent may connect to Feishu, WeCom, Feishu Docs, Alibaba Cloud, GitHub, Jira, Slack, Salesforce, Stripe, AWS, Datadog, ServiceNow, Notion, PostgreSQL, and dozens of internal systems — hundreds of MCP servers, each exposing 20-30 tools, totalling thousands of operations.

Naively passing all tools to the model picks the wrong abstraction level. The fundamental unit of agent reasoning should be a task unit (operation) — an explicit intent like "list Alibaba Cloud services" or "calculate monthly cloud spend" — not a low-level command or UI click.

Task units (operations) represent the intent to achieve; agents should identify and organize user needs at this level.

Three implementation patterns for task units:

Skill : natural-language description of which tools to call in what order, how to aggregate, and what output format to use; the model then invokes tools or writes code accordingly.

Composite tool : multiple steps bundled inside MCP server code; the agent calls it once, yielding predictable results and letting the policy layer treat it as a single operation.

Simple tasks may directly call a single existing tool (e.g., "which Alibaba Cloud services are we using?" → one MCP tool).

Many flows start as skills and later "compile" into composite tools once stable and critical.

Capability = boundary by connected system (e.g., aws, github, jira), grouping related tools and serving as connection, auth, and policy unit. For the agent, capabilities also act as metadata filters to find relevant tools without scanning every catalog.

Capability → system-level boundary (aws, github, jira)
Tool      → single action with defined input schema
Skill     → references tools, describes how to combine them
Operation → work unit the agent looks up and executes, implemented by any of the above

With these distinctions, the agent discovers needed tools on demand. Tool I/O schemas are defined once in MCP; skills reference tools by ID, and the Harness resolves definitions into the model context at load time. Simple tasks stay simple: a query returning one tool goes straight to execution without skill indirection. list_capabilities() lets users ask "what can you access?" and the agent can disambiguate when multiple connected systems (e.g., Jira and Linear) offer similar operations.

2. Load Knowledge on Demand: Don't Stuff All Skills into the Prompt

Separating "what to do" from "how to do it" clarifies Harness components:

Memory       → what we know
Skills       → how we work
Capabilities → which systems we can connect
Tools        → what the systems can execute
Operations   → what the agent actually needs to accomplish

Mixing these causes chaos (e.g., stuffing full workflows into tool descriptions or tool I/O into workflow docs). As the agent accumulates hundreds of skills (incident handling, release management, onboarding, security reviews, architecture reviews), loading all into the prompt repeats the tool-overload problem. Better: show the model only skill names and one-line descriptions; load full workflow into working context only when the task requires it. For dozens of skills, keeping names in the system prompt is fine; for large, frequently changing tool sets, use search.

On-demand loading applies to knowledge as well as capabilities.

3. Memory Isn't Chat Summaries: Layer, Retrieve, and Curate

Common naive approach: summarize conversation, store summary, stuff into context every turn. Works for tiny systems; fails at scale. A proper architecture distinguishes at least three things:

Working context (what's in the current window)

Durable memory (what's worth keeping for later)

Memory retrieval (how to fetch relevant info at the right time)

Conversation ≠ memory; memory is the distilled, worth-keeping part of conversation.

Retrieval should combine automatic and active modes. Before calling the main model, the Harness can automatically inject relevant project preferences, past architecture decisions, or org constraints into the initial context. But automatic retrieval can't anticipate everything, so the model also needs an explicit search_memory(query) tool to proactively fetch more when mid-task it realizes it needs extra background (e.g., "what naming convention does the team use for production deployments?").

User request
  ↓
Auto-retrieve memory → inject into initial context
  ↓
Agent starts processing
  ↓
"I need more context"
  ↓
search_memory(...) → fetch additional memories

Memory curation should be a separate, asynchronous pipeline. If a user says "stop using PlantUML, switch to Mermaid," storing the whole chat is useless. The valuable artifact is a curated fact: "Architecture diagrams use Mermaid, not PlantUML." Extracting, evaluating relevance, normalizing phrasing, deduplicating, and merging are batch jobs run after a conversation ends, not in the real-time path.

Memory must have explicit scope. In enterprise settings, default global sharing creates risk. User memory (personal preferences), workspace memory (team norms, architecture decisions, role definitions), task/project memory (current goals, intermediate findings, open issues), and skill knowledge (procedures, known integration gotchas) have different owners and lifecycles. Personal preferences must not silently become org policy; project decisions must not silently become company-wide rules. Memory needs explicit scope, provenance, owner, and TTL; vector similarity alone is not a memory architecture.

4. After Retrieval, Filter: 2026 Reference — Jev Model

Retrieval returns candidates (e.g., 40 memory chunks, 25 candidate tools, 12 relevant skills) but doesn't answer "which of these should actually go to the agent?" Traditional stack: embeddings, heuristics, cross-encoders, rerankers, another LLM call, or hybrids.

In 2026, a new option: Jev (and similar decision-oriented models) from TypeSafe AI. Jev is not a chat LLM; it's a "System One" model (Kahneman's fast thinking) for rapid structured decisions. Input: context + a constrained-choice question. Output: categories, scores, probabilities — not generated text. Like a BERT with a classification head, but labels and question are defined per request, so you don't train a separate classifier per decision.

Fits many Harness tasks:

Retrieved memories → select/rerank → final memories injected into context
Candidate ops      → filter/score   → ops worth loading
Incoming request   → classify       → routing / model / policy signals

Public benchmarks are mixed: Parallel found Jev matches or beats their custom models in some scenarios but struggles with many labels; at least one public report shows Jev didn't beat their vector baseline. Deterministic rules, dedicated classifiers, cross-encoders, plain vector similarity, or a small LLM may be better for your case. Evaluate with your own data.

Broader architectural lesson: filtering doesn't have to be generation. Frontier reasoning models shouldn't automatically own classification, routing, reranking, relevance, and policy scoring just because they can . The Harness should narrow the decision space first, then let the expensive model do real reasoning.

Harness constantly makes small decisions: which memory space to search? which capability is relevant? which operation to load? which candidates deserve context? does this request need the strongest model? These are classification, routing, ranking, filtering — not generation. The right question isn't "should we add Jev?" but "is this a generation problem or a constrained-choice filtering problem?" Correct classification saves context, latency, and cost, reserving the reasoning model for where it's truly needed.

5. Let Code Handle Repetition: Don't Make the Model Orchestrate Every Call

The classic loop alternates model ↔ tool calls. Fine for few steps. But processing 500 records across three systems with per-record model orchestration is slow, expensive, and error-prone.

After the agent understands the task and available tools, it can write a program instead:

customers = crm.search_customers(status="active")

results = []
for customer in customers:
    invoices = billing.list_invoices(customer_id=customer.id)
    overdue = [i for i in invoices if i.status == "overdue"]
    if overdue:
        results.append({"customer": customer, "invoices": overdue})

Flow becomes: model generates execution logic → sandbox runs code calling tools → runtime aggregates and transforms data → only results return to model. LLM reasons the workflow; Harness handles repetitive, mechanical execution.

Once code execution is core, the sandbox becomes part of the agent — not a side calculator but the agent's actual work computer, with filesystem, shell, Python, temp DB, Git repos, downloaded files, generated artifacts, and tool proxies. This explains why OpenAI's dots, Meta's Muse, and serious coding agents give each agent its own cloud machine — a fundamentally different architecture from an LLM behind a REST endpoint.

6. API First, Browser as Fallback

Computer-use capabilities are strong but shouldn't auto-replace APIs. If a deterministic API call fetches an invoice, driving a browser through six pages is slower, flakier, and harder to audit. Reasonable capability priority:

Structured API / MCP → prefer
Code execution       → for composition & transformation
Browser / computer   → fallback, or for UI-only systems

Browser is justified when no API exists, API is incomplete, legacy systems only have UI, you must verify rendered content, or the task is UI interaction. It should extend the agent's capability boundary, not replace good software interfaces.

7. Credentials Stay Out of Model Context: Every Execution Passes Policy

If the agent touches many systems, credential architecture is critical. Wrong: put STRIPE_API_KEY, GITHUB_TOKEN near model context. Right: model expresses intent only; system handles the rest.

The model only needs to know "I can read invoices," not "here's an OAuth refresh token." Capabilities and credentials are separate concerns and must be managed separately.

Policy lives at the same gate. Agent deciding to act ≠ action should execute. search_customer() and refund_customer($18,000) are both valid tools but need different execution rules. Policy layer classifies by risk:

Read                → execute directly
Low-risk write      → execute + audit log
Sensitive write     → confirm first
Destructive/high-impact → require explicit approval

Real execution path: intent → operation → policy engine → permission check → risk assessment → execute or wait for human approval. OpenAI's dots uses this pattern: user-defined rules, sensitive ops require consent. Production Harness can follow similar design.

8. High-Risk Ops Intercepted by System; Approval Not Left to Model

Example: agent prepares a production deploy of payments-api v4.7.2 replacing three pods. Bad: model types "continue?" in chat and hopes to parse reply. Better: Harness emits a structured pending action . The UI renders real "Approve/Reject" buttons; user decision resumes the paused execution. Human-in-the-loop becomes part of the execution lifecycle, not a bolted-on UI. The interface participates in the execution protocol — exactly what MCP elicitation and MCP Apps enable.

Crucially: whether to approve cannot be decided by the model. When a high-risk execute_operation hits the policy engine, Harness must pause and return "pending" regardless of model intent. A separate ask_human() tool serves the model when it lacks info (which month? which task system — Jira or Linear? test or prod environment?). For short sync chats, user reply suffices; for long background tasks, the mechanism is essential: task saves state, pauses, sends question to appropriate entry (Slack button, Teams card), resumes from answer. On screen they may look similar, but they live at different layers: model asking for info vs. system enforcing policy.

9. Different Tasks, Different Models

"Not every decision needs a frontier model" applies to the main model too. Harness can route based on task complexity, risk, latency budget, cost, and type: fast model, balanced model, frontier reasoning model, coding model, vision model, or specialized classifier. Flow: task → classify → required capability → select model → execute. Many apps don't need dynamic routing; but at thousands to millions of tasks, model selection becomes an architectural concern, not just a config knob.

10. Agent Shouldn't Be Bound to One Chat App

Many agents start as a Slack bot; later asked to support Feishu, half the codebase rewrites — proof that communication entry and agent runtime are over-coupled. Better: every channel is an adapter to the same runtime.

Unify diverse events (Slack events, Teams activities, WhatsApp messages, voice transcripts, API requests) into a single internal format:

{
  "actor": "...",
  "workspace": "...",
  "conversation": "...",
  "message": "...",
  "attachments": [],
  "surface": "slack",
  "permissions": []
}

Harness processes only this unified format. On output, the entry adapter translates agent responses into channel-specific forms: Slack blocks, Teams adaptive cards, web components, voice replies. Pending approvals (section 8) appear as buttons in Slack, cards in Teams, voice prompts — same operation and state, different presentation. This is why Viktor, dots, and Muse can live in users' existing workspaces. The agent can swap entry points without swapping its brain. The lasting value isn't the Slack bot; it's the persistent agent identity and its Harness.

11. Long Tasks Must Pause, Resume, and Never Lose State

When agents run for minutes, hours, or days, the assumption "one HTTP request = one agent task" breaks. Long tasks need durable execution state: goal, current plan, completed steps, artifacts, pending ops, approvals, failures, checkpoints; and reliable transitions between "running → waiting for API → waiting for human → running → done" without data loss. The system starts to resemble a workflow runtime, not chatbot infra. Modern agents are becoming stateful distributed systems with an LLM in the control loop. MCP's new background-task and multi-turn request specs reflect the protocol catching up to this reality.

12. Record Execution Traces, Improve Continuously via Evaluation

Traditional monitoring checks request success, latency, 500s. Agent observability needs far more: what goal did the agent think it had? Which memories were retrieved? Which skill loaded, which capability chosen, which operations attempted? What policies triggered, which approvals requested, what code ran? Where did it fail, how did it recover? Total cost? Did the result actually achieve the goal?

The atomic unit of observability is the full execution trace . Without traces, debugging agent behavior is guesswork.

This leads to system evaluation (eval) . Harness shapes agent behavior. Changing memory retrieval, tool descriptions, skill instructions, model routing, context compression, or approval rules — even with the same model — can change outcomes. Therefore eval must measure the whole system, not just the model:

Model + Context + Memory + Skills + Tools + Policy + Execution = Agent Behavior

Metrics to track: task success rate, tool selection accuracy, unnecessary tool calls, policy violations, failure recovery rate, latency, token consumption, cost per successful task, human intervention rate, memory retrieval quality.

Execution traces enable observability; observability enables evaluation; evaluation drives Harness improvement. Most significant quality gains come from this loop.

Putting It All Together

Complete mental model (see diagram in source). Key takeaway:

The model sits inside the architecture, not the architecture inside the model.

With responsibilities separated, the model-facing interface becomes surprisingly minimal:

search_memory(query)

list_capabilities()
search_operations(query, capability?)
load_operation(id)
execute_operation(id, args)

run_code(code)
delegate_task(task)
ask_human(question, options?)

Discovery relies on three functions: list_capabilities shows connected systems; search_operations finds suitable task units from tools, composite tools, and skills; load_operation loads only needed tool definitions or operation instructions. Approval is deliberately absent — it's not a model-initiated call but a Harness-enforced rule.

Behind these simple interfaces may lie hundreds of skills, thousands of tools across dozens of MCP servers, millions of memories, multiple models, and MCP servers, browsers, databases, cloud environments. Complexity hasn't vanished; it's been moved behind on-demand discoverable abstraction layers instead of being stuffed into every model call. This, not endlessly adding tools to prompts, is the more interesting way to scale agents.

Harness Is Becoming the Product Itself

AI engineering evolved from "app + LLM API" → "app + LLM + RAG + tools" → now a full Harness: context management, memory, on-demand-discoverable operation catalog, filtering, tool execution, code runtime, browser, model routing, policy, identity, credentials, human approval, durable state, observability, evaluation; the model plugs in.

Models remain crucial. But swapping a strong model is getting easier compared to rebuilding everything around it. What truly accumulates: operational knowledge, permissions, memory, integrations, workflows, security boundaries, eval frameworks, UX, agent identity.

Back to the opening point: Harness increasingly is the product; the entry point is just where people access it today.

Not a Copy-Paste Blueprint

Not every AI app needs all of this. Five tools, short sync tasks → pass five tools to model. No long-term user info → no complex memory. Vector search works → rerankers/decision models add latency and maintenance. No critical ops → no approval machinery. One model handles all → dynamic routing adds needless complexity. Architecture should follow real constraints, not hype.

Scope disclaimer: this describes a high-level, relatively lean architecture suitable for public discussion. Serious products build proprietary ranking methods, memory heuristics, eval datasets, safety mechanisms, orchestration details — those are the real competitive moats and should stay private. But as agents become persistent, deeply integrated, multi-entry, long-running, and responsible for important work, a common skeleton emerges, worth discussing together. Some systems need 20%; some need more.

The Real Shift: From Stronger Models to More Complete Systems

The most interesting change in agent engineering isn't just models getting better , but the systems around models getting more complete :

From "which tools to give the model?" → "how to discover the right operations?"

From "how much chat history to stuff?" → "which information deserves to become memory?"

From "how to retrieve relevant context?" → "how to retrieve, rank, filter, and inject only context worth paying for?"

From "how to write the system prompt?" → "what knowledge does this task need to load?"

From "how to call three tools?" → "should the model orchestrate each call or just write a program?"

From "can the agent do this?" → "does it have the right, which identity and credentials, does it need human approval?"

We're moving from "built a Feishu bot" to "built an agent that can be used via Feishu." That last shift may be the most important.

Closing: Turning Model Intelligence into Controllable Capability

Without a good harness, even the strongest horse is hard to drive. It may run fast, be clever, even surprise you. But production systems can't be built on surprises — they need controllable capability .

Modern Harness transforms model intelligence into a capability that runs continuously, discovers on demand, actually executes, is continuously observed, works across entry points, and accepts governance.

Model handles reasoning; memory stores knowledge; operations define what the agent must achieve, skills describe how, capabilities connect real systems. Filtering removes noise first; sandbox provides workspace; policy draws boundaries; approval keeps humans in control. Entry points are doors; Harness connects the parts.

When evaluating the next agent architecture, don't just ask how smart the model is — also ask:

What system have you actually built around that intelligence?