Why the Real Production Bottleneck for AI Agents Is the Harness, Not the Model

Real Production Bottleneck for AI Agents

In prototype stages teams often focus on swapping models, adding prompts or few‑shot examples. This works while the problem is "does the answer look correct?" In real business scenarios the bottleneck moves to the Harness – the execution system that wraps the model. The harness is analogous to an operating system: it schedules, isolates, handles I/O, persists state, enforces permissions and provides observability.

Three Engineering Stages

Prompt Engineering   →   Context Engineering   →   Harness Engineering
what to ask               what the model sees          make the system safe, stable, recoverable

Prompt Engineering : focuses on task description, output format and few‑shot coverage. Assumes a reliable execution environment.

Context Engineering : adds retrieval, session memory, history summarisation and context trimming, but still leaves execution‑level concerns to the model.

Harness Engineering : solves execution‑system problems – scheduling, isolation, I/O, state persistence, error handling and observability.

ETCLOVG Seven‑Layer Model

E

– Execution Environment (sandbox, isolation) T – Tool Interface & Protocol (validation, rate‑limiting, timeouts, approvals) C – Context & Memory (state budgeting, hierarchical context) L – Lifecycle & Orchestration (state machine, retry semantics, human approval) O – Observability (per‑step metrics, tracing) V – Verification & Evaluation (exact match, functional test, LLM‑as‑judge, human review) G – Governance (model/tool/runtime boundaries, audit)

Typical Failure Order (E → T → C → L)

E – command runaway, environment contamination, privilege overrun (no isolation boundary)

T – tool timeout, parameter distortion, erroneous retries (no stable protocol)

C – context explosion, goal drift, important information loss (no explicit state budgeting)

L – mid‑task failure, unrecoverable state, infinite loops (no explicit state machine)

Later Layers (O, V, G)

O – without observability you cannot locate the failure (model, tool or sandbox).

V – without verification success rates are misleading; agents can appear to finish while producing wrong output.

G – governance defines what the system may do, cannot do and how violations are audited; essential for multi‑team, high‑risk deployments.

Execution Environment (E Layer)

Production agents often need to read/write files, run shell commands, open browsers or access networks, turning the execution environment into part of the business boundary. Choose isolation based on risk and acceptable startup overhead:

Sub‑process / WASM – low‑to‑medium isolation, very low startup cost – suitable for pure computation, lightweight scripts, no external side‑effects.

Docker container – medium isolation, medium startup cost – suitable for standard toolchains, code changes, file operations.

Firecracker microVM – high isolation, medium startup cost – suitable for multi‑tenant, high‑privilege tools.

Full desktop VM – very high isolation, high startup cost – suitable for browser automation, GUI testing.

A common mistake is to over‑engineer isolation for low‑risk tasks; instead classify tasks by risk and match them to an appropriate runtime.

Tool Interface & Protocol (T Layer)

Four typical tool‑related failure modes:

Unstable parameters generated by the model cause call failures.

A single slow tool drags down the whole pipeline.

High‑value tools lack rate‑limiting and circuit‑breaking, leading to cascade failures.

Tool output is too large, polluting the model's context.

Engineering focus should be on "protocolisation" and "constraint enforcement" rather than merely exposing more function schemas.

{
  "name": "execute_sql",
  "version": "2.1.0",
  "protocol": "mcp",
  "endpoint": "grpc://sql-executor:9090",
  "rateLimit": {"rpm": 100, "concurrency": 5},
  "timeout": "30s",
  "schema": {
    "type": "object",
    "properties": {
      "query": {"type": "string", "maxLength": 2000},
      "database": {"type": "string", "enum": ["prod_ro", "staging"]}
    },
    "required": ["query", "database"]
  },
  "auth": {"scope": "db:readonly"},
  "retry": {"maxAttempts": 2, "backoff": "exponential"}
}

Tool router implementation (Go) that enforces validation, rate‑limiting, circuit‑breaking, timeout handling and error classification:

type ToolRouter struct {
    registry    map[string]*ToolSpec
    breaker     *circuitBreaker
    rateLimiter *rate.Limiter
    logger      *slog.Logger
}

func (r *ToolRouter) Invoke(ctx context.Context, tool string, args json.RawMessage) (json.RawMessage, error) {
    spec, ok := r.registry[tool]
    if !ok {
        return nil, ErrToolNotFound
    }
    if err := spec.Validate(args); err != nil {
        return nil, fmt.Errorf("validation failed: %w", err)
    }
    if !r.rateLimiter.Allow() {
        return nil, ErrRateLimit
    }
    if err := r.breaker.Ready(); err != nil {
        return nil, err
    }
    callCtx, cancel := context.WithTimeout(ctx, spec.Timeout)
    defer cancel()
    start := time.Now()
    resp, err := spec.Invoke(callCtx, args)
    latency := time.Since(start)
    r.logger.Info("tool invoked", "tool", tool, "latency_ms", latency.Milliseconds(), "resp_bytes", len(resp), "error", err)
    if err != nil {
        r.breaker.Failure()
        return nil, classifyToolError(err)
    }
    r.breaker.Success()
    return resp, nil
}

Production‑grade router must also:

Validate parameters (prevent dirty data from the model).

Enforce call timeouts (prevent a single tool from hanging the whole task).

Classify errors for proper retry or circuit‑break decisions.

Structure‑trim tool results before feeding them back to the model.

Introduce approval or double‑confirmation for high‑risk tools.

Context Management (C Layer)

Three failure modes related to context: Context blindness – important information is drowned out. Context drift – long tasks cause the model to deviate from the original goal. Context blowout – tool results keep expanding token count.

Hierarchical context budgeting is recommended:

L1: Current execution window – system constraints, current goal, recent interactions, essential tool summary
L2: Session‑level working memory – completed steps, key observations, failure reasons, human decisions
L3: Cross‑session persistent memory – user preferences, team rules, domain knowledge, compliance policies

Example ContextCompressor (Python) that prioritises messages and trims them to a token budget:

import tiktoken

class ContextCompressor:
    def __init__(self, max_tokens: int = 128000, model: str = "gpt-4"):
        self.max_tokens = max_tokens
        self.encoder = tiktoken.encoding_for_model(model)
        self.priorities = {"system": 1.0, "conversation": 0.8, "file_content": 0.6, "tool_result": 0.4}

    def compress(self, messages: list[dict]) -> list[dict]:
        total = sum(len(self.encoder.encode(m["content"])) for m in messages)
        if total <= self.max_tokens:
            return messages
        remaining = self.max_tokens
        compressed = []
        for msg in sorted(messages, key=self._priority_key, reverse=True):
            budget = int(remaining * self.priorities.get(msg["kind"], 0.3))
            tokens = len(self.encoder.encode(msg["content"]))
            if tokens <= budget:
                compressed.append(msg)
                remaining -= tokens
            else:
                compressed.append(self._summarize(msg, budget))
                remaining -= budget
        return compressed

    def _priority_key(self, msg: dict) -> int:
        order = {"system": 4, "conversation": 3, "file_content": 2, "tool_result": 1}
        return order.get(msg["kind"], 0)

Lifecycle & Orchestration (L Layer)

Without an explicit lifecycle a process crash leaves the task in an undefined state. A Temporal‑style workflow makes the state explicit, separates retry semantics and inserts human approval points.

type AgentState struct {
    TaskID     string
    StepIndex  int
    Context    []byte
    Results    []string
    Status     string
    TokenUsage int
}

func AgentWorkflow(ctx workflow.Context, task TaskInput) error {
    state := &AgentState{TaskID: task.ID, Status: "running"}
    if err := workflow.ExecuteChildWorkflow(ctx, SandboxPrepare, task.SandboxSpec).Get(ctx, nil); err != nil {
        return fmt.Errorf("sandbox init failed: %w", err)
    }
    for state.StepIndex < len(task.Steps) {
        step := task.Steps[state.StepIndex]
        var result ActivityResult
        err := workflow.ExecuteActivity(ctx, ExecuteStep, state, step).Get(ctx, &result)
        if err != nil {
            if IsRetryable(err) {
                state.Status = "retrying"
                continue
            }
            state.Status = "awaiting_approval"
            workflow.ExecuteActivity(ctx, NotifyHuman, task.Owner, state).Get(ctx, nil)
            var decision string
            workflow.GetSignalChannel(ctx, "decision").Receive(ctx, &decision)
            if decision != "approve" {
                return fmt.Errorf("step %d rejected by human", state.StepIndex)
            }
            state.Status = "running"
            continue
        }
        state.Results = append(state.Results, result.Summary)
        state.StepIndex++
        state.Context = compressContext(state.Context, result.Observations)
    }
    state.Status = "completed"
    return nil
}

Key benefits:

State stored in a durable object, not hidden in memory.

Explicit retry semantics (e.g., except → retry).

Human approval points replace "please be careful" prompts.

Observability (O Layer)

Beyond latency and error rate, agents need per‑step metrics:

Prompt tokens vs. completion tokens per step.

Tool call latency.

Context size growth.

Whether success was due to model correctness or human fallback.

Span‑based tracing that records both model and tool metrics:

type AgentTracer struct { tracer trace.Tracer }

func (t *AgentTracer) TraceStep(ctx context.Context, step string) func() {
    _, span := t.tracer.Start(ctx, "agent.step."+step,
        trace.WithAttributes(
            attribute.String("agent.id", GetAgentID(ctx)),
            attribute.String("session.id", GetSessionID(ctx)),
            attribute.Int("step.index", GetStepIndex(ctx)),
        ),
    )
    return func() {
        span.SetAttributes(
            attribute.Int("llm.prompt_tokens", GetPromptTokens(ctx)),
            attribute.Int("llm.completion_tokens", GetCompletionTokens(ctx)),
            attribute.Int("tool.calls", GetToolCallCount(ctx)),
            attribute.String("context.size", GetContextSize(ctx)),
        )
        span.End()
    }
}

Signal patterns that indicate where to optimise: tokens/step rising → context management failure. tool_latency rising while model latency stays flat → external dependency bottleneck. retry_count rising on a specific tool → protocol or isolation issue. human_intervention_rate rising → governance thresholds too strict or model not ready.

Verification (V Layer)

Four verification levels:

Exact Match – byte‑for‑byte comparison; suitable for fixed, structured answers.

Functional Test – end‑to‑end pass/fail of the workflow; suitable for code changes or workflow execution.

LLM‑as‑Judge – model evaluates execution quality; useful for open‑ended tasks needing semantic judgement.

Human Review – manual approval or spot‑check; required for high‑risk, low‑tolerance tasks.

Example verifier that asks a model to produce a structured JSON score:

class VerificationResult(BaseModel):
    score: int = Field(ge=1, le=5)
    errors: list[str] = Field(default_factory=list)
    is_correct: bool

class AgentVerifier:
    def __init__(self, client, model: str = "gpt-4o"):
        self.client = client
        self.model = model
    def verify_step(self, task: str, expected: str | None, actual: str, tools_used: list[str]) -> VerificationResult:
        prompt = f"""
Task: {task}
Expected: {expected or "N/A"}
Actual: {actual[:2000]}
Tools: {', '.join(tools_used)}

Check:
1. Final output correctness
2. Tool choice and ordering
3. Hallucination or unsupported claims
4. Unnecessary steps
Return JSON only.
"""
        resp = self.client.responses.parse(model=self.model, input=prompt, text_format=VerificationResult)
        return resp.output_parsed

Governance (G Layer)

Governance defines what the system may do, cannot do and how violations are audited. Four boundaries are covered:

Model boundary – allowed models, token limits, temperature ranges.

Tool boundary – allow‑list, deny‑list, approval requirements.

Runtime boundary – max steps, max duration, max cost, permitted environments.

Audit boundary – which actions are logged, retention period, who can query logs.

Example policy (YAML) for a production coding agent:

apiVersion: governance.agent.io/v1
kind: AgentPolicy
metadata:
  name: production-coding-agent
spec:
  model:
    allowed: ["gpt-4o", "claude-3.5-sonnet"]
    maxTokens: 4096
    temperature: [0, 0.7]
  tools:
    allowList: ["read_file", "write_file", "execute_test", "git_commit"]
    denyList: ["delete_resource", "network_request"]
    toolApproval:
      - tool: "deploy_service"
        requireHuman: true
  runtime:
    maxSteps: 50
    maxDuration: "30m"
    maxCost: "$0.50"
    allowedEnvs: ["staging", "dev"]
  audit:
    enabled: true
    retention: "90d"

Practical Build Order

Phase 1 – Single‑Agent Harness (no platform)

Single‑process controller.

Sub‑process or container sandbox.

File‑based or SQLite persistence.

Explicit tool definitions.

Essential safeguards: step‑count limit, tool timeout, basic context compression, minimal execution logging.

Phase 2 – Add Workflow and External State

Introduce L layer – explicit state machine or workflow engine.

Externalise working memory (C layer).

Step‑level tracing (O layer).

Phase 3 – Platform‑Level Governance

Implement G layer – unified policy, approval and audit.

Add V layer – automated verification and regression testing.

Centralised observability (O layer).

Decision Matrix (When to Add Complexity)

If you have a single model, low concurrency and very short tasks → thin harness + few tools.

If tasks are multi‑step, modify files or call external systems → add E/T/C/L layers first.

If tool count grows and failure reasons are opaque → add O layer for observability.

If correctness is hard to verify manually → add V layer.

If multiple teams share tools, tenants are many and operations are high‑risk → add G layer.

Pre‑Launch Checklist

Define max steps, max duration and max cost per task type.

Classify errors into retryable, non‑retryable and human‑intervention.

Set tool timeout, concurrency limits and response size caps.

Establish explicit context budgets instead of keeping full history.

Expose per‑step token usage, tool latency and context growth.

Require approval for high‑risk tools and enforce environment whitelists.

Ensure state can be recovered after a process restart.

Provide basic regression tests beyond ad‑hoc manual checks.

Conclusion

The dominant complexity of production agents shifts from "generating text" to "managing execution" once tools, state and side‑effects are involved. The ETCLOVG seven‑layer framework is not decorative architecture; it is a practical checklist for building reliable, observable, verifiable and governed agent systems.

Models answer questions; Harnesses make agents reliably do work.