Productionizing LLM Agent Harness: Architecture, Backend Design, and Optimization

Agent Harness

1.1 What it is

Agent Harness is the infrastructure layer that wraps a language model into a usable system. It is not the model or a prompt; it is a supporting structure that answers questions the model itself cannot, such as where to store intermediate artifacts, what context each agent may see, how to prevent agents from stepping on each other's context windows, how to handle bad outputs or call failures, and how to track and limit costs.

1.2 Five components

A well‑designed harness consists of five concepts, each corresponding to a class of failure mode in production:

Orchestrator: reads the brief, dispatches tasks in order, validates completion, and reports task end.

Subagents: isolated execution contexts, each responsible for a single task and a single output artifact.

Skills: per‑agent knowledge documents that define role, output format, and rules.

Backend: a shared virtual file‑system state layer that agents use to pass work products between steps.

Context Engineering: disciplined control over what each agent sees, when, and in what order.

Understanding why each component exists is more useful than memorising APIs; it makes later design decisions easier to extend.

2. Backend

2.1 Messaging‑based state problems

The most intuitive approach is to pass agent outputs as conversation messages. Orchestrator saves Agent A’s response and forwards it to Agent B. This introduces three stacked problems:

Context bloat: every message consumes tokens on subsequent calls, leading to thousands of tokens carried for the whole pipeline.

Lack of inspection UI: to see Agent A’s output you must parse the orchestrator’s dialogue history.

Coupling: Agent B depends on a precise schema in Agent A’s response; any schema change breaks the downstream agent silently.

2.2 Virtual file system

StateBackend

solves these issues with an in‑memory work‑area scoped to a single run. Agents write files to the work‑area and read them back, avoiding message passing.

A typical five‑agent pipeline pre‑creates a brief.md file as the sole input. Each sub‑agent writes its own output file (insights, drafts, review notes) and adds a one‑line confirmation. Orchestrator only stores the confirmation line, reducing cumulative context by roughly 85 % and enabling easy inspection and decoupling.

2.3 Pre‑populating files and assembling results

Before the orchestrator runs, all required files (skill documents, shared context, brief) are written to the work‑area. The first LLM call then sees a fully defined workspace. After the pipeline finishes, the orchestrator reads the workspace once to assemble the structured result. Best practice: orchestrator never reads workspace contents during execution; it infers progress solely from file existence.

2.4 Inferring progress from the workspace

Because the workspace’s file set reflects pipeline state, agents can infer progress without explicit callbacks—the presence of a file signals completion of the corresponding step.

3. Skills

3.1 What a Skill is

Each agent has a concrete task defined in a Markdown skill file, which is loaded into the agent’s context together with the system prompt. Only the skill relevant to the agent is loaded, following a progressive disclosure principle.

3.2 Tool scope follows the same rule

Tools are extensions of skills and also increase token cost and attack surface. Only give an agent the tools it truly needs; unused tools add token cost, increase behavior surface, and provide no benefit.

4. Sub‑agents and isolated contexts

4.1 Why shared context windows fail

Running all agents in a single conversation thread forces each agent to carry the full history, inflating token usage (15 000–30 000 tokens for a five‑agent pipeline) and degrading quality because irrelevant context interferes with reasoning.

4.2 Isolated sub‑agents

Each sub‑agent runs in its own session with its own system prompt, skill, and explicitly read workspace files. Orchestrator maintains a flat list of agent descriptors and dispatches accordingly. Context size per sub‑agent stays between 2 000–5 000 tokens, keeping the orchestrator lightweight.

5. Context Engineering

5.1 Static before dynamic

Static content (system prompts, skill files, tool definitions, shared docs) must always precede dynamic content (task ID, user input, timestamps). Prompt caching relies on this order: static prefixes are cached at ~10 % of normal token cost; any violation breaks the cache.

5.2 Persistent memory vs. per‑task context

Persistent memory contains invariant information (project conventions, handling guidelines) and is stored in shared documents loaded once. Per‑task context (user README, platform request) is placed in the brief file and is the only dynamic input.

5.3 Thread compression

Long‑running pipelines cause thread length to grow. A middleware compresses early rounds into summary paragraphs once a token threshold is crossed, keeping the most recent N messages intact. Best practice: compress at ~80 % of context capacity, not lower.

6. Orchestrator

6.1 Coordination, not execution

The orchestrator only reads the brief, dispatches tasks, validates completion, and reports end. It never produces content; if it starts generating domain content, a sub‑agent is missing.

6.2 Build‑once, reuse‑forever

Orchestrator construction (loading skills, initializing model client, compiling graph) is expensive. Cache the orchestrator at process level; a worker handling 40 tasks per hour builds the graph once and reuses it.

6.3 Enforcing pipeline invariants in prompts

System prompts for the orchestrator should hard‑code pipeline rules: generate only platforms listed in the brief, always run verification last, pass task ID explicitly, never write draft files directly, and confirm file existence before reporting completion.

7. Caching stack

7.1 Provider prompt caching

Anthropic’s prompt cache stores the KV tensor of stable prompt prefixes; subsequent identical prefixes cost only 10 % of normal token price. This requires strict static‑before‑dynamic ordering and a minimum token threshold of 1 024.

7.2 Redis LLM response cache

Before each API call, the system checks a Redis cache keyed by the serialized message list and model configuration. A cache hit eliminates HTTP latency and token cost. Include model version in the key; upgrade automatically creates a new key.

7.3 Content‑identity cache

Hash the original source material; identical documents share a cache entry regardless of task‑specific parameters. A hit bypasses the LLM entirely (no API call, no tokens). TTL can be long (e.g., seven days) because the content is immutable.

8. Token optimization

8.1 Estimate before execution

Never run a task without estimating token cost. A rough estimator divides character count by four and adds fixed overhead for skills and context files. Record estimates and, after a week of production traffic, derive P50/P95 values to set alert thresholds.

8.2 Validate and truncate at boundaries

Validate input size before queuing. If too large, truncate at logical boundaries (paragraph breaks, section headings) and append a truncation marker so the model knows the document is incomplete.

8.3 Route models by task

Not every agent needs the most powerful model. Use smaller models for simple extraction or formatting, and reserve strong models for cross‑document reasoning or final review. Proper routing can cut total pipeline cost by 40 %–60 % without quality loss.

9. Asynchronous task architecture

9.1 Need for a task queue

Non‑trivial pipelines take 45–120 seconds, exceeding the default 30‑second HTTP timeout. Accept the request, return a task ID, run the pipeline asynchronously, and let the client poll for status. This reduces resource waste.

9.2 Using Celery for LLM workloads

LLM work is I/O‑bound; workers spend 70 %–80 % of time waiting for API responses. Using gevent pools allows many concurrent tasks (e.g., 32 tasks on a 4‑core machine).

9.3 Dual‑storage mode

Redis provides fast, volatile storage for real‑time polling; Postgres offers durable storage for history, billing, and debugging. Write to both on state change; read from Redis first, fall back to Postgres on expiry.

10. Development workflow

10.1 Local model then cloud validation

Develop against any LangChain‑compatible local model (e.g., Ollama) to separate iteration cost from production cost. Local runs verify file paths, task ordering, result assembly, and error handling. Switch to a cloud model only for final quality checks.

10.2 Test the harness, not the model

Unit tests should target harness behavior: workspace initialization, result assembly, token estimation, proper truncation, cache‑key generation, and state‑transition compliance. Model output quality belongs to integration tests run periodically, not on every commit.

11. Observability

11.1 Structured logs

Every log line must be machine‑parseable with consistent fields (task ID, status, step, latency, cache hit). Even without a log aggregation system, this enables grep‑based analysis and P95 latency extraction via a single shell command.

11.2 LLM call tracing

Structured logs give task‑level visibility; a tracing layer records full prompts, responses, token counts, latency split (first token vs. generation), and the full call tree between orchestrator and sub‑agents.

11.3 Cache hit rate as a cost signal

Track cache hit rates; sudden drops indicate prompt changes, model upgrades, or accidental skill modifications. Alert on hit‑rate anomalies because they often precede unexpected cost spikes.

12. Design principles summary

All decisions stem from five principles:

Reduce cumulative context – smaller context = cheaper, faster, more focused agents.

Static before dynamic – prompt caching yields the highest ROI; static content must always precede dynamic content.

Limit knowledge to role scope – narrow skills, tools, and context minimize token waste and distraction.

Clear separation of responsibilities – orchestrator coordinates, sub‑agents execute, backend stores state.

Estimate, validate, and cap token usage before spending – prevents runaway costs.

The infrastructure described is unglamorous but essential for moving an agent from a notebook prototype to a reliable production service.