From QA to Task Completion: Survey of LLM Agent Systems and Harness Design

Problem Statement

Identical foundational models exhibit large variations in task‑completion rate, stability, cost, and safety when embedded in different agent systems. The variation is attributed to the coupling between the base model and its execution harness , not solely to model size or data.

Agent and Harness Definitions

An agent is a closed‑loop system that performs five operations: perception, state maintenance, reasoning/decision, action, and feedback adaptation. In practice an LLM agent consists of a model layer (language understanding, reasoning, planning, action proposal) and an execution harness (observable environment, context assembly, action execution, state persistence, failure detection, and recovery).

Limits of Model‑Centric Scaling

Resource‑Performance Boundary : Larger models increase inference cost, latency, and deployment complexity. In agents the cost is amplified because the model is invoked repeatedly across long trajectories.

Measurement Boundary : Static benchmarks (e.g., MMLU‑Pro, GPQA Diamond) do not capture long‑term task ability. Evaluation must consider task length, step count, environment interaction, tool use, state persistence, safety constraints, and recovery.

Paradigm Shifts in Agent Engineering

Prompt Engineering : Few‑shot, chain‑of‑thought, self‑consistency, tree‑of‑thought, ReAct – focus on eliciting model capabilities.

Workflows and Context Engineering : Retrieval, memory, tool definition, context compression, skill disclosure, intermediate artifact management – shift from single‑prompt quality to multi‑step information lifecycle.

Harness Engineering : Close the observe‑reason‑act‑feedback loop; add action execution, result observation, verification, error handling, rollback, and recovery.

Agent‑Native Training and Co‑Evolution : Incorporate planning, tool use, verification, and recovery into model parameters while the harness and deployment experience co‑evolve.

Execution Harness Anatomy

Observation Interface : Transforms terminal output, file diffs, screenshots, DOM, API responses, logs, and retrieval snippets into model‑usable observations, exposing only state useful for the next decision.

Context Manager : Determines what information enters the context, when, and in what form (prompts, system instructions, retrieval results, memory selection, compression, summarization, tool descriptions, task state).

Control Loop : Orchestrates observation, reasoning, action, and feedback; handles step scheduling, stop conditions, retries, reflection, delegation, handoff, multi‑agent collaboration, and model routing.

Action Interface : Maps model outputs to executable actions such as function calls, MCP tools, shell or code execution, browser actions, file operations, API calls, and sub‑agent invocations.

State and Artifact Store : Persists dialogue history, plans, scratchpads, checkpoints, logs, execution traces, diffs, memory records, generated files, and task artifacts; failures often stem from incorrect state handling rather than reasoning.

Verification and Governance : Provides testing, assertions, validator models, sandboxes, permission gates, human approval, budget control, safety constraints, rollback, retry, safe termination, and audit trails, moving agents from “obedient execution” to “controllable execution”.

These responsibilities are inter‑dependent: richer observation improves grounding but raises context‑selection cost; stronger action interfaces expand capability but increase security pressure; comprehensive state stores improve continuity but risk stale information and drift.

Task Landscape and Harness Configuration

Tasks impose pressures along three dimensions:

Task duration : Single‑step tasks rely mainly on context management and verification; multi‑step tasks add demands on action interface and state storage; long‑running tasks require cross‑step plan, evidence, and artifact preservation.

Environment type : Code/terminal tasks can leverage file systems, tests, and logs; browser/desktop tasks involve visual state, DOM grounding, and irreversible side effects; knowledge‑centric tasks depend on evidence quality; physical environments introduce real‑time constraints and risks.

Autonomy level : Open‑ended tasks need budget control, stop conditions, rollback, and human escalation.

Configuration should be driven by these pressures rather than by application tags. A browser agent, a coding agent, and a research agent may share the same LLM but require distinct observation, action, state, and verification strategies.

Evaluation and Analysis

Agent evaluation must report success rate, stability, efficiency, latency, safety, and process quality. Surveyed benchmarks include SWE‑bench Verified, Terminal‑Bench 2.0, and WebArena. With a fixed model, different harnesses cause >10 % score variation on Terminal‑Bench due to differences in command interface, state representation, context management, stop conditions, and recovery strategies. WebArena results show that browser‑agent success depends on page‑state exposure, action constraints, and search/retry budgeting.

Recommended runtime metrics: token/API cost, tool‑call count, retry count, P95 latency, recovery behavior, safety violations, and auditability. This shifts evaluation from leaderboard rankings to detailed runtime profiling.

Outlook and Future Directions

Value‑aware Optimization : Optimize for user‑perceived value by balancing success rate, cost, latency, and risk. High‑value, high‑risk tasks may tolerate stronger verification and higher cost; high‑frequency routine tasks need low cost, short trajectories, and strict stop policies.

Learning Verification, Recovery, and Adaptation : Execution traces become training resources; agents should learn to validate intermediate states, diagnose failures, recover locally, and adapt across tasks. Beyond planning and tool use, reliable agents must know when to stop, retry, or hand over to humans.

Generalization and Specification Boundaries : A harness that works on SWE‑bench may not suit WebArena or Terminal‑Bench; an enterprise‑tool agent may not generalize to the open web. Research is needed on model‑harness compatibility—the match between model capabilities and runtime exposure mechanisms.

Paper URL: https://arxiv.org/abs/2606.20683<br/>GitHub collection: https://github.com/ggjy/Awesome-Agent-Engineering

Code example

来源：专知
本文
约4300字
，建议阅读
5
分钟
现代 LLM Agent 不能只被理解为“模型加工具”，而应被理解为“基础模型 + 执行 harness”的耦合系统。