How Automated Harnesses Are Revolutionizing LLM Agents: Memory and Action Constraints

1. M⋆: Task‑Specific Memory Harness

1.1 Core Problem: Limitations of Fixed Memory Structures

Current LLM agents employ a one‑size‑fits‑all memory design, whether using semantic retrieval for dialogue agents, skill systems for code agents, or structured databases for domain‑specific agents. Such fixed designs cannot be transferred across domains because each task requires a tailored memory layout.

1.2 Method: Executable Program Evolution

M⋆ encodes the memory harness as a Python program composed of three core components:

Schema : defines the data format using Python dataclasses.

Logic : implements read/write operations and can invoke vector databases, SQL engines, or LLMs.

Instruction : provides prompt constants that guide how the agent interacts with the memory.

The system applies reflective code evolution, iterating through evaluation, reflection & mutation, and quality‑check stages.

Validation sampling : assess programs on static and rotated validation sets.

Coder‑agent iteration : LLM analyzes failure cases and generates code patches.

Constraint checking & auto‑repair : compile checks, smoke tests, and runtime constraints (e.g., output length limits).

A population‑based search strategy balances exploration and exploitation via softmax temperature sampling.

1.3 Experimental Results

Across four heterogeneous benchmarks—LoCoMo dialogue, ALFWorld embodied, HealthBench medical, and PRBench legal‑financial—M⋆ achieves the best performance in 7 out of 8 configurations.

Structural diversity : different tasks evolve distinct memory structures (e.g., list + LLM summary for ALFWorld, SQL + ChromaDB hybrid for LoCoMo).

Task specificity : cross‑task transfer degrades performance, confirming that memory designs must be co‑optimized with the target task.

2. AutoHarness: Automated Code‑Level Constraints

2.1 Core Problem: Illegal Actions in LLM Agents

LLMs often propose illegal moves in tightly defined environments; for example, 78 % of Gemini‑2.5‑Flash failures in a Kaggle GameArena chess competition stem from illegal actions.

2.2 Method: Tree Search + Thompson Sampling for Code Synthesis

AutoHarness frames harness generation as a program‑search problem, using Thompson‑sampled tree search to balance exploration of new logic structures with exploitation of promising harnesses.

AutoHarness supports three harness modes:

harness‑as‑action‑filter : generate a set of candidate actions and let the LLM rank them.

harness‑as‑action‑verifier (primary experiment): generate an action, verify legality with code, and retry if illegal.

harness‑as‑policy : implement the full policy in Python; no LLM calls are needed at test time.

Feedback‑driven : the environment returns legality signals and rewards.

Iterative optimization : based on error cases and execution traces, the LLM produces code patches (V4A format).

Compile‑repair loop : automatically fixes syntax errors and runtime constraint violations.

2.3 Experimental Results

Evaluated on 145 TextArena games (excluding free‑form dialogue), AutoHarness reaches a 100 % legal‑action rate after an average of 14.5 tree‑search iterations; 19 of 32 games converge within 10 iterations.

In two‑player games, Gemini‑2.5‑Flash + harness achieves a 56.3 % win rate versus 38.2 % for the baseline Gemini‑2.5‑Pro, demonstrating that a smaller model equipped with a dedicated harness can outperform a larger model.

Smaller model with dedicated harness can beat larger models.

In single‑player games, the harness‑as‑policy mode attains an average reward of 0.870, surpassing GPT‑5.2‑High (0.844) while incurring near‑zero test‑time cost because no LLM calls are required.

Conclusion

Both papers illustrate a clear trend in LLM‑agent research: the focus is shifting from making models intrinsically smarter to designing appropriate harness frameworks that provide task‑specific memory and safe, constraint‑aware action execution.