Theseus Lab Opens Three Benchmarks for Self-Evolving AI Agents

Anthropic engineers recently noted that 95% of engineers are running self-improvement loops, yet frontier models still fall short of reliable end-to-end project delivery. Theseus Lab presents a three-layer self-evolution roadmap and open-sources three benchmarks to measure progress.

Three-Layer Self-Evolution Vision

1.1 Autonomous Exploration in Open Environments. Models must operate in real, messy business and creative settings, not just static test sets. In workspace experiments, raw GLM-5.3-Flash scored 34.3 on cross-source evidence integration and constrained exploration, while a context-rich version reached 100. Scores across GLM, MiMo, DeepSeek, and HY4 rose ~22 points from August to November, yet over-behavior suppression and constrained decision-making improved only marginally.

1.2 Continuous Internalization of Practice. Borrowing from Anthropic's "sleep" concept, agents should consolidate logs after each task into reusable skills. In a 12-month personal workflow test, this approach led retrieval-augmented and skill-memory benchmarks.

1.3 From First-Order to Second-Order Evolution. Instead of fixing outputs under a fixed method, the improver itself evolves — changing experiment order, judgment criteria, or strategy. In trials, self-iterating improvers yielded a 12-percentage-point gain over static improvement methods.

Open-Source Benchmarks

DeliverBench: End-to-End Project Delivery

Tests seven agent systems (Codex, Claude Code, etc.) on 11 complete professional projects requiring multi-step coordination, incoming materials, and verifiable deliverables. Key results:

Highest average final delivery score: 65.0/100

At a 90-point sub-task pass threshold, best pass rate: 50.0%

Case: Qwen3.8Max misclassified all company loans as a single loan's outstanding balance; the error propagated through the dependency chain into the final deliverable.

DeliverBench exposes uneven capabilities: models handle well-defined subtasks but falter on sustained project ownership — impact scoping, evidence-conclusion traceability, and early-error propagation.

SelfImprove-Bench: Transferability of Self-Improvement

Defines five generalization tiers (G0–G4) based on train/eval relationship:

G0: Same instances (retention)

G1: Same game, same difficulty (new seeds)

G2: Same game, higher difficulty

G3: Same task family, different game

G4: Cross-task-family transfer

Uses HCI = (post−pre)/(ceiling−pre) to measure improvement efficiency. Preliminary findings:

Positive gains drop sharply: G1 88.9%, G2 82.4%, G3 55.6%, G4 only 5/positive.

Method rankings reverse across tiers.

87.9% of improvements remain at prompt/memory level; no meta-level improver modifications observed.

Effective evolution revises 30–40% of prior opinions at framework and second-order levels.

AmbiHTMLBench: Visual Document Reconstruction

Given a "visual document" image, can a model rebuild content-accurate, structurally complete, visually faithful HTML? Benchmark comprises 50k documents across four types: institutional PDFs, native web pages, newspapers/magazines, posters. Dual evaluation:

Objective: render in fixed viewport, extract visible content/position, match against reference (text, table, layout).

Subjective: 33 annotators, 7,920 pairwise judgments comparing visual fidelity.

Results: Gemini 3.1 Pro Preview leads objective composite (0.8092) via lower text errors and stronger table fidelity; GPT-5.6-Sol tops subjective ranking and objective layout but lags composite. The split shows text accuracy and visual fidelity diverge — task-specific model selection matters.

All three benchmarks and infrastructure are open-sourced: https://deliverbench.github.io, https://theseus-labs-rsi.github.io/SelfImprove-Bench, https://ambihtmlbench.tech.