Who Owns Your AI Memory? The Risks of Closed Agent Harnesses

Agent Harness Definition

An Agent Harness is a framework that sits between a large language model (e.g., GPT‑4, Claude) and the user. It injects context, manages memory, orchestrates tool calls, compresses the context window, and feeds execution results back to the model, turning a simple chat interface into a functional agent capable of project management, code generation, and long‑term interaction.

Why a Harness Remains Necessary

Harrison Chase cites Anthropic’s Claude Code as a concrete counterexample. Claude Code contains 512,000 lines of code dedicated to the Harness layer alone, showing that even the most capable models require extensive surrounding infrastructure.

The model is a general‑purpose brain; operational concerns such as task decomposition, tool selection, context‑window limits, user preferences, error recovery, and memory management belong to the Harness.

Decomposing a user request into sub‑tasks.

Selecting appropriate tools, managing file systems, executing commands.

Compressing and preserving essential information within limited context windows.

Remembering user habits and preferences.

Handling failures, roll‑backs, and recovery.

LangChain’s blog diagram (shown below) maps each desired Agent behavior to a Harness responsibility, illustrating that as Agent capabilities grow, the Harness responsibilities become more complex.

Memory Is Integral to the Harness

"Inserting memory into an Agent Harness is like putting a steering wheel into a car—it’s an integral part of the vehicle, not a detachable component. Managing context and memory is the core capability of an Agent Harness," – Letta CTO Sarah Wooders.

Memory management involves decisions such as:

How AGENTS.md or CLAUDE.md files are loaded into the context.

How skill metadata is presented to the model (system prompt vs. separate system message).

Whether the AI may modify its own system instructions.

What is retained or discarded during context compression.

Whether interaction history is stored, searchable, and how its metadata is exposed.

How the current working directory and file‑system visibility are represented.

Three Levels of Memory Lock‑In Risk

Chase classifies memory‑related risks into three tiers:

🟢 Light risk – Responses API mode: Agent state lives on the provider’s servers. Switching providers is possible, but the interaction history remains under the provider’s control.

🟡 Medium risk – Closed‑source Harness: Products such as Anthropic’s Claude Managed Agents or OpenAI’s Codex hide the internal compression‑summary mechanism, making stored memory opaque to the user.

🔴 Severe risk – Fully API‑driven memory: Anthropic’s Claude Managed Agents lock the entire memory behind an API, preventing export, inspection, or migration. Changing providers results in loss of accumulated memory.

Commercial Motivation for Lock‑In

Locking user memory raises switching costs, effectively tying users to a single vendor. Chase notes that model switching is cheap, but once a memory layer is added, the cost spikes dramatically. An anecdote about a deleted email‑assistant Agent illustrates the pain of losing stored preferences.

Open‑Source Harness Alternatives

Chase highlights several open‑source Harness projects: Claude Code, Deep Agents, OpenCode, Pi, and OpenClaw (built on Pi). These projects aim to return control of memory and infrastructure to developers.

OpenClaw Features

MEMORY.md file: Long‑term memory stored as a readable Markdown file, editable, backup‑able, and portable.

AGENTS.md file: Behavior specifications stored in a file that users can modify; users decide whether the AI may alter its own system prompts.

Local data control: All data resides locally; switching models only requires a new API key, no migration of memory files.

Skill marketplace: Transparent skill modules that clearly indicate what the AI can or cannot do.

LangChain’s Deep Agents project follows a similar open, transparent approach, supporting self‑hosted memory stores (MongoDB, Postgres, Redis) and deployment on any cloud.

Practical Takeaways

Identify where Agent memory is stored; treat it as a critical asset.

Prefer open Harness implementations; if using a closed system, regularly export memory files.

Contribute to open Harness projects to ensure their longevity and feature completeness.

References

Original tweet by Harrison Chase: https://x.com/hwchase17/status/2042978500567609738

LangChain Deep Agents repository: https://github.com/langchain-ai/deepagents

Sarah Wooders tweet on memory: https://x.com/sarahwooders/status/2040121230473457921

OpenClaw documentation: https://docs.openclaw.ai