How Clawdbot Implements a Persistent, Search‑Driven Memory System

1. Context Construction

Before memory, the model sees a request composed of:

[0] System prompt (static + conditional)
[1] Project context (files like AGENTS.md, SOUL.md)
[2] Dialogue history (messages, tool calls, compressed summary)
[3] Current message

The project context includes editable Markdown files injected into each request.

2. Context vs. Memory

Context : what the model sees in a single request; temporary, limited to about 200 K tokens, each token adds compute cost.

Context = system prompt + dialogue history + tool results + attachments

Transient, used only for the current request.

Token‑limited.

Increases inference cost.

Memory : content stored on disk. Memory = MEMORY.md + memory/*.md + session logs Persistent across restarts, days, months.

Effectively unlimited growth.

Low storage cost and searchable via semantic index.

3. Memory Tools

Agents access memory through two tools.

memory_search

Finds relevant content across all memory files.

{
  "name": "memory_search",
  "description": "Semantic search in MEMORY.md + memory/*.md before answering questions about past work, decisions, dates, people, or to‑dos.",
  "parameters": {
    "query": "What was our API decision?",
    "maxResults": 6,
    "minScore": 0.35
  }
}

Example response:

{
  "results": [
    {
      "path": "memory/2026-01-20.md",
      "startLine": 45,
      "endLine": 52,
      "score": 0.87,
      "snippet": "## API discussion
   Decision to use REST instead of GraphQL...",
      "source": "memory"
    }
  ]
}

memory_get

Retrieves specific lines after a search.

{
  "name": "memory_get",
  "description": "Read concrete lines from a memory file after memory_search.",
  "parameters": {
    "path": "memory/2026-01-20.md",
    "from": 45,
    "lines": 15
  }
}

Example response:

{
  "path": "memory/2026-01-20.md",
  "text": "## API discussion

   Discussion of API architecture.

   ### Decision
   Choose REST over GraphQL because: 1) simpler implementation 2) cache‑friendly 3) team familiarity

   ### Endpoints
   - GET /users
   - POST /auth/login
   - GET /projects/:id"
}

4. Writing to Memory

There is no dedicated memory_write tool; agents edit Markdown files directly. The write location is controlled by AGENTS.md, and changes trigger automatic re‑indexing during pre‑compaction and at session end.

5. Memory Storage Structure

The memory principle is "memory = Markdown files in the workspace".

~/clawd/
├── MEMORY.md          # long‑term memory
└── memory/
    ├── 2026-01-26.md  # today’s log
    ├── 2026-01-25.md  # yesterday’s log
    └── ...

Layered Layout

Layer 1 – Daily logs (memory/YYYY‑MM‑DD.md): continuously appended, contain events, decisions, preferences.

Layer 2 – Long‑term memory (MEMORY.md): records important decisions, contacts, written by agents when key events occur.

6. How Agents Read Memory

Each session loads:

1. SOUL.md – agent definition
2. USER.md – user profile
3. memory/YYYY‑MM‑DD.md (today & yesterday) – recent context
4. MEMORY.md – main session persistent context

Agents read these files directly without user authorization.

7. Memory Indexing Pipeline

File Save : e.g., ~/clawd/memory/2026-01-26.md File Watch : Chokidar watches MEMORY.md and memory/**/*.md, batches changes after 1.5 s.

Chunking : ~400‑token chunks with 80‑token overlap.

Vectorization : each chunk is embedded and stored in SQLite (sqlite‑vec).

Storage includes tables for chunks (metadata), chunks_vec (vectors), chunks_fts (full‑text index), and embedding_cache (deduplication).

Hybrid search combines semantic and BM25 scores:

finalScore = 0.7 * vectorScore + 0.3 * textScore

Results with a score below the default minScore (0.35) are filtered out.

8. Multi‑Agent Support

Each agent has an isolated state directory and its own SQLite index:

~/.clawdbot/memory/          # state directory
├── main.sqlite   # vector index for "main" agent
└── work.sqlite   # vector index for "work" agent

~/clawd/          # "main" workspace
├── MEMORY.md
└── memory/
    └── 2026-01-26.md

~/clawd-work/    # "work" workspace
├── MEMORY.md
└── memory/
    └── 2026-01-26.md

Agents cannot access each other’s memory unless absolute paths are allowed, optionally under a strict sandbox.

9. Compaction

When model context limits (e.g., GPT‑5.1 1 M tokens) are exceeded, early dialogue is summarized into compact entries, recent dialogue is kept, and a JSONL session record is written. Compaction can be triggered automatically or manually via the /compact command.

10. Pre‑Compaction Memory Write

Before compaction, Clawdbot checks for important content, writes it to the daily memory/YYYY‑MM‑DD.md file, and does not produce user‑visible output. The behavior is configurable in clawdbot.yaml or clawdbot.json.

11. Pruning

Large tool outputs (e.g., 50 000 characters) are trimmed:

Soft prune : keep recent portions.

Hard prune : replace old content with [Old tool result content cleared].

Cache‑TTL pruning removes stale vector results to avoid repeated high‑cost embeddings.

12. Session Lifecycle

Sessions reset daily by default but can be customized. Hooks include: /new – start a new session.

Extract the last 15 messages from the previous session.

Generate a descriptive filename.

Save under memory/ for searchable persistence.

13. Summary

Transparency : memory is plain Markdown, readable, editable, version‑controlled.

Search‑driven : only relevant content is retrieved, reducing context bloat.

Persistence : important information lives on disk, independent of model context.

Hybrid search : combines vector similarity with keyword matching for balanced relevance.