Hermes Agent + Karpathy's LLM Wiki: Three-Layer Self-Learning Knowledge Base Architecture

Overview

Hermes Agent now bundles Karpathy's LLM-Wiki as a built-in skill. The system converts scattered notes into reusable, structured assets that grow smarter with each use. Running hermes update followed by code/llm-wiki research x/code triggers an automated pipeline that fetches, parses, generates pages, and links them — all without manual knowledge-base construction.

Three-Layer Architecture

Layer 1: raw/ Directory (Immutable Fact Source)

Stores original materials — PDFs, scraped HTML, source code — exactly as retrieved. The rule is strict: read-only, never modify. This layer acts as the "bedrock" evidence; if wiki pages conflict with raw sources, the raw files settle the dispute. Tampering with raw sources is what turns knowledge bases into unreliable "mud" over time.

Layer 2: Wiki Pages (AI Understanding & Rewriting)

The agent reads raw materials and rewrites them into structured pages with headings, lists, and — critically — wikilinks (Obsidian-style [[Page Title]]). These links automatically connect related concepts (e.g., "Transformer" links to "Self-Attention", "Feed-Forward Network", "Positional Encoding"). When new raw content arrives, the agent rescans existing pages and adds missing links, keeping the knowledge graph current.

Layer 3: Schema Config (Consistency Rules)

Defines mandatory YAML frontmatter fields (title, created date, tags, source file path), directory structure, naming conventions, and link syntax. This "grammar" prevents drift: without it, pages would develop inconsistent formats (dates as titles, emojis as tags) and become unreadable. The agent auto-validates pages against schema, rejecting or auto-correcting non-compliant entries.

Three Core Operations

1. Ingestion

Given a command like "ingest this LoRA paper", the agent:

Stores the original PDF in raw/ verbatim.

Writes a summary, creates/updates entity pages ("Low-Rank Decomposition", "Adapter Fine-Tuning", "Parameter Efficiency").

Inserts wikilinks between related pages (e.g., "Parameter-Efficient Fine-Tuning" → "LoRA").

Updates index.md (table of contents) and log.md (operation log with timestamp).

These indexes and logs prevent the knowledge base from becoming a "black hole" and allow traceability months later.

2. Query

When asked "LoRA vs Adapter for LLM fine-tuning?", the agent:

Reads the index to locate relevant wiki pages instantly (no live web search or full paper re-read).

Synthesizes an answer from pre-structured content.

Persists the result as a new page ("LoRA vs Adapter Comparative Analysis") or updates an existing one, adding pros/cons, use cases, and experimental data tables.

This turns every query into a permanent asset. The agent evolves from a "chatbot" (ask once, answer once, forget) into a "knowledge partner" that documents its own answers for future reuse.

3. Check (Self-Audit)

Periodic scans detect three problem types:

Contradictions : e.g., one page says "LoRA params 0.1%", another says "1%" — flagged for resolution via timestamps or source authority.

Orphan pages : unlinked, never-referenced pages — candidates for deletion or merging.

Stale content : e.g., "As of 2023, largest open model is Llama 2" in 2026 — flagged for update.

Without this "librarian", any large library decays into a warehouse of rotting paper.

LLM Wiki vs. Traditional RAG

Traditional RAG resembles re-reading all textbooks before every exam: each query triggers fresh retrieval, ranking, and on-the-fly assembly. Two fatal flaws:

Slow — repeated retrieval/ranking per query.

Unstable — same question yields different answers due to chunk ordering; no memory of prior answers (goldfish memory).

LLM Wiki does the organization once, then stores results as living, interlinked, continuously updated wiki pages. Ten LoRA papers → core "LoRA" page (definition, math, results) + "Parameter-Efficient Fine-Tuning" family tree (LoRA, Adapter, Prefix Tuning). New papers only update affected pages. Quality compounds over time: each ingestion, query, and check corrects errors, fills gaps, strengthens links. Analogy: RAG = buying fresh-cut flowers daily (pretty today, wilted tomorrow); LLM Wiki = tending a plant that grows denser with daily care.

YAML Frontmatter + Wikilinks: Structured Metadata & Graph Structure

YAML Frontmatter

Each page starts with a YAML block (delimited by ---):

---
title: Attention Mechanism
tags: Transformer, Deep Learning, Core Algorithm
source: papers/AttentionIsAllYouNeed.pdf
---

Machines read this for instant filtering: e.g., "all pages tagged Transformer" or "all pages sourced from AttentionIsAllYouNeed.pdf". Without YAML, the agent would need to full-text scan every page to know its topic — like hunting a movie in unlabeled VHS tapes by playing each one.

Wikilinks

Syntax: [[Backpropagation]]. Clicking jumps to that page; if missing, a blank page is created. This transforms linear documents into a network. Agents traverse links automatically during reasoning: asked "Transformer attention vs RNN differences?", the agent hops from "Transformer" → "Attention Mechanism" → "Recurrent Neural Network" and compares content — path planning handled automatically, like a navigation app.

File Structure & Knowledge Graph: Obsidian Integration

The system natively fits Obsidian (local Markdown + wikilinks). Opening the Hermes-generated folder in Obsidian instantly yields a knowledge graph: nodes = pages, edges = wikilinks. Node size = reference count; color = tag groups. Core concepts (highly referenced) and orphans (zero references) are visible at a glance. This visualization is free — no custom frontend development needed.

Obsidian also supports YAML-based queries (e.g., "tag:fine-tuning AND created:>2025") with real-time results — effectively SQL over the knowledge base with simpler syntax. Graph analysis reveals hidden connections: selecting "LoRA" shows direct links to "Low-Rank Decomposition" and "Adapter", and two-hop links to "Prefix Tuning" via "Adapter", exposing shared mathematical foundations.

All these capabilities come from Obsidian out-of-the-box; Hermes Agent only needs to emit compliant YAML and wikilinks. This is "convention over configuration" in action.

Compounding & Self-Learning: From One-Off Retrieval to Permanent Assets

The core value is "compounding" (knowledge begets knowledge). Traditional use: ask → answer → value ends (disposable chopsticks). With Hermes + LLM Wiki:

Query 1: "What is LoRA?" → basic page created.

Query 2: "LoRA vs Adapter?" → cites basic page, adds comparison → higher-level page.

Query 3: "LoRA or Adapter for code generation?" → cites comparison page, adds task-specific experiments → even higher-level page.

Each query stacks on prior work. After a month, hundreds of densely linked pages replace raw-material collages. Answers become traceable: "Are you sure?" → agent pulls original paper excerpt or discussion log. This transparency is impossible with black-box RAG. The result: a continuous accumulation loop for research, code analysis, and knowledge organization — collaborating with a digital researcher that never forgets and grows smarter with every interaction.

Summary

Hermes Agent + LLM-Wiki upgrades AI from a notepad to a long-term knowledge system. The three-layer architecture, write-back mechanism, and structured organization solve the knowledge sedimentation problem while introducing engineering challenges (data protection, format stability, schema evolution). The true value: every use becomes part of future capability.