Packing 775 Saved Articles into a 4 MB Vector Store: A RAG Skill That Beats Karpathy’s Wiki

I stored 775 saved articles in a 4.8 MB vector store with the open‑source skill chao-rag-wiki (https://github.com/smallnest/chao-rag-wiki) and asked it, “What have I collected about loop engineering?” In three seconds it assembled a coherent answer with citations from scattered sources.

1. Karpathy’s “LLM Wiki” concept

Andrej Karpathy proposed that an LLM writes and maintains a wiki while humans read and ask questions. Implementations use two directories: raw/ – read‑only collection of original material (tweets, articles, papers). wiki/ – AI‑generated compiled articles.

When a new file appears in raw/, the AI classifies its topic, merges it with existing articles or creates a new one, and updates a global index file wiki/index.md (one line per article with link, summary, date). Queries first read the index, locate relevant articles, then open them for answer generation.

2. chao‑rag‑wiki: Retrieval‑instead‑of‑compilation

Borrowing the raw/ philosophy, chao‑rag‑wiki replaces the compilation step with Retrieval‑Augmented Generation (RAG). The key distinction is:

karpathy‑llm‑wiki writes first, then reads; chao‑rag‑wiki reads the raw material directly.

Technical stack:

Vector search via turbovec (Rust implementation of Google’s TurboQuant, 16× compression with negligible loss). The 775 articles become 9,089 chunks after deduplication, yielding a 4.8 MB index that fits comfortably in memory.

Embeddings default to Qianfan’s bge-large-zh (1024‑dim). The provider can be swapped (OpenAI, Voyage, local models) by changing an environment variable; API keys are never hard‑coded.

Hybrid retrieval (dense vector + BM25 keyword) with Reciprocal Rank Fusion (RRF) to combine results. Dense search captures semantic similarity (e.g., “self‑iteration” vs. “loop”), while BM25 hits exact identifiers such as rotate_writer.go.

Optional LLM reranking via the --rerank flag for higher precision.

Chunk‑level deduplication using text hashes; identical chunks are merged, preserving source references for duplicate copies.

Pros

Zero compilation latency – new material in raw/ becomes searchable after a single incremental index run.

Full recall – every fragment in the original corpus is searchable, covering long‑tail topics.

Hybrid + rerank handles mixed‑language, proper nouns, and rare queries robustly.

Cons compared with karpathy‑llm‑wiki

No knowledge condensation; answers are assembled from raw fragments rather than a curated article.

Deduplication works only for exact duplicates; near‑duplicate detection would require MinHash/SimHash.

Relies on online embedding services (or a locally deployed model), incurring API cost for the dense path.

In short, karpathy‑llm‑wiki emphasizes “knowledge sedimentation” (structured, readable articles), while chao‑rag‑wiki emphasizes “recall” (comprehensive raw retrieval).

3. Real‑world comparison: two queries

Query 1: “loop engineering”

chao‑rag‑wiki (hybrid + rerank) returned multiple fragments covering the five stages (DISCOVER→PLAN→EXECUTE→VERIFY→ITERATE) and even a contrarian view that most developers don’t need loops yet.

[1] rerank=8.0  PART 2: WHAT LOOP ENGINEERING ACTUALLY IS …
[2] rerank=7.0  工程化的三个层次 …
[3] rerank=7.0  Most Developers Do Not Need Agent Loops Yet …
[4] rerank=7.0  Agent Harness 拆解 …
[5] rerank=6.0  > I don’t talk to an agent anymore, I talk to a loop …

karpathy‑llm‑wiki located a single compiled article in wiki/index.md that summarized loop engineering with a structured table, definitions, and eight citations.

Query 2: “grill skill”

chao‑rag‑wiki retrieved relevant fragments from the original “Mattpocock Skills” and “grill‑me” documents, merging duplicate copies and preserving source markers.

karpathy‑llm‑wiki found no entry because the topic had never been compiled into a wiki article, demonstrating a miss for long‑tail queries.

4. Which tool to use?

Core mechanism : karpathy‑llm‑wiki = AI‑compiled articles; chao‑rag‑wiki = full‑text semantic retrieval.

Query method : karpathy reads index.md then opens an article; chao‑rag runs vector + BM25 hybrid + optional rerank.

Strengths : karpathy excels at popular topics and structured knowledge; chao‑rag excels at exhaustive recall, long‑tail terms, and proper nouns.

Weaknesses : karpathy misses uncompiled topics; chao‑rag returns raw material and only exact‑duplicate deduplication.

Latency : karpathy has slower compilation but fast queries; chao‑rag has fast indexing and slower queries due to online embeddings.

They are complementary: use karpathy‑llm‑wiki for polished summaries, and chao‑rag‑wiki as a safety net to ensure nothing is missed.

5. Wiring it into Obsidian

The end‑to‑end pipeline consists of three stages:

Stage 1 – Web Clipper

Obsidian’s official Web Clipper (Chrome/Edge/Firefox) extracts the article body as clean Markdown and saves it to raw/articles/ with a filename template {{date}} -{{author}}-{{title}} plus front‑matter containing the source URL and capture date.

Stage 2 – Scheduled incremental indexing

A daily cron job runs the build script with the --update flag, which hashes chunk content to process only new or changed files, avoiding unnecessary API calls.

# crontab: every day at 3:07 AM
7 3 * * * cd ~/Obsidian-Vault && \
  ~/.claude/skills/chao-rag-wiki/.venv/bin/python \
  ~/.claude/skills/chao-rag-wiki/scripts/build_rag.py --raw raw --out .rag --update

Stage 3 – On‑demand skill query

From Claude Code you can invoke: /chao-rag-wiki find X – full‑text hybrid search, ideal for long‑tail or obscure terms. /karpathy-llm-wiki I know about X – reads the curated article produced by the compilation pipeline.

This closed loop lets you capture anything you find interesting with a single click, have it automatically indexed, and query it instantly.

6. Installation

The skill is open‑source and can be installed with one command: npx skills add smallnest/chao-rag-wiki After setting an embedding provider key (default Qianfan, optional OpenAI/Voyage), drop material into raw/ and run build to start asking questions.

During writing, the two comparison queries were actually performed by chao‑rag‑wiki itself, illustrating a self‑retrieval loop where the knowledge base can query its own contents.