Can Cache‑Augmented Generation Outperform RAG? A Deep Dive into LLM Efficiency

Overview

Cache‑augmented generation (CAG) preloads a collection of documents into the context window of a large language model (LLM) and stores the model’s intermediate key‑value (KV) attention cache. During inference the model reuses the KV cache to answer queries, eliminating the need for a real‑time retrieval step and therefore reducing response latency.

Comparison with Retrieval‑Augmented Generation (RAG)

Definition : CAG = pre‑load documents + pre‑computed KV cache; RAG = dynamic retrieval of documents at query time.

Process : CAG – (1) preload documents, (2) run a forward pass to capture KV states, (3) store the KV cache, (4) generate answers using the cache, (5) reset cache for new sessions. RAG – (1) receive query, (2) retrieve relevant documents, (3) concatenate query and docs, (4) generate answer.

Latency : CAG removes retrieval delay. Reported latencies on HotPotQA: small‑scale CAG ≈ 0.85 s vs. RAG ≈ 9.25 s; medium‑scale CAG ≈ 1.66 s vs. RAG ≈ 28.8 s.

Quality (BERTScore) : CAG often scores higher. Example: HotPotQA small‑scale CAG 0.776 vs. RAG 0.752; SQuAD small‑scale CAG 0.8265 vs. RAG 0.7516.

System complexity : CAG simplifies architecture (no retrieval service). RAG adds a retrieval component (e.g., BM25, dense retriever) and associated maintenance.

Applicability : CAG is suited for static, moderate‑size knowledge bases that fit within the model’s context window (e.g., Llama 3.1 8B with 128k tokens). RAG is preferable for open‑domain or frequently updated corpora.

Implementation details : CAG relies on pre‑computed KV caches; reference implementation is available at https://github.com/hhhuang/CAG. RAG commonly uses frameworks such as LlamaIndex with BM25 or OpenAI indexing.

Advantages : Faster inference, unified context, lower operational overhead.

Limitations : Bounded by the LLM’s context window; not suitable for very large or rapidly changing datasets.

Typical Use Cases

Static knowledge repositories (e.g., hospital procedure manuals).

High‑frequency FAQ‑style queries where the same documents are repeatedly needed.

Latency‑sensitive applications such as real‑time customer support.

Implementation Steps (Concrete Tutorial)

Preparation : Obtain a HuggingFace access token and create a plain‑text file document.txt containing the knowledge base.

Environment setup : Install required Python packages: pip install torch transformers dynamiccache Generation function : Define a function that performs greedy decoding on a model (e.g., Mistral‑7B). Example:

def generate(model, input_ids, kv_cache):
    outputs = model.generate(input_ids, max_new_tokens=256, do_sample=False, kv_cache=kv_cache)
    return tokenizer.decode(outputs[0], skip_special_tokens=True)

KV‑cache utilities : Implement two helpers:

def get_kv_cache(model, prompt):
    inputs = tokenizer(prompt, return_tensors="pt")
    with torch.no_grad():
        model(**inputs)  # forward pass populates internal KV cache
    return model.get_kv_cache()

def clean_up(model):
    model.reset_kv_cache()

Load the LLM : Load the Mistral‑7B model from HuggingFace, using the token and selecting CUDA if available:

from transformers import AutoModelForCausalLM, AutoTokenizer
model_name = "mistralai/Mistral-7B-Instruct-v0.1"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    token=HF_TOKEN,
    device_map="auto",
    torch_dtype=torch.float16,
)

Create the knowledge cache : Read document.txt, build a system prompt (e.g., "You are an assistant with the following knowledge: ..."), and generate the KV cache:

with open("document.txt", "r", encoding="utf-8") as f:
    knowledge = f.read()
system_prompt = f"You are an assistant. Knowledge:
{knowledge}
Answer the following question:"
kv_cache = get_kv_cache(model, system_prompt)

Answer a user query : Reset the cache, tokenize the query, concatenate it to the cached context, and call generate:

clean_up(model)
query = "How should a patient be prepared for MRI?"
input_ids = tokenizer(query, return_tensors="pt").input_ids
answer = generate(model, input_ids, kv_cache)
print(answer)

Challenges and Open Issues

Context‑window limitation: the total size of preloaded documents plus the query must fit within the model’s maximum token length.

Memory overhead: storing KV caches for large models can consume significant GPU memory.

Cache invalidation: when the underlying knowledge changes, the KV cache must be recomputed, which adds a preprocessing cost.

Security considerations: preloaded static data may contain sensitive information that remains in memory throughout inference.

Key References

Don’t Do RAG: When Cache Augmented Generation is All You Need for Knowledge Tasks – https://arxiv.org/abs/2412.15605 Cache‑Augmented Generation (CAG) in LLMs: A Step‑by‑Step Tutorial – https://medium.com/@ronantech/article-slug Cache‑Augmented Generation: A Simple, Efficient Alternative to RAG – https://github.com/hhhuang/CAG Exploring the Shift from Traditional RAG to Cache Augmented Generation (CAG) –

https://medium.com/@ajayverma23/exploring-the-shift-from-traditional-rag-to-cache-augmented-generation-cag-a672942ab420

Illustration