How GAG Enables Zero‑Retrieval, Single‑Token Private Knowledge Injection in LLMs

Overview

In high‑value private domains such as biomedicine, materials, and finance, large language models (LLMs) struggle with proprietary, rapidly evolving, and sparsely available public corpora. Two mainstream approaches—continuous fine‑tuning and Retrieval‑Augmented Generation (RAG)—each have critical drawbacks.

Continuous fine‑tuning : expensive iterations, catastrophic forgetting, and degradation of general capabilities.

RAG : evidence fragmentation, retrieval drift, long‑text pressure, and uncontrolled prompt length.

A joint team from the Chinese Academy of Sciences and 360AI treats private knowledge as a new modality and proposes a third‑generation solution called GAG (Generation‑Augmented Generation), which follows an "alignment‑fusion" strategy from multimodal LLMs.

Core Idea

Compress expert knowledge into a single continuous vector and directly insert it into the frozen LLM’s embedding space, achieving:

Zero retrieval : no inverted index or vector store.

Zero base‑model update : the Qwen3‑8B weights remain frozen.

Constant budget : inference cost increases by only one token regardless of private corpus size.

Modular : new domains require only adding a small expert module and a projector, leaving existing components untouched.

System Architecture

The architecture consists of a general router, a frozen LLM base, domain‑specific routers, and lightweight projectors (Πᵢ). The inference flow is:

User query → General router → frozen LLM base
          ↓
          Domain router → LLMᵢ → Projector Πᵢ → 1‑Token injection → LLM base

Two‑Stage Training Process

Stage I – Domain Expert Acquisition : a small 1.7 B model is trained on private QA pairs to produce domain‑specific embeddings (LLMᵢ).

Stage II – Projector Alignment : a 2‑layer MLP (Πᵢ) maps the expert embedding to the base model’s embedding space, using the same QA pairs.

During inference, the domain router selects the appropriate expert token, which replaces a reserved slot in the prompt.

Key Technical Details

1. Which hidden state to use?

Layer‑wise ablation shows that the last hidden state of the 4th‑from‑last layer (L₂‑4) provides the best trade‑off between semantic richness and specialization.

2. Routing Mechanism (PPR)

Offline : a frozen encoder clusters historical queries into 32 prototypes per domain using K‑Means.

Online : the incoming query’s similarity to prototypes (cosine distance) determines the selected router.

No training or threshold needed; adding a new domain only requires extending the prototype library, achieving >99.5 % micro‑average accuracy.

Experimental Results

Private Domain QA

Datasets: immune adjuvant (1 135 questions) and catalytic material (646 questions). Metric: BERTScore × 100.

Compared systems:

Base‑Only: 56.12 / 60.01

RAG‑best (with 375 extra tokens): 59.97 / 62.13

GAG (adds only 1 token) : 69.17 / 71.36

GAG improves over RAG by 15.3 % (adjuvant) and 14.9 % (material) while reducing token budget by 375×.

General Capability Preservation

On six public QA benchmarks, GAG’s performance drops by at most ±0.5 %, whereas a naïve expert‑text‑in‑Prompt (EGC) approach loses ~37 % accuracy, highlighting the necessity of reliable routing and representation‑level injection.

Routing Scalability

Expanding from 2 to 6 domains (including aviation, law, mathematics) maintains a micro‑average routing accuracy of 99.7 % with zero re‑training, demonstrating true plug‑and‑play capability.

Typical Case Comparison

Question: “Can AbISCO‑300 enhance T‑cell response?”

RAG : retrieves fragmented evidence about AbISCO‑100, leading to entity mismatch and refusal to answer.

GAG : the expert token encodes the causal chain “adjuvant‑300 → APC activation → CD4⁺/CD8⁺ boost”, providing an accurate mechanistic answer.

Limitations & Outlook

Cross‑domain composition : currently only one domain can be activated per query; future work will explore probabilistic mixing for multi‑domain queries.

Numeric fidelity : a single token struggles with rare numbers or units; a lightweight post‑processing module can be added to correct such values.

GAG elevates private textual evidence to expert representations, injecting knowledge with a single continuous token while keeping the base model frozen and the inference budget constant, offering a governance‑friendly, hot‑swappable paradigm for enterprise‑grade multi‑domain LLMs.

Reference

Generation‑Augmented Generation: A Plug‑and‑Play Framework for Private Knowledge Injection in Large Language Models
https://arxiv.org/pdf/2601.08209