How to Efficiently Incrementally Update Knowledge in RAG Applications

Background

In Retrieval‑Augmented Generation (RAG) applications, including GraphRAG, importing and indexing domain knowledge is the foundation for later generation. When the knowledge base changes, the challenge is to update the corresponding vector or graph indexes quickly, cheaply, and with minimal effort.

Typical RAG Indexing Pipeline

Enterprise knowledge‑management systems usually split documents, embed them, and create a vector index. When updates occur, the system must detect which parts of the source documents have changed and apply one of three actions: ignore, add, delete, or update.

Incremental Update Strategies

Two levels of incremental update are commonly used:

Document‑level simple update : Detect new or changed documents, fully parse and embed them, then merge the new vectors into the existing index.

Chunk‑level update : For a changed document, identify which chunks are new, which have been modified, and which remain unchanged, then apply add, delete, or skip actions accordingly.

Chunk‑level Incremental Update Process

Compute a hash fingerprint for each chunk based on its content and metadata.

Persist chunk information (source document ID, hash, timestamp, etc.) using a record manager such as LangChain’s SQLRecordManager or LlamaIndex’s DocumentStore.

During each run, compare current chunk hashes with the stored ones to decide the action:

If the hash exists, skip processing.

If the hash is new, add the chunk.

If a previously stored hash is missing, delete the chunk.

Perform embedding and vector‑index updates for the chunks that require addition or deletion. Note that the vector database must support incremental index modifications.

Implementation with LangChain

LangChain provides an index() API that supports incremental updates via the cleanup parameter. The following example demonstrates the required components:

from langchain.indexes import SQLRecordManager, index
from langchain_openai import OpenAIEmbeddings
from langchain_chroma import Chroma
from langchain_text_splitters import CharacterTextSplitter
from langchain_community.document_loaders import DirectoryLoader

# Embedding model and vector store
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
vector_store = Chroma(collection_name="example_collection", embedding_function=embeddings, persist_directory="./db_chroma")

# Record manager to track chunk hashes
namespace = f"chroma/mydocs"
record_manager = SQLRecordManager(namespace, db_url="sqlite:///record_manager_cache.sql")
record_manager.create_schema()

# Load and split documents
loader = DirectoryLoader("../data", glob='*.txt')
docs = loader.load()
docs = CharacterTextSplitter(separator='
', chunk_size=30, chunk_overlap=2).split_documents(docs)

# Incremental indexing
result = index(
    docs,
    record_manager,
    vector_store,
    cleanup='incremental',
    source_id_key="source",
)
print(result)

The cleanup argument can be set to:

none : No cleanup of existing chunks.

incremental : Remove old versions only when a chunk’s hash changes.

full : Remove old versions whenever a chunk is missing or its hash changes, effectively handling deletions.

Implementation with LlamaIndex

LlamaIndex uses an ingestion pipeline that can perform upserts based on chunk fingerprints. A minimal example:

pipeline = IngestionPipeline(
    transformations=[
        TokenTextSplitter(chunk_size=20, chunk_overlap=0, separator="
"),
        embedded_model,
    ],
    vector_store=vector_store,
    docstore=RedisDocumentStore.from_host_and_port("localhost", 6379, namespace="document_store"),
    docstore_strategy='upserts',
)

docs = SimpleDirectoryReader(input_files=["../data/datafile1.txt"], filename_as_id=True).load_data()
nodes = pipeline.run(documents=docs, show_progress=False)

Both frameworks rely on hash‑based fingerprinting and a persistent record of chunk metadata to achieve efficient incremental updates.

GraphRAG Incremental Update Considerations

GraphRAG extends RAG by incorporating knowledge graphs. Incremental updates are more complex because they must modify both vector embeddings and graph structures (entities, relationships, communities). An open‑source project called nano-GraphRAG demonstrates a lightweight approach: it hashes original documents and chunks, identifies new chunks, and updates the graph by inserting new entities and edges while re‑computing community detection each time (full community recomputation, not incremental).

Open Issues and Future Directions

Chunk‑size based fingerprinting can cause massive hash changes for minor edits, reducing the benefit of incremental updates.

Semantic‑level change detection (e.g., using LLMs) could avoid unnecessary updates but adds cost.

Handling multimodal or complex knowledge structures, and updating non‑vector indexes such as graph indexes, remains an open research problem.

Real‑world deployments may need hybrid strategies: frequent incremental updates for high‑velocity data and batch updates for static knowledge.

Addressing these challenges will make enterprise‑scale RAG systems more cost‑effective and responsive to evolving knowledge.