Why Filename-Based Deduplication Fails in Knowledge‑Base Incremental Uploads

When uploading a batch of documents to a vector‑based knowledge base, many assume that running the import program twice will simply skip duplicate chunks, confirming that no extra embeddings are computed. Two simple tests—changing only the filename while keeping content unchanged, and changing only a paragraph while keeping the filename—demonstrate that filename‑based deduplication can either re‑ingest identical content or silently skip genuine updates.

Filename deduplication creates duplicates and omissions

The common implementation checks the database for a record with the same filename; if found, the upload is aborted, otherwise the file is parsed, split into chunks, embedded, and written to the vector store. This works in demos where the same file name is used for both uploads, but filenames are merely display fields and not stable identifiers.

For example, a policy document renamed from 报销制度.pdf to 财务报销制度最新版.pdf has unchanged content, yet the system treats it as a new document and re‑generates embeddings. Conversely, 员工手册.pdf may retain its name while its vacation rules change; the system skips processing, leaving outdated clauses in the knowledge base.

Thus, filename deduplication answers "Is the uploaded file name the same?" while the knowledge base needs to answer "Has this piece of content already been processed?"

Need for stable document and chunk identities

Incremental ingestion must identify the same document, replace changed parts, fill failed tasks, and retire old versions from retrieval. A two‑layer identity is required: a stable document ID and a stable chunk ID.

In practice, a simple method generates a chunk ID from the source and content, checks for its existence before embedding, and skips only truly new chunks. The deduplication must occur before embedding; otherwise, unnecessary embedding computations still happen.

Pure "source+content" hashing is a starting point, but real projects need richer semantics: tenant or knowledge‑base scope, stable document identifier, section or original position, normalized content, and processing version. Over‑normalization can collapse distinct clauses, while under‑normalization can cause excessive ID churn.

Version handling and isolation

When a document version changes (e.g., deadline from "10th" to "5th"), the new paragraph gets a new chunk ID, but the old paragraph remains in the vector store, leading to ambiguous retrieval results. A robust design isolates the new version: parse, split, embed, and write it in a temporary state, verify completeness, then switch the "current version" pointer to the new set, finally retiring the old version from default search.

Deletion of the old version before the new one is fully written can create gaps if embedding or batch writes fail. Keeping the old version for audit while excluding it from default queries avoids such gaps.

Batch processing pitfalls

Knowledge‑base ingestion is usually performed in batches. A successful API response can mask partial failures due to network issues, missing fields, dimension mismatches, or source identification errors. If only the overall batch status is checked, either already‑successful chunks are re‑processed or failed chunks are silently ignored.

More reliable handling stores each chunk's stable ID, processing stage, and error type. On retry, the system reads persisted state and resumes from the appropriate stage, preserving successful chunks and re‑processing failures with the same IDs.

Concurrent uploads of the same content can cause race conditions where both workers see the ID as absent and write duplicate records. Therefore, "check‑then‑write" must be complemented by atomic upserts, unique constraints, or explicit locking, depending on the storage backend.

Model and splitter upgrades

Embedding models evolve; the same text embedded with different model versions yields vectors in different semantic spaces. Without recording the model version, a new task may mistakenly reuse an old vector, mixing incompatible spaces.

Similarly, changing the chunking strategy (e.g., from fixed‑length to heading‑based splitting) alters chunk boundaries even if the raw text is unchanged. Such changes should be treated as index migrations, requiring versioned processing records.

Thus, each chunk should carry metadata about the document version, parsing version, splitter version, and embedding model version to ensure reproducibility and correct retrieval.

Acceptance criteria

Beyond checking a 200 response or total vector count, four test scenarios should be validated:

Exact duplicate upload: no new chunks and no re‑embedding.

Rename only: if the rename is considered the same document, the stable document ID stays unchanged and unchanged content is not re‑vectorized; otherwise, a new source relationship is recorded.

Partial modification: unchanged chunks are reused, modified chunks receive new IDs, and old chunks are removed from default search.

Fault injection: a deliberately malformed record in a batch must not cause the entire batch to be marked successful, and retries must not duplicate already written chunks.

If the project supports concurrent uploads, an additional check ensures that unique constraints and version switches remain correct under parallel execution.

These tests focus on whether repeated execution leads to duplicate writes, whether genuine updates are missed, whether partial failures can be recovered, and whether old versions continue to pollute retrieval results.

Final takeaways

Filename deduplication fails because it answers the wrong question. A reliable knowledge base must identify content identity, track active versions, and recover correctly after failures. Proper chunk IDs, versioned metadata, atomic upserts, and thorough state‑based acceptance testing are essential for robust RAG pipelines.