RAG System Operations: Vector Database Selection & Performance Tuning

1. Problem Background

A knowledge base initially with 100k documents had fast retrieval. After six months, data growth and complex permission filtering caused user complaints: "can find old answers but not new documents." Vector database P95 latency increased while RAG answer accuracy dropped. Simply adding machines to the vector database may not solve document chunking, index freshness, or filter condition issues.

This article provides an operational pipeline from data ingestion to online recall: establish evaluation sets, compare storage solutions, measure indexing and query performance, tune HNSW and filtering, monitor updates, backups, and recovery. Example domains, data volumes, and metrics illustrate methods, not real incidents. Different vector database versions, deployment forms, and hardware affect supported features; selection must be validated locally.

2. Breaking Down "RAG Slow" into Stages

A RAG request passes through query rewriting, embedding, vector search, metadata filtering, reranking, context assembly, and generation. User-perceived time-to-first-token (TTFT) is the sum of these stages plus queueing and network. Looking only at average vector database query latency cannot explain generation slowdowns.

Query → query embedding → Retrieval (dense/sparse) → ACL filter → rerank → context truncation → LLM prefill → streaming output

from time import monotonic
trace = {}
start = monotonic()
# Record stage latency and candidate count after each stage
trace["embedding_ms"] = 22.0
trace["vector_search_ms"] = 45.0
trace["rerank_ms"] = 130.0
trace["generation_ttft_ms"] = 700.0

First confirm whether retrieval is slow, recall is poor, or generation is slow. Performance targets should include retrieval P95, answer TTFT P95, recall rate, unauthorized document count, index freshness, and cost. A high-QPS, low-recall system just returns wrong content faster.

3. List Workload Before Selection

Vector database choice depends on data scale, vector dimensions, update frequency, tenant count, filter complexity, hybrid search needs, backup requirements, ops team skills, and cost. Candidates may include pgvector, Qdrant, Milvus; compare with fixed data, hardware, and query distribution.

Key dimensions to validate:

Data: Daily insert/delete volume, dimensions → validate with one day ingestion replay.

Retrieval: top-k, filter selectivity, target recall → validate with labeled query set and ground truth.

Tenants: Can strictly limit document scope → validate with cross-tenant negative cases.

Operations: Replicas, snapshots, recovery time → validate with failure drills.

Cost: Memory, SSD, CPU, network → validate with full-load cost calculation.

If PostgreSQL already exists with small data and strong transaction/permission needs, include pgvector in experiments; for large independent vector search volumes, test dedicated vector databases. These are experimental candidates, not predetermined winners.

4. Build Reproducible Test Data

Retain document ID, tenant, source, version, update time, access level, chunking strategy, and embedding version. Over-large chunks introduce noise and increase context cost; over-fine chunks lose semantic integrity. Every chunk strategy change requires re-indexing and re-evaluation, not just retrieval time checks.

{
  "chunk_id": "doc-81#v3#part-02",
  "tenant_id": "tenant-a",
  "source_id": "doc-81",
  "revision": 3,
  "embedding_version": "embed-v2",
  "updated_at": "2026-09-24T10:00:00Z"
}

Permission tags must be searchable in index data; retrieval must inject trusted tenant filter conditions. Client-submitted tenant_id cannot directly decide retrieval scope, otherwise unauthorized content recall easily occurs.

5. Evaluate Recall, Not Just Latency

Prepare real queries, annotate "documents that should be found." Compute Recall@k, MRR, nDCG, and also test final answer factual correctness and citation validity. Retrieval recall can use exact search as algorithmic upper bound, then compare ANN parameter speed/loss trade-offs; human-annotated ground truth remains the benchmark for business relevance.

relevant = {"doc-1", "doc-7"}
returned = ["doc-4", "doc-1", "doc-8", "doc-7"]
recall_at_4 = len(relevant & set(returned[:4])) / len(relevant)
print(recall_at_4)  # 1.0; does not mean ranking good or answer correct

Evaluation log must retain: query_id, business scenario, expected documents, tenant permissions, returned top-k documents, retrieval latency, rerank latency, final answer and citations.

6. How to Tune HNSW Parameters

HNSW is a graph index. Build parameters: m determines connectivity, ef_construct controls build-time search depth; query parameter ef (or hnsw_ef) controls search scope. Larger search scope usually improves approximate recall but increases CPU and latency; build parameter changes often trigger rebuilds, consuming memory and time. Qdrant documentation explicitly distinguishes these parameters; payload indexes on filter fields also affect filtered retrieval efficiency.

{
  "hnsw_config": {
    "m": 16,
    "ef_construct": 100
  },
  "query_params": {
    "hnsw_ef": 64
  }
}

Above is conceptual, not cross-database universal API. First fix data, filter ratio, top-k, and concurrency; adjust one parameter at a time, plot Recall@10 vs P95 latency curve. If P95 improvement comes from disabling filters, permission requirements aren't met — not a successful tuning.

Option A: Recall@10=0.89, Retrieval P95=50ms
Option B: Recall@10=0.94, Retrieval P95=75ms
Option C: Recall@10=0.95, Retrieval P95=160ms
Above are teaching numbers; choice depends on business recall threshold and latency budget.

7. Filtering and Hybrid Search

Same vector query performs completely differently under "no filter", "tenant filter 1% data", "high-selectivity time range filter". Filter fields need appropriate indexes and must test real combined conditions. Qdrant docs note payload indexes should be created before HNSW graph build; adding indexes later may require graph rebuild to fully utilize filter optimizations.

{
  "must": [
    {"key": "tenant_id", "match": {"value": "tenant-a"}},
    {"key": "visibility", "match": {"value": "public"}}
  ]
}

When exact keyword matching is strong, introduce sparse/BM25 with dense hybrid search, then rerank. Don't directly add raw scores from two paths without scaling; define fusion method and candidate count, validate on labeled set. Online, log each path's candidate count, final hit source, and empty result ratio.

8. Writes and Index Freshness

Document update isn't done at "write API success." Distinguish six timestamps: message arrival, chunking complete, embedding success, vector write, index searchable, old version deletion. If business promises 5-minute visibility, monitor end-to-end latency P95/P99 and backlog queues, not just write TPS.

freshness_lag = first_searchable_at - document_updated_at
index_backlog = pending event count
orphan_chunks = chunks deleted in source but still searchable

SELECT source_id, MAX(revision) AS revision, COUNT(*) AS chunks
FROM ingested_chunks
WHERE tenant_id = 'tenant-a'
GROUP BY source_id;

Use deterministic chunk_id or version key for idempotent writes. For delete/update, ensure old versions disappear from search results, design retry and dead-letter queues. During vector index rebuild, queries hitting mixed versions may cause citation errors.

9. Embedding Model Upgrade

Changing embedding model or dimension means old vectors cannot be directly compared with new ones in same space. Create new collection/index, full backfill, dual-write new documents, compare recall and latency on same queries, then gradually switch traffic; delete old collection after verification and rollback window.

active_index: knowledge-v2
shadow_index: knowledge-v3
write_both: true
shadow_query_percent: 5
switch_condition: "recall, latency, ACL and freshness pass"

Shadow queries on a small fraction of real requests don't return shadow answers to users; keep consistent permission filters to avoid attributing differences to model. After switch, continuously check old version remnants, missed writes, and citation format compatibility.

10. Capacity, Backup, and Recovery

Rough raw vector size = count × dimensions × bytes-per-dimension, but actual memory includes graph index, metadata, segment management, replication, and OS cache. Don't buy memory based only on raw vector size. Peak resource usage higher when high-concurrency filtering, rebuilds, and writes happen simultaneously.

vectors = 10_000_000
scope_bytes = vectors * 1024 * 4
print(round(scope_bytes / (1024**3), 1), "GiB raw vectors")

This calculates ~38.1 GiB for raw float32 vectors, not production capacity recommendation. Backup must cover vector index, metadata, collection config, and corresponding embedding version; periodically restore in isolated environment, measure RPO/RTO, permission consistency, and query correctness.

11. Online Fault Diagnosis Table

Symptom → First Check → Follow-up Actions:

New documents not searchable: Write backlog, version, index status → Compare ingestion vs searchable time.

Unauthorized results: Filter conditions and cache keys → Pause related queries, verify access scope.

P95 spike: CPU, memory, segment merge, rebuild → Rate-limit writes or adjust window.

Recall drop: Embedding version, chunking changes → Replay labeled set with exact search.

Empty results increase: Filter conditions, tenant scope → Check field types and data distribution.

# Check query service health and resources; adjust path per actual product API
curl -fsS http://vector-db.example.internal:6333/healthz

Once unauthorized access occurs, prioritize data exposure control before performance fixes. Performance alerts can tolerate short fluctuations; access isolation failure cannot be compromised for "lower latency."

12. Release, Acceptance, and Rollback

Before changing HNSW or filter indexes, record old config and recall curves; reserve disk, memory, and time for rebuild, control new index traffic. Acceptance checks data freshness, Recall@k, retrieval P95/P99, write backlog, permission negative cases, and backup recovery simultaneously. Rollback must handle post-release new writes catch-up; cannot simply switch back to an already-stale old collection.

release:
  index_version: v3
  replay_since: 2026-09-24T00:00:00Z
  checks: [recall_at_10, p95_search, acl_negative, freshness_p95]
rollback: "switch alias to v2 after catch-up verification"

13. FAQ and Phase Summary

Vector query 20ms, why TTFT 2s? Check embedding, rerank, generation prefill, queueing.

Larger index always better? Larger graph has build, memory, query costs; choose by recall-latency curve.

Vector recall alone enough? Exact terms, IDs, permission filters often need keyword, structured conditions, and rerank.

RAG operations acceptance unit: "Can user query get reliable evidence within permission scope in time?" Database QPS is just one local metric. Put data version, labeled evaluation, index freshness, and failure recovery on same chain; tuning must not turn system into faster wrong-answer generator.

14. Three Real Query Distributions for Load Testing

Uniform random vectors don't represent online users. Real queries split into: popular content hits, long-tail knowledge hits, high-selectivity permission filters. Popular queries get more OS cache benefit; long-tail may trigger cold reads; high-selectivity filters may change ANN candidate search cost. Test each group single-request and concurrent, then mix by online ratio.

query_mix:
  popular: 0.5
  long_tail: 0.3
  selective_filter: 0.2

Ratios are examples. If service promises tenant isolation, also create negative cases: "similar text but belongs to another tenant." This tests ACL and reveals post-filtering insufficient results: global top-10 then application-layer filter may squeeze out target tenant's relevant docs.

Wrong: global top-10 → app filter → only 1 left
Correct: retrieve within trusted tenant scope → rerank per business need

15. Vector Dimensions, Quantization, and Storage Tiering

Higher dimensions increase raw storage, memory bandwidth, index overhead; dimensionality reduction or quantization may save cost but change similarity ranking. If using compressed vectors, first compare Recall@k and downstream answer quality on labeled set, then check memory savings and query latency. For cold-data-on-disk schemes, test both cache-hot and cold states, not just pre-warmed best values.

sizes = [(1_000_000, 768, 4), (1_000_000, 1536, 4)]
for n, d, bytes_per_dim in sizes:
    print(n, d, n * d * bytes_per_dim / (1024**3), "GiB raw")

Formula excludes HNSW graph, payload, replicas. Capacity planning uses peak document count, dual indexes during build, and recovery staging space; disk full during rebuild causes severe production incidents.

16. Jitter from Updates and Segment Merges

Vector databases allocate resources among writes, index building, background optimization. Sustained high writes cause many unoptimized segments, increased query scans, or read-write contention; deletes may leave tombstones needing background compaction. Troubleshooting: watch write queue, segment count, index build progress, background CPU/IO, query P95 simultaneously.

Timeline: 10:00 batch update start → 10:03 segment count rise
10:08 P95 rise → 10:15 optimization complete → P95 recover

This is teaching timeline, not performance causality proof; need same-period node resource and version change records. Short-term: rate-limit batch writes, avoid peak rebuilds; long-term: redesign write batching and resource isolation.

17. Snapshot ≠ Recoverable

Validate snapshots on four items: restorable on new node, collection config consistent, full document count and sampled queries match, ACL and embedding version match. Only checking "backup task shows success" doesn't prove service recoverable. Cross-AZ replication isn't backup; accidental deletes may replicate to all replicas.

restore_drill:
  source_snapshot: example-2026-09-24
  isolated_target: staging-vector
  checks: [document_count, acl_negative, recall_sample, freshness]
  record_rto_minutes: true

If service has incremental writes, clarify which message log replays data after snapshot point, then re-verify read/write after completion. Don't put recovered old index directly into production; it may miss subsequent permission deletion events.

18. Performance Reports to Support Selection

Selection report fixes hardware and data, lists each solution's retrieval P50/P95/P99, write speed, build time, peak memory, single-node failure behavior, recovery time under same Recall@10 and filter conditions. Allow each product reasonable native config but must attach config and ops cost. A solution fastest without filters may not stay fastest with tenant ACL.

| Solution | Recall@10 | P95 Search | Filter P95 | Recovery RTO | Peak Memory |
| --- | ---: | ---: | ---: | ---: | ---: |
| A | TBD | TBD | TBD | TBD | TBD |
| B | TBD | TBD | TBD | TBD | TBD |

Finally link retrieval metrics with answer accuracy, citation precision, and LLM TTFT. Retrieval 20ms faster but needing extra 300ms rerank may degrade business experience.

19. Observability of Document Ingestion Pipeline

RAG ops often focus on online vector search but lack offline ingestion monitoring. After document upload, it may stall at OCR, chunking, embedding rate limits, or vector write. Record queue wait, execution time, failure count, pending retries per stage, linked by source ID.

{
  "source_id": "doc-81",
  "revision": 3,
  "stage": "embedding",
  "queued_at": "2026-09-24T09:00:00Z",
  "started_at": "2026-09-24T09:01:00Z",
  "ended_at": "2026-09-24T09:01:30Z",
  "status": "ok"
}

When online retrieval misses new document, check latest revision's stage first; don't rebuild entire vector database. Retry strategy must handle temporary API rate limits; permanent errors (corrupt files, unparsable formats) go to dead-letter queue and notify data owners.

Upload success ≠ OCR success ≠ embedding complete ≠ index searchable.
Promise to user is visibility time of final stage.

20. Semantic Difference of Top-k Before/After Filtering

If system takes global top-k then filters by tenant, relevant legal docs may be pushed out by other tenants' candidates; worse, intermediate layer already read unauthorized docs. Pass server-generated filter into retriever, then observe filtered candidate count and recall. Security and performance both need testing under same constraints.

# Pseudo-code: tenant_id from auth session
results = vector_db.search(
    query_vector=vector,
    limit=20,
    filter={"tenant_id": authenticated_session.tenant_id}
)

Acceptance case: query text highly similar to tenant B's doc;
Tenant A's results must not contain B's doc or citations,
and A's legal candidates must still meet minimum Recall@k.

21. Chunking Strategy Regression Tests

Paragraph chunking may preserve semantics; fixed-token chunking eases context control; overlap reduces cross-boundary loss but increases index size and duplicate candidates. Test with fixed embedding, retrieval params, query set; only change chunking method; record chunks per doc, duplicate ratio, final answer citations.

experiments:
- chunk_tokens: 256
  overlap_tokens: 32
- chunk_tokens: 512
  overlap_tokens: 64
- chunk_tokens: 1024
  overlap_tokens: 128

Not directly importable vector DB config. For tables, code, contract clauses, verify chunking doesn't separate row headers from content. A seemingly faster short-chunk retrieval may lose critical context for answers.

22. Reranking Cost vs Benefit

Vector index recalls top-50, rerank model takes top-5, may improve relevance but adds model inference. Measure stage-1 Recall@50, post-rerank nDCG@5, rerank P95, final answer accuracy; compare with "no rerank direct top-5". If TTFT budget extremely tight, reduce candidates or rerank only complex queries.

Budget example: query embedding 30ms + retrieval 60ms
+ rerank 120ms + context build 20ms + generation first token 700ms
= client TTFT at least 930ms, plus network and queue.

Don't treat teaching numbers as component default performance. Real pipeline uses distributions, not simple P95 sums; per-request stage latencies needed for accurate reconstruction.

23. Hotspots, Sharding, and Tenant Isolation

Large tenants may own most vectors and query traffic; small tenants many and scattered. Too few shards cause hotspots; too many increase query fan-out and metadata overhead. Choose shared collection with tenant filter vs large-tenant dedicated collections based on data scale, isolation level, and operational cost — validated by measurement.

Check: per-shard QPS, P95, index size, write queue, CPU, disk IO;
Also observe other tenants' search latency when one large tenant publishes batch docs.

Business isolation must be implemented across data model, retrieval filter, result cache, and citation links. Cache key missing tenant ID or ACL version is more severe than mis-tuned HNSW params.

24. Preserve Incremental Writes During Rollback

When switching index alias back to old version, old version may not have received new writes. Use dual-write during migration or retain replayable event log; before rollback, verify offset and catch up new adds, updates, deletes. Especially delete events — missing them lets revoked documents reappear in search results.

rollback_gate:
  old_index_caught_up: true
  delete_events_replayed: true
  acl_negative_passed: true
  query_sample_passed: true

Any index migration needs audit info: old/new index versions, embedding version, chunker version, switch time. Just writing "switch back to v2" without data catch-up plan is not executable rollback.

25. Eight Online Symptom Diagnosis Paths

1. Only one tenant slow. Check that tenant's vector count, filter selectivity, hotspot QPS. Data skew makes average latency normal but single-tenant P99 high; test cost of per-tenant rate limiting or sharding first.

SELECT tenant_id, COUNT(*) AS chunks FROM ingested_chunks
GROUP BY tenant_id ORDER BY chunks DESC LIMIT 20;

2. Write success but not searchable. For same chunk ID, check data pipeline, collection existence, index build progress, filter fields. Don't assume write lost from one empty search; could be wrong embedding version or filter condition.

{
  "source_id": "doc-81",
  "revision": 3,
  "chunk_id": "doc-81#v3#part-02"
}

3. Retrieval faster but answers worse. Verify top-k, ef, rerank params, context truncation. Lower retrieval latency may come from halving candidates; check Recall@k and answer citation set changes.

before, after = .94, .82
print("recall_change", after - before)

4. CPU and disk rise together. Check segment merge, HNSW rebuild, bulk import, snapshot tasks colliding at peak; define priorities and rate limits to avoid dragging search service.

Record: index task start/end, search P95, CPU/IO, segment count.

5. Filter returns too few. Check field type changes (string to number), case sensitivity, null values, and whether filter applied pre- or post-retrieval. Permission conditions cannot be removed for recall.

{"filter": {"tenant_id": "tenant-a", "visibility": "public"}}

6. Results disordered after new embedding migration. Verify query embedding and document embedding use same version, dimension, normalization; mixing old vector spaces makes distances incomparable.

SELECT embedding_version, COUNT(*) FROM ingested_chunks
GROUP BY embedding_version;

7. Slow only on cold start. Test first query vs warmed query, check disk page cache and index load time; capacity planning must consider instant experience when failing over to cold replica.

Report: cold-start P95, warm P95, index load seconds, recovery ready time.

8. Citing old documents. Check old chunk deletion events, cache invalidation, index alias, rerank cache. Set propagation deadline for delete business; if doc contains sensitive revocation, prioritize blocking its continued citation.

source_id: doc-81
old_revision: 2
new_revision: 3
old_chunks_searchable: false

Each symptom must replay with same query ID, save retrieval candidates and final answer. If only reproducible in dev, verify production data scale, filter ratio, cache state weren't simplified away.

26. One Vector Database Migration Drill

Assume current KB uses old embedding, plan new model and new vector DB. Step 1: Freeze evaluation set and permission negative cases — save not only queries and positive docs but also cross-tenant similar docs, recently modified/deleted docs. Otherwise post-migration "top-10 looks similar" cannot judge real improvement.

benchmark_dataset:
  queries: 1000
  include: [popular, long_tail, filtered, recently_updated, deleted]
  metrics: [recall_at_10, ndcg_at_10, p95_search, acl_violations]

1000 is example scale; actual samples must cover enough business types. Step 2: Keep old index read-only, re-chunk and re-vectorize in new index, dual-write or log replayable increments for writes. Compare chunk count, version, embedding dimension, ACL labels per source ID.

SELECT source_id, COUNT(*) AS chunk_count, MAX(revision) AS revision
FROM ingested_chunks WHERE embedding_version = 'embed-v3'
GROUP BY source_id;

Step 3: Shadow traffic — same real queries hit old and new indexes without affecting users, record recall, latency, permission results. Shadow traffic adds load; don't double capacity at peak. When differences found, sample-check raw docs, tokenization, filter semantics — not just tweak ANN params.

{
  "query_id": "q-84",
  "old_top_ids": ["d-1", "d-7"],
  "new_top_ids": ["d-7", "d-1"],
  "acl_passed": true
}

Step 4: Small traffic switch, monitor answer citations, retrieval latency, empty results, freshness. Step 5: Rollback drill — let old index catch up all updates, deletes, ACL changes, then switch alias back. If old index can't catch up in time, rollback plan must pre-state which features degrade; no ad-hoc decisions.

27. Vector DB Failure and Application Degradation

When retrieval service unavailable, app can return "knowledge base temporarily unavailable" or provide limited answers from authorized static materials. Must not pretend retrieval happened and let model hallucinate citations. Set retrieval timeout and bounded retries to avoid downstream slow requests occupying app worker threads.

async def retrieve_with_timeout(query, tenant):
    return await asyncio.wait_for(
        vector_client.search(query, tenant_filter=tenant), timeout=0.3
    )

Above is pseudo-code; production must catch timeout, log error type, clearly inform user. If security requires answers must have citations, refuse generating business facts on retrieval failure.

28. Layered Monitoring Dashboard

Entry layer: RPS, answer TTFT, success rate. Retrieval layer: vector search P95, filter P95, top-k returned count, empty result rate. Data layer: document ingestion lag, failure queue, delete lag. Node layer: CPU, memory, disk, background rebuild, replica status. Group by tenant and query category to avoid global averages drowning large customer issues.

dashboards:
  user: [answer_success, ttft_p95, citation_rate]
  retrieval: [search_p95, empty_result_rate, recall_canary]
  ingestion: [freshness_lag_p95, dead_letter_count]
  node: [cpu, memory, disk, index_build_progress]

Recall hard to measure precisely in real-time on unlabeled queries; use fixed canary query set periodic replay, supplemented by human sampling and user feedback. Canary passing doesn't guarantee all long-tail queries healthy; must watch alongside online distribution shifts.

29. Predict Capacity Boundaries with Data Scale

With 10M chunks, 1024-dim float32 vectors, raw vectors ~38 GiB; but instance must also hold HNSW graph, payload, segment info, OS cache, write buffers, replicas. If planning new index for migration, rebuild window may hold both old and new data. Capacity budget must separate steady state, rebuild peak, post-single-replica-failure load, and recovery space.

Procurement: run 1%, 10%, 100% data volume experiments, observe if memory and index build time scale as expected, then add real filter queries. Cannot linearly extrapolate from 100k vectors P95 to 10M: cache hits, index graph layers, shard count, disk paths all change.

Capacity record: raw vector size, index size, payload index size,
disk free space, rebuild peak memory, recovery temp space needed.

Comparing dedicated vector DB vs relational DB extension, include system maintenance cost. Existing DB with strong consistency data and vector index needing joint updates may benefit from single transaction boundary; but high dimensions, large query volume, sharding needs may favor dedicated service. Decision from controlled experiments and ops capability, not marketing peak QPS.

30. RAG End-to-End Correctness > ANN Recall

High Recall@10 on labeled queries ≠ reliable final answers. Documents may be stale, chunking may drop qualifiers, rerank may push key clauses out of context, model may ignore citations. Need layered acceptance along pipeline: data source latest? Legal docs visible? Retrieval recalls? Rerank retains? Citations support answer?

Common issue: "Where is this year's policy?" Vector retrieval may rank last year's detailed explanation higher than this year's brief announcement. Fix not necessarily increasing HNSW ef; may need document recency weighting, structured filters, or query rewrite redesign. For exact version numbers, keyword search or field filters often beat pure vector similarity.

For permissioned documents, correctness also requires "not using unauthorized materials as evidence." Even if final answer doesn't cite unauthorized doc, if it entered model context, it's a data access boundary violation. Retrieval permissions must enforce before recall; result cache also isolated by permission version.

31. Performance Tuning Must Keep "Explainable Rollback"

Reducing ef, top-k, or removing rerank may lower latency but also reduce recall or answer quality. Before each change, save labeled set, index config, online query samples; after change, rerun in same environment and compare. If business metrics drop, even if DB P95 improves, rollback or apply per-query-category optimization.

If index structure param changes require rebuild, rollback difficulty is ongoing doc updates during migration. Safest model: "New index builds while old serves, new writes and deletes dual-written or logged", switch via alias at release. Rollback not only switches alias but confirms old index caught up post-release writes, especially permission revocation deletes.

Pre-change: fixed workload, old index version, ACL tests, baseline.
During: monitor rebuild resources and index freshness, limit shadow traffic.
Post-change: compare recall, answer quality, P95, permission negatives, recovery capability.

32. Troubleshooting "Irrelevant Answers"

First use query ID to find model's actual retrieval candidates; avoid guessing DB behavior from final text. Check query embedding version vs document vector version consistency; verify filter conditions, top-k, rerank scores; see if context truncated by token limit. Only if retrieval candidates themselves miss key docs, then check index params, chunking, raw data.

If candidates correct but model answers wrong, adjusting vector DB usually ineffective; check prompt, citation requirements, generation model, context assembly. If candidates missing but exact vector search finds them, then reason to increase ANN search depth. If exact search also misses, go back to embedding model, chunking strategy, document update pipeline. This order avoids blaming vector DB for all effectiveness issues.

33. Benchmarking After Vector Index Built

Without business queries, first use few fixed queries to verify API, dimensions, filter fields; quickly switch to de-identified real queries. Test set must include not only "findable" questions but also unanswerable questions, old-version docs, permission-external similar docs, obscure jargon, spelling variants. Unanswerable questions especially important: retriever always returns nearest vectors, but nearest ≠ reliable evidence; app must allow refusal.

Prepare exact search baseline and human relevance labels. Exact search measures ANN approximation loss; human labels measure embedding+document alignment with business semantics. If exact search Recall@10 itself low, increasing HNSW ef won't magically improve semantic matching; check model, chunking, hybrid search, or labeling.

Experiment 1: Exact vector search vs ANN → measure approximation loss.
Experiment 2: ANN vs ANN+filter → measure permission condition impact.
Experiment 3: Dense vs hybrid+rerank → measure business relevance.

Each experiment runs same query set, same document snapshot, same permissions. Frequently updated datasets inconsistent between two system tests make results incomparable.

34. Comparing Common Database Candidates

pgvector appeals for integration with existing PostgreSQL ecosystem, suitable where team has DB backup, permission, query ops experience; but whether it withstands target vector scale and concurrency needs testing with own filter and index configs. Qdrant provides independent vector search with payload filtering; index, segment, persistence strategies need dedicated ops. Milvus and other distributed solutions candidate for larger scale, but cluster components and ops complexity add cost. No static "who's fastest" ranking here.

Keep dimensions, distance function, normalization, top-k, permission filter, data snapshot consistent across three candidates; allow each to use reasonable index params per official docs, then compare P95/P99, build time, memory, failure recovery at same recall target. One DB using exact search vs another using ANN — direct latency comparison isn't same task.

| Candidate | Recall Target | Filter | Search P95 | Build Time | Failure Recovery |
| --- | --- | --- | --- | --- | --- |
| pgvector | Same threshold | Same semantics | Measured | Measured | Measured |
| Qdrant | Same threshold | Same semantics | Measured | Measured | Measured |
| Milvus | Same threshold | Same semantics | Measured | Measured | Measured |

35. Hot Update and Delete Correctness Checks

After new doc added, system must not only show "written" but confirm expected queries can retrieve, filter labels correct, citations openable. After doc modification, old revision chunks must not mix into results; after deletion, all associated chunks, caches, citations must expire per service SLA. For restricted doc permission revocation, propagation latency shorter the better, and monitored separately.

# Pseudo-code: deterministic canary query after update
assert new_revision in [r.revision for r in search(canary_query, tenant)]
assert old_revision not in [r.revision for r in search(canary_query, tenant)]

Cannot use title as sole identifier; same-name docs and duplicate uploads cause citations to wrong version. Use stable source ID + revision, link chunk ID to source doc for deletion and audit.

36. Partition and Replica Failure Semantics

More replicas increase read availability but may have brief sync lag; more shards increase capacity but queries may aggregate across shards. Query consistency, new doc visibility time, node failure degradation vary by product and config; must test per actual deployment version. Don't assume "two replicas" prevents accidental delete — erroneous deletes usually replicate too.

Test by simulating single node stop: record query success rate, P95, data freshness, recovery duration; also simulate recovering node catch-up period search performance. Include replica health and index version in monitoring to avoid lagging replica serving stale answers.

37. Index Metadata for Ops Handover

Handover doc records vector dimension and distance function, embedding model version, chunking rules, HNSW params, filter field indexes, collection alias, snapshot location, estimated rebuild time, rollback method. Without these, on-call sees retrieval latency spike but only knows "vector DB slow" — cannot distinguish rebuild, cold cache, write surge, or model migration.

index_manifest:
  alias: knowledge-active
  embedding_version: embed-v3
  dimensions: 1024
  distance: cosine
  hnsw: {m: 16, ef_construct: 100}
  indexed_filters: [tenant_id, visibility, updated_at]
  snapshot_policy: daily

Values are demo. Update this manifest each release; auto-compare actual collection config with manifest to prevent "doc says v3, production still v2" config drift.

38. One High-Selectivity Filter Tuning Experiment

Business says "new customer profile not searchable", ops sees overall vector DB P95 40ms, assumes DB normal. Actually fault only under one tenant and one permission combo. First extract that tenant's representative queries, confirm correct doc ingested and accessible by tenant; then compare filtered vs unfiltered candidates, record search latency, returned count, recall.

Assume unfiltered relevant doc in top-20, filtered returns zero. Could be metadata field type error, ACL label not synced, retriever used wrong tenant ID, or ANN search depth insufficient under high-selectivity filter. Prioritize permission and field verification, then adjust index or HNSW params; cannot "fix recall" by turning off filter.

{
  "query_id": "q-451",
  "tenant": "tenant-a",
  "expected_source": "doc-81",
  "filter": {"tenant_id": "tenant-a"},
  "returned_ids": [],
  "indexed_revision": 3
}

If filter field needs payload index, confirm creation timing, background rebuild, resource consumption per product docs. Qdrant official docs note: adding payload index later must consider HNSW graph rebuild to fully utilize filter optimization; schedule in low-traffic window, monitor build progress. After launch, compare same query set's Recall@k and P95, not just index task success.

39. Exact Search for Diagnosis, Not Production Replacement

ANN indexes trade approximation for speed; exact search on same vectors establishes "algorithm-findable neighbor set." If exact search finds relevant doc but ANN doesn't, adjusting search depth or index structure may help; if exact search also misses, problem likely in embedding, chunking, missing docs, or business labels. Exact search on large high-dim vectors costly; use for sampling or isolated diagnosis.

Doc in library? → ACL allows? → Exact search finds? → ANN finds?
→ Rerank retains? → Generation cites?

This chain also clarifies team responsibilities: data ingestion, vector index, rerank, generation each need which team to provide evidence, avoiding blame-shifting.

40. Index Param Changes Must Watch Build Cost

m

, ef_construct differ from search ef. First two affect index structure and build cost; changes may trigger full rebuild. Search param mainly affects query-time search scope, can do controlled canary. Increasing search depth usually improves recall and CPU latency, but returns may diminish rapidly. Plot Recall@10, filter P95, P99, CPU per request for each tier; pick lowest cost meeting target.

| Param Tier | Recall@10 | Filter P95 | CPU/Request | Build Time |
| --- | --- | --- | --- | --- |
| Small | TBD | TBD | TBD | TBD |
| Medium | TBD | TBD | TBD | TBD |
| Large | TBD | TBD | TBD | TBD |

If larger index causes node memory pressure, cache hit rate drop may worsen real latency; don't select params based on single-request warm P50. Rebuild must reserve disk and memory to avoid optimization task crushing production cluster.

41. Business Document Version and Index Alias Coordination

KB operators see doc v5, vector DB may still serve v4. Index alias switch solves whole collection version, not per-doc sync across replicas. Write document revision into chunk payload and answer citations; online can audit which version a response used. Permission deletion stricter: if v4 still contains revoked content, even with new v5 ingested, old version must not remain searchable.

During index migration, retain incremental log recording full backfill snapshot point and subsequent event offsets. Acceptance: from post-snapshot point, sample new adds, modifications, deletes for visibility test; before rollback, catch up old index to target offset. If log retention insufficient for rebuild and rollback, adjust retention at design phase, not hope for no new writes at release.

42. Align Ops Metrics with User Perception Causally

If users complain answers slow: check client TTFT, then query embedding, vector retrieval, rerank, generation per-request latency. If users complain irrelevant answers: check retrieval candidates, citation quality, doc versions; vector DB QPS high doesn't prove answer reliability. If users complain new docs invisible: check ingestion queue and index freshness, not just average query latency.

Slow: latency breakdown → saturated resources → queues and hotspots.
Wrong: labeled set → retrieval candidates → rerank → answer citations.
Stale: source revision → ingestion pipeline → searchable time → cache.

Each problem has different evidence; only then discuss vector DB param adjustments. Classifying all complaints as "retrieval performance issue" leads to ineffective scaling and wrong tuning.

43. How to Fill Final Selection Matrix

First write hard constraints: data residency, IAM integration, recovery time objective, doc update visibility SLA. Candidates failing hard constraints excluded from performance ranking. Remaining candidates tested on same data snapshot, filter ratio, recall target: search P95/P99, index build time, rebuild peak resources, node failure recovery. Finally add team maintenance difficulty and long-term cost.

| Dimension | Requirement | pgvector | Qdrant | Milvus |
| --- | --- | --- | --- | --- |
| ACL Filter | Strict | Measured | Measured | Measured |
| Recall@10 | Met | Measured | Measured | Measured |
| Filter P95 | Met | Measured | Measured | Measured |
| Data Freshness | Met | Measured | Measured | Measured |
| Recovery Time | Met | Drilled | Drilled | Drilled |
| Maintenance Cost | Acceptable | Estimated | Estimated | Estimated |

Table doesn't pre-fill winner; real results depend on data and config. If a solution excels without filters but all business queries require tenant filter, that score is supplementary, not decisive.

44. Executable Release Acceptance Sequence

First verify new index doc count and versions; then test recall and citations with labeled query set; next positive/negative cases from test tenants with different permissions; then load test with real query distribution, observe retrieval and answer TTFT; finally drill backup restore, single-node failure, old index rollback. Each gate failure pauses cutover and preserves scene.

release_gates:
- document_revision_consistent
- recall_target_met
- acl_negative_zero
- p95_latency_met
- freshness_lag_met
- backup_restore_passed
- rollback_catchup_passed

This YAML is acceptance design, not product config. After query returns, also verify cited doc URLs openable under current user permissions; otherwise "model answer has citations" is only superficially complete.

45. Final Boundaries Ops Lead Must Confirm

Vector DB healthy ≠ RAG healthy: model answers may be unfaithful to sources. RAG answers correct ≠ permission safe: unauthorized docs may have entered context but not cited. Monitoring splits retrieval performance, data freshness, answer quality, access control; incident handling judges layer by layer. Index param optimization only solves part of retrieval layer.

As doc scale and query distribution grow, periodically replay labeled sets and recovery drills, not just at initial launch. Every embedding version, chunker, index structure, permission field change is a production change needing controlled experiment, canary, and executable rollback.

46. Final Check: Permissions and Recovery Both Hold

In isolated recovery env, replay same questions with two different tenant test identities: each identity only gets authorized docs, citations point to correct revision. Then spot-check post-recovery-point new adds and deletes replayed, compute recovery duration and data gap. Only verifying vector count consistency insufficient to prove recovered RAG service safe.

Recovery drill should also simulate single node loss at peak: can remaining replicas continue ACL-filtered queries, is index freshness affected, does recovering node catch-up drag online P95. All conclusions validated against same labeled set and permission negative cases; not judged by "service process started."

If a KB document has access revoked, first verify retrieval stage no longer returns it, then check old answer caches, summaries, citation links. Index rebuild cannot be excuse for prolonged permission revocation delay; high-sensitivity data needs fast isolation or shielding.

All tuning gains must include comparative results under same permission conditions and query distribution.

Fast retrieval, correct permissions, fresh data — all three indispensable.

References

Qdrant Indexing: https://qdrant.tech/documentation/manage-data/indexing/

Qdrant Optimize Performance: https://qdrant.tech/documentation/operations/optimize/

Milvus Index Explained: https://milvus.io/docs/index-explained.md