Spring AI + Milvus: Enterprise RAG Production Engineering – Verifiable Knowledge, Permissions, Capacity & Recovery

The article opens with a clear boundary statement: Spring AI handles model, embedding, and vector-store integration; Milvus handles vector data management and approximate retrieval. Neither guarantees four critical production properties: (1) that a user has permission to see a specific piece of knowledge, (2) when a new release becomes visible on all read paths, (3) whether retrieved evidence actually supports the answer, and (4) how the system converges under overload, revocation, circuit-breaking, or node failure. The correct enterprise RAG goal is to produce traceable answers under given permissions, knowledge versions, model quotas, and SLOs — and to refuse safely and explainably when those preconditions cannot be met.

1. Four Core Production Threads

A starter system often runs everything in one Spring Boot process. It cannot answer:

Capacity : Max inbound, in-flight, queue limits; how to reject when exceeded. Typical anti-pattern: advertising 10k gateway QPS as 10k real-time generations.

Evidence : Which chunks, which version? What if evidence is insufficient? Typical anti-pattern: calling the model to “try anyway” when retrieval returns empty.

Permission : Is filtering applied before any external call? Can caches leak across tenants? Typical anti-pattern: removing unauthorized citations only at the final display step.

Recovery : How to return to a consistent state after import interruption, failed release, or revocation? Typical anti-pattern: treating a write ACK as globally visible.

All subsequent components serve these four threads. Splitting services is not the goal; making them verifiable system properties is.

2. Fixed Version Baseline – No “Example API Salad”

2.1 Baseline as a Deliverable

The article provides an example verified baseline (not a prescription) to illustrate what version records must contain. Upgrades still require dependency convergence, compilation, and regression testing.

JDK : 21.x — must add exact distribution & container image digest at release.

Spring Boot : 3.4.x / 3.5.x (pick a patch) — must add actual parent/BOM version.

Spring AI : 1.0.9 — must add BOM, starter, Advisor module exact versions.

Milvus : Team-locked Server/SDK/Chart combo — must add Server, Chart, operator, object storage/WAL config.

Embedding : Model name, dimension, normalization, preprocessing — must add provider version, tokenizer, batch limits.

Chat/Rerank : Model & prompt version — must add quota, region, data-processing boundary.

Spring AI 1.0.9 officially supports Spring Boot 3.4.x and 3.5.x. Do not mix 1.x examples, current docs, and SNAPSHOT dependencies in the same project. A sample dependencyManagement block shows importing the Spring AI BOM with a locked version property and declaring starters for OpenAI, Milvus vector store, and the RAG advisor module.

2.2 Configuration Layering

Production configuration uses placeholders for secrets and environment-specific values (API keys, model names, Milvus host/port, collection name, embedding dimension). initialize-schema: false is mandatory in production; schema, indexes, and DB permissions are managed by the deployment pipeline. The application runtime account must not have create/drop/alter privileges.

Embedding dimension matching the collection is necessary but not sufficient. If the model, normalization, or preprocessing changes, the vectors belong to a different semantic space even if dimensions match. A new version must be built; old and new vectors must not be mixed in the same collection.

3. Unified Capacity Terminology: Inbound Traffic ≠ Generation Capacity

3.1 Five Distinct Metrics

Inbound QPS : Questions arriving at gateway — must not be confused with model generation throughput.

Retrieval QPS : Actual retrieval operations issued — must not be confused with user request count; query rewriting and hybrid recall amplify this.

Generation Start Rate : Model invocations started per second — must not be confused with completed answers per second.

Full Answer Throughput : Successful completions per second — must not be confused with SSE connections established.

Active Stream Count : Concurrent streaming sessions — must not be confused with average QPS.

Example: 10,000 inbound QPS, 80% exact-answer cache hit, 25% of remaining rejected for insufficient evidence → generation start rate = 10,000 × (1−0.80) × (1−0.25) = 1,500 starts/sec. With 1,800 input tokens + 200 output tokens average, token demand = 1,500 × 2,000 × 60 = 180M tokens/minute. This formula excludes retries, query rewriting, rerank, or cache tokens; its purpose is to expose the gap between “high gateway QPS” and “high real-time generation capacity.”

3.2 Little’s Law for In-Flight Estimation

Stable systems can roughly use L = λ × W (average in-flight = completion rate × average residence time). Example: 120 generations/sec × 6 sec = ~720 in-flight generation calls. If a downstream dependency allows 200 concurrent blocking calls at 5 sec each, its ideal completion rate is ~40/sec. Averages cannot be replaced by P99; connection pools, async models, SSE sessions, and vendor concurrency quotas must be considered.

Queues absorb short bursts only; they are not a capacity solution. At 100 ops/sec capacity with 200 ms extra wait budget, ~20 queued requests already exhaust that budget. 5,000 queue slots often just turn rejections into more expensive, later timeouts.

4. Overall Architecture: Online & Offline Path Isolation

A text diagram shows the online path: User → Gateway (auth, rate limit, request size) → QA Orchestration (authz, deadline, admission) → ACL/Versioned Answer Cache → Constrained Retrieval & Hybrid Recall → Milvus → Evidence/Permission/Injection Policy → Model Gateway (global quota) → Explicit Refusal → SSE (cancellation, backpressure, citations). Offline path: Document Upload → Object Storage → Task Table + Outbox → Independent Workers (parse, chunk, embed) → Validation & Release Catalog.

QA, retrieval, and rerank may initially share one app but must use separate concurrency bulkheads and budgets. Document parsing and batch vectorization should move to independent workers early to avoid starving online embedding, connections, and CPU. Ordinary synchronous QA should not be dumped into Kafka; async is only for import, rebuild, batch evaluation, or explicit long tasks returning 202 + taskId. Async tasks must persist task + outbox first, then return “accepted”; client timeout cannot justify unlimited background execution.

5. Secure Main Path: Permission, Version, Vector Space as Inseparable Context

5.1 Server-Side Construction of Unforgeable Access Scope

Never deserialize tenantId, kbId, roles, or revision directly from request JSON. They must be derived from the authenticated principal, authorization service, and release catalog on the server side. Two records illustrate the contract:

public record AccessScope(String tenantId, String knowledgeBaseId, String aclRevision, Set<String> permittedDocumentIds) {}

public record RetrievalRelease(String knowledgeRevision, String embeddingSpaceId, String collectionAlias, String retrievalConfigVersion) {}

embeddingSpaceId

represents the immutable combination of model, dimension, normalization, and preprocessing. The release catalog must guarantee this space matches the query embedding model; thus retrieval cannot accidentally hit vectors written by another model just because dimensions coincide.

5.2 Filter Conditions Generated Only by Adapter Layer

A ScopedRetriever example builds a filter using Spring AI’s FilterExpressionBuilder with mandatory clauses for tenant_id, kb_id, kb_revision, and embedding_space_id. Spring AI’s Milvus adapter supports portable metadata filters and a native expression escape hatch; therefore MilvusSearchRequest must never be constructed directly by business callers. If native parameters are required, the wrapper must merge authorization and version constraints as non-removable conditions.

Missing tenant context, unresolved release catalog, unavailable authz service, or filter build failure → refuse protected knowledge access. Insufficient recall is never a reason to drop filters and re-query.

5.3 Main Orchestration Must Complete Evidence Gate Before Model Call

Pseudocode shows the sequence: authorize → resolve release → cache key (scope, release, normalized question, conversation digest) → check cache (re-verify permission) → retrieve → re-check permissions → rerank → evidence selection → if insufficient, return REFUSED_NO_EVIDENCE → generate → validate & attach citations.

Spring AI’s QuestionAnswerAdvisor and RetrievalAugmentationAdvisor assist standard RAG flow; the latter instructs the model not to answer on empty context, but that is not equivalent to “don’t call the model.” Businesses requiring zero generation cost without evidence must explicitly short-circuit in the application orchestration layer.

6. Schema & Metadata: Make Security Semantics Performant

Every chunk must carry the following fields. High-selectivity filter fields should be explicit scalar fields or verified indexable metadata paths — not hidden inside opaque JSON. tenant_id (Scalar, required) — Tenant filtering, audit kb_id, kb_revision (Scalar, required) — Knowledge ownership & release snapshot embedding_space_id (Scalar, required) — Vector space isolation document_id, document_version (Scalar, required) — Source traceability & versioning chunk_id, chunk_index (Unique ID + integer) — Idempotency, deduplication, positioning source_uri, page, section (Metadata) — Verifiable citations parser_version, splitter_version (Metadata) — Explain content changes content_hash (Metadata) — Content deduplication & verification sensitivity, revocation_epoch (Scalar/metadata) — Data handling & fast revocation

Capacity baselines must include real filter selectivity: e.g., measure recall, P95/P99, topK, nprobe/ef, index and memory when a tenant occupies 50%, 1%, 0.01% of data. Unfiltered ANN benchmarks cannot predict multi-tenant production retrieval.

Chunk ID should be deterministically generated from tenant + kb + documentVersion + splitterVersion + chunkIndex (necessary for idempotency). Still, verify with the locked VectorStore.add /native SDK whether same-ID writes insert, overwrite, or fail; record this in import retry tests.

7. Retrieval Quality: Top-K, Fusion, Thresholds Must Have Clear Responsibilities

7.1 Chunking Respects Document Structure First

FAQs keep Q&A pairs; regulations keep heading hierarchy and effective dates; tables keep column names and row associations; code splits by function or API boundary. Scanned PDFs, cross-page tables, headers/footers need separate parsing acceptance rules. Token counts (300/600/900) and overlap ratios are experimental candidates, not universal optima. Use the target model’s tokenizer; character counts cannot substitute token counts.

7.2 Hybrid Retrieval Responsibility Boundaries

Dense retrieval finds semantically similar expressions; BM25/keyword handles SKUs, error codes, contract numbers, exact terminology. Typical flow: both paths recall limited candidates under identical ACL, version, vector-space conditions; deduplicate by chunk ID; fuse via rank-based methods (e.g., RRF); pass to reranker.

Do not directly add BM25 scores, vector similarity, and cross-encoder scores — they lack a common scale. If weighted fusion is needed, calibrate on labeled data first.

7.3 Thresholds Are Not Confidence Probabilities

A threshold of 0.55 does not mean “55% confident.” It is affected by distance metric, vector normalization, model, and adapter score mapping. Thresholds must be calibrated on business positive/negative samples and re-validated after model, index, or preprocessing changes.

Evidence gate checks at minimum: (1) coverage of key conditions/exceptions, (2) source still within validity period, (3) no conflicts among multiple sources, (4) citations come from the current authorized evidence set, (5) final context fits token budget without losing qualifiers due to truncation.

Citation ID allow-lists prevent hallucinated sources but do not prove assertions are supported. High-risk assertions can use rules, NLI/LLM-assisted review, and human sampling; automated reviewers can also err and are not deterministic judges.

8. Prompt Injection: Retrieved Text Is Never Instruction

Knowledge bases may contain “ignore previous instructions,” “print keys,” “call tools.” Relying on “model obeys system prompt” does not reduce this risk to zero. Layered controls:

At import: record source, author, approval status, sensitivity level; tag external, user-generated, unreviewed content with risk markers.

At prompt assembly: place retrieved content in a clearly marked non-executable evidence block, separated from system instructions and tool policies.

Tool calls are re-authorized server-side based on user permissions, parameter policies, and business state; document content cannot extend tool capabilities.

Send only minimal evidence needed for the answer to external models; log exported document versions and approval chains.

High-risk content undergoes human review, sampling detection, and revocable release.

These measures don’t guarantee the model is never attacked, but they move the risk of “document text influencing app permissions” from the prompt layer to an auditable server-side control plane.

9. Knowledge Import & Release: Write Success ≠ Readable

State machine:

PENDING → BUILDING → VALIDATING → FAILED / READY → ACTIVE → RETIRED

.

Recommended flow: 1. Original text to object storage; relational DB writes document version, import task, outbox in same transaction. 2. Worker idempotently consumes task: parse, chunk, embed, bulk write, record stage progress & chunk manifest. 3. Validate new version using production-same read consistency, client, filters, permissions — verify retrievability, counts, hashes, citation positioning. 4. Release catalog atomically switches current pointer from A to B via conditional update or monotonic sequence. 5. New requests bind to B; in-flight requests continue with bound version or abort per safety policy. 6. Wait for cache TTL, in-flight requests, and rollback window to expire before reclaiming A.

Milvus consistency level affects read visibility; a single write ACK or one probe read seeing new data does not prove all read paths are consistent. The release barrier must record the consistency level used, read client, collection/alias, and validation samples.

9.1 Full Snapshots vs. Incremental Versions Cannot Mix

Full Snapshot : One kb_revision contains entire knowledge base. Pros: clear query & rollback. Risks: high storage & build cost.

Incremental Doc Versions : Release manifest points to each doc’s current version. Pros: cheaper writes. Risks: retrieval must ensure old versions don’t crowd candidates.

Cannot write new kb_revision only for changed docs then switch whole-library query to that value — unchanged docs would disappear. Parallel builds require conditional updates or incrementing sequence numbers to prevent slow tasks overwriting newer releases.

9.2 Revocation & Deletion Override Normal Cache Policies

Permission revocation and sensitive deletion cannot wait for TTL: update authoritative ACL/revocation catalog first, block new requests from hitting evidence/answers; then asynchronously clean vectors, cache, originals, previews, evaluation copies, logs per backup retention policy.

If business promises “revocation stops streaming leakage immediately,” system must maintain requestId → session → downstream-call registry: revocation event matches active sessions by principal/document, cancels downstream calls, releases permits, logs time from revocation to actual stop. Bytes already delivered to client cannot be recalled; product/compliance must acknowledge this boundary.

10. Milvus: Choose Topology, Index, Isolation by Constraints

Standalone vs. Distributed differs in topology, HA, and independent scaling — not in whether object storage exists or fixed memory size. Small workloads tolerating recovery downtime can use Standalone; need node fault tolerance, independent scaling, or higher HA → Distributed or managed service.

For 1M vectors × 1536-dim float32: raw vectors = 1,000,000 × 1536 × 4 = 6.14 GB. This is not node memory requirement; index, scalar fields, text, segments, query workspace, loaded replicas, index build peak, rolling upgrade redundancy all add up.

FLAT : Small-sample exact baseline — must measure scan cost vs. correctness.

HNSW : Low latency, high recall — must measure graph index memory, build time, ef.

IVF_FLAT : Controlled cluster search budget — must measure nlist/nprobe vs. filtered recall.

HNSW_PQ (if supported) : Reduce storage/load cost — must measure quantization loss, rerank compensation, end-to-end quality.

Milvus documents HNSW_PQ but it should not be described as an “HNSW switch” present in all versions, nor promise fixed compression ratio or recall loss.

Multi-tenancy can use separate clusters, databases/collections, manual partitions, or partition keys. These are resource/ops boundaries, not automatic application-layer authz isolation; all shared paths must still enforce ACL filters server-side.

11. Concurrency, Deadlines & Global Budgets

Separate bulkheads for online embedding, Milvus, rerank, generation, offline embedding, SSE sessions. Example Bulkhead class uses a Semaphore for in-process admission (not cluster quota or RPM/TPM limiter). Streaming calls must release permits on completion/exception/cancellation, not when returning the Publisher. Ten pods each allowing 100 in-flight generations could create 1,000 concurrent vendor calls; a model gateway or shared quota component must enforce global limits, each pod holding only a conservative local budget.

Each request carries an immutable RequestContext: tenant, ACL snapshot, knowledge version, vector space, traceId, absolute deadline. Do not use ThreadLocal as cross-thread authorization source.

Example budgets: embedding 250 ms, retrieval 200 ms, rerank 350 ms, first-token target 2 s, total generation cap 20 s. Each hop uses min(remaining total budget, stage cap). Connection, first-response, stream idle, and total duration each have separate limits. CompletableFuture.orTimeout may only stop waiting; real HTTP/gRPC calls continue — deadline and cancellation must propagate to underlying clients.

Virtual threads improve scheduling efficiency for many blocking waits but do not increase model quota. MVC + virtual threads and WebFlux both need bounded buffers, connection limits, and cancellation propagation; calling blocking SDKs on WebFlux event loop still causes stalls.

12. Cache: Hit Same Security Semantics Before Hitting Answer

Final answer cache key includes at minimum:

tenant + kb + kb_revision + embedding_space_id + authorization_scope_digest + acl_revision + normalized_question_hash + conversation_digest + prompt_version + retrieval_config_version + reranker_version + generation_model_version + language + output_policy_version

.

New knowledge revision naturally invalidates old cache (no need to scan Redis for prefix deletes). Revocation differs: verify ACL version on read, or use authz cache with explicit propagation bound and revocation catalog.

Request coalescing only allowed within identical knowledge version, permission scope, and output policy. Semantic caching is especially dangerous: “refund allowed?” vs “refund allowed after 7 days?” may be similar but have opposite answers. For orders, prices, regions, contracts, time, personal state — prove exact caching correct before evaluating semantic reuse.

13. SSE: Streaming Success ≠ HTTP 200

Nginx config example:

proxy_buffering off; proxy_cache off; proxy_read_timeout 60s;

. Application uses text/event-stream, sets X-Accel-Buffering: no; must verify every hop (CDN, gateway, ingress, LB, client) for buffering and idle timeouts. Fixing Nginx alone does not guarantee end-to-end streaming.

Define custom event protocol: • meta: requestId, knowledge version, allowed source summary • delta: text increments bounded by size & session buffer • done: completion status, final citations, usage, cacheable flag • error: failure after stream start, explicit partial=true • revoked: revocation or security abort, client stops rendering further deltas

After HTTP response starts, status code cannot express failure. Browser disconnect, server deadline, revocation, downstream error must all invoke same idempotent cancel path: cancel model call, deregister session, release bulkhead permits, stop metering. Partial answers do not enter full-answer cache.

Observe separately: first token, completion, client ack, cancel success, “tokens generated after cancel” waste. If pre-output content moderation is required, choose buffered or segmented review and accept its first-token latency impact.

14. Observability, Evaluation & Load Testing

One request correlates auth, cache, embedding, retrieval, rerank, generation, stream close. Log trace_id, version chain, candidate counts, refusal reasons, stage latencies, tokens, cancel reasons, egress audit. Default: do not log full questions, chunks, outputs; debugging uses controlled sampling, masking, retention.

Capacity : In-flight, queue, reject, connection utilization — Key question: Should we shed load or scale?

Dependencies : P95/P99, timeouts, retries, throttling — Key question: Bottleneck local or external?

Retrieval : Evidence Recall@K, nDCG, zero-result rate — Key question: Denominator: chunks, docs, or required evidence?

Answer : Citation support rate, correct/incorrect refusal rate — Key question: Reliable or just has citations?

Security : Unauthorized hits, revocation propagation latency — Key question: Should we halt release?

Cost : Cost per successful answer, wasted tokens, cache cost — Key question: Within budget?

Data : Import backlog, release duration, build failures — Key question: Can we recover & rollback?

Capacity experiments in three layers: 1. Retrieval layer : fixed data volume, dimension, index, filter selectivity, topK, consistency, cold/hot state; compare against exact baseline recall. 2. Orchestration layer : model stubs inject first-token cadence, throttling, slow responses, errors, cancellations; results only represent orchestration capability. 3. End-to-end : real model quotas; report RPM/TPM, input/output distribution, refusals, cache, cost, completion throughput.

Use open-loop arrival-rate load tests; report timeout and latency measurement definitions; avoid load tools that auto-throttle on slow responses and mask overload.

15. Pre-Release Fault Injection Matrix

Model latency ×5 : In-flight capped, explicit refusal, no unbounded queueing — Evidence: permits, connections, rejections, tokens.

Model sustained throttling : Retry count & total budget bounded — Evidence: actual attempts & cost.

Milvus timeout : No ACL relaxation, no no-evidence fallback — Evidence: filter & egress audit.

Worker crash after write : Replay produces no logical duplicates, no premature release — Evidence: chunk manifest, task state.

Late completion of old task : Cannot overwrite higher release sequence — Evidence: conditional update records.

Redis failure : Fallback stays within local/global budgets — Evidence: downstream concurrency & inbound rejection.

Cross-tenant forged request : Server rejects, no chunks sent to model — Evidence: retrieval filter & model audit.

Read cache after revocation : Unreadable after promised bound — Evidence: ACL version, event timeline.

Client disconnect : Downstream cancel, permits & buffers released — Evidence: active streams, cancel ack, wasted tokens.

Compute node failure : Recovery within defined SLO — Evidence: failure duration, data & errors.

Backup restore : Originals, release catalog, vectors, citations aligned — Evidence: RPO/RTO & sampling results.

Security negative test “no unauthorized access found during testing” cannot prove future inputs safe, but it is the minimum release gate. On failure, rollback release pointer, retrieval config, model adapter; permission revocation and sensitive deletion must never be restored by content rollback.

16. Minimum Production Loop: First Week Deliverables

Don’t start with hybrid retrieval, semantic cache, or multi-cluster. Complete this minimal loop first:

Identity & Permission : Server-side AccessScope, deny policy, audit — Acceptance: tampering tenant/kb params emits zero chunks.

Knowledge Version : Originals, doc versions, chunk manifest, release catalog — Acceptance: single answer traceable to page/section & knowledge revision.

Retrieval : Vector space ID, forced filter adapter — Acceptance: vectors from different models cannot be mixed in retrieval.

Evidence : No-evidence short-circuit, structured citation validation — Acceptance: zero generation calls when evidence insufficient.

Governance : Full-chain deadline, bulkheads, cancellation — Acceptance: slow model cannot exhaust online resources.

Streaming : SSE event protocol, disconnect & revocation cancel — Acceptance: permits eventually returned, no partial answer cached.

Validation : Small labeled set & fault matrix — Acceptance: quality, security, latency, cost all have baselines.

After loop closure, incrementally invest in structured chunking, hybrid retrieval, rerank, global quota, HA, and limited semantic caching. Each step requires written “benefit hypothesis, cost, failure modes, exit criteria.”

Conclusion

Spring AI lowers the cost of integrating models and vector stores; Milvus provides vector retrieval capability. The hard part of enterprise RAG is keeping evidence, permissions, versions, vector spaces, budgets, and recovery consistent on every real request.

Trustworthy performance claims should state: given knowledge scale, tenant distribution, ACL selectivity, model quota, index config, and quality threshold, the system stably completes X requests; how it rejects beyond capacity; how fast it recovers after revocation/failure. Only after load testing, fault injection, and versioned evidence validation do throughput numbers have business meaning.

References & Verification Boundaries

All links are official documentation. URLs change with product updates; production implementation must archive chosen version URLs, lockfiles, Chart values, and test reports together.

Spring AI 1.0.9 Getting Started: Boot compatibility & BOM

Spring AI 1.0.9 Milvus Vector Store: config, filters, native expressions

Spring AI 1.0.9 RAG: Advisor, RetrievalAugmentationAdvisor, empty context

Milvus Consistency

Milvus Deployment Options

Milvus HNSW_PQ

Milvus Implement Multi-tenancy

Oracle Java 21 Virtual Threads

Nginx ngx_http_proxy_module

The article does not execute model calls, Milvus cluster deployment, or performance benchmarks in a specific production environment; example parameters are not go-live metrics. Complete compilation, integration, load testing, and security acceptance with locked versions, real permission models, and real data distributions.