Optimizing Retrieval and Generation Latency in High‑Concurrency RAG Agents

1. Problem Analysis

In high‑concurrency Agent systems, latency accumulates across many small delays rather than a single bottleneck. A typical request flow is: user query → query rewrite → vector retrieval → re‑ranking → prompt assembly → LLM generation → post‑processing, with each step costing hundreds of milliseconds, leading to several seconds total. Under load, resource contention further worsens latency. The interview question expects a systematic optimization of the two heaviest stages—retrieval and generation—rather than superficial tricks like caching or swapping to a smaller model.

1.1 Retrieval Stage

The retrieval latency stems from three sub‑steps: query preprocessing (rewrite/expansion), the vector search itself, and post‑search re‑ranking. Optimizations differ markedly under load.

ANN index selection and tuning are crucial. IVF indexes cluster vectors into buckets; a small nprobe speeds search but harms recall, while a large nprobe improves recall at the cost of latency. HNSW builds a multi‑layer proximity graph offering the best latency‑recall trade‑off but consumes more memory. For billions of vectors, IVF‑PQ quantization reduces memory and speeds search at the expense of precision.

Partitioned search —splitting the vector store by business dimensions (document type, tenant ID, time range)—narrows the search space and isolates tenant workloads, reducing contention. Milvus’s Partition feature and Qdrant’s Payload Index support this.

Re‑ranking bottleneck : after a coarse top‑100 recall, a Cross‑Encoder re‑ranks candidates, but its O(N) scoring becomes a choke point under concurrency. Two mitigation paths are offered: (1) reduce candidates entering re‑ranking by inserting a lightweight pre‑filter such as ColBERT, whose token‑level vectors can be pre‑computed and whose interaction is an order of magnitude faster than Cross‑Encoder; (2) batch re‑ranking requests so multiple users share a GPU inference batch, improving hardware utilization.

Another often‑overlooked tactic is hybrid retrieval . Pure vector search excels at semantic similarity but misses exact matches; sparse methods like BM25 excel at exact matches. Running both in parallel and fusing results with Reciprocal Rank Fusion improves recall quality while keeping overall latency bounded by the slower of the two streams.

1.2 Generation Stage

Generation latency is more complex because LLM inference is compute‑intensive and autoregressive, making token generation inherently sequential.

KV‑Cache caches the Key and Value matrices of previous tokens to avoid recomputation, a standard feature in inference frameworks. Under high load, per‑request KV‑Cache memory becomes a new bottleneck as GPU memory fills quickly.

vLLM’s PagedAttention solves this by allocating KV‑Cache in fixed‑size pages, eliminating fragmentation and enabling cache sharing for requests that share the same system prompt prefix, dramatically improving memory efficiency and allowing more concurrent requests.

Continuous Batching replaces static batching with an “as‑soon‑as‑finished‑go‑next” policy: when a short request finishes, its GPU slot is immediately reclaimed for a waiting request, preventing long requests from blocking short ones and boosting throughput 2‑5×. vLLM, TGI, and TensorRT‑LLM all support this.

Speculative Decoding uses a fast draft model to guess several tokens, then validates them in parallel with the large model. Correct guesses yield multiple tokens per large‑model forward pass, achieving 2‑3× decoding speedups without degrading output quality.

1.3 Model‑Level Trade‑offs

Beyond framework tweaks, model‑specific optimizations further cut latency.

Model quantization (FP16 → INT8/INT4) halves or quarters model size, speeding inference with minimal (<2%) accuracy loss. Techniques like GPTQ and AWQ are highlighted.

Model routing directs simple queries to a small, fast model and only sends complex queries to a larger, more accurate model, leveraging a lightweight classifier or rule‑based filter. In practice, 60‑70% of requests are simple, so routing can substantially reduce queue pressure.

1.4 System‑Level Global Optimizations

Isolating retrieval or generation is insufficient; a holistic system design is needed.

Pipeline parallelism overlaps retrieval and generation: as soon as the first batch of retrieved results arrives, prompt assembly and LLM generation start (streaming). Subsequent retrieval results can be merged dynamically or used in the next dialogue turn, reducing perceived latency.

Semantic cache stores query embeddings and their answers; new queries are matched against this cache using similarity search. Tools like GPTCache implement this, achieving high hit rates in high‑concurrency scenarios because many users ask similar questions.

Asynchronous pre‑loading and warm‑up anticipate likely next queries, pre‑loading relevant vector partitions and warming KV‑Cache prefixes so that the actual request starts from a “hot” state.

Output length control limits token count via prompt constraints or max_tokens, cutting generation time roughly in half when responses are trimmed from verbose to concise.

2. Reference Answer

For a high‑concurrency Agent, the latency reduction strategy spans three layers:

Retrieval: use HNSW as the base index, apply partitioned search per tenant, pre‑filter candidates with ColBERT before Cross‑Encoder re‑ranking, and fuse vector + BM25 results with RRF.

Generation: adopt vLLM’s PagedAttention, enable Continuous Batching, quantize the model to INT4 with AWQ, and route simple queries to a smaller model.

System: employ semantic caching (e.g., GPTCache), pipeline retrieval‑generation with streaming, and control output length to halve token count.