Production-Grade RAG Architecture: HA, Caching & Cost Control for AI Knowledge Bases

This article, part eleven of the "Building AI Knowledge Base from Scratch" series, transitions from a working RAG prototype to a production-ready system. It contrasts development and production requirements across availability, performance, concurrency, cost, monitoring, and security, then presents a comprehensive architecture and implementation patterns.

Development vs Production Environment

A comparison table highlights key gaps:

Availability : Single-node vs high-availability multi-replica deployment

Performance : "Answers correctly" vs P99 latency < 2 seconds

Concurrency : Single-user testing vs hundreds to thousands of QPS

Cost : Ignored vs strict control

Monitoring : Log inspection vs full-chain tracing

Security : No requirements vs permissions, auditing, rate limiting

Overall Production RAG Architecture

┌─────────────────────────────────────────────────────────────────┐
│                  Production-Grade RAG Architecture               │
├─────────────────────────────────────────────────────────────────┤
│                                                                  │
│  Client → API Gateway → Load Balancer → RAG Service Cluster     │
│                                      │              │           │
│                    ┌───────────────────┼───────────────────┐   │
│                    ▼                   ▼                   ▼   │
│              ┌───────────┐        ┌───────────┐        ┌───────────┐
│              │  RAG 1    │        │  RAG 2    │        │  RAG N    │
│              └─────┬─────┘        └─────┬─────┘        └─────┬─────┘
│                    │                   │                   │
│                    └───────────┬───────────┴───────────────────┘
│                                │
│              ┌─────────────────┼─────────────────┐
│              ▼                 ▼                 ▼
│        ┌─────────┐        ┌─────────┐        ┌─────────┐
│        │  Redis  │        │   MQ    │        │ Vector  │
│        │ Cache/  │        │ Async   │        │ Store   │
│        │ Session │        │ Tasks   │        │(Cluster)│
│        └─────────┘        └─────────┘        └─────────┘
│                                │
│                                ▼
│                        ┌─────────────┐
│                        │  LLM Service│
│                        │(Multi-model │
│                        │ fallback)   │
│                        └─────────────┘
└─────────────────────────────────────────────────────────────────┘

High Availability Design

3.1 Stateless Service Design

RAG services are stateless; session state is stored in Redis. The example shows a StatelessRagService that loads conversation history from Redis, executes the RAG pipeline, then saves updated history back to Redis with a 3600-second TTL.

@Service
public class StatelessRagService {
    @Autowired
    private RagPipeline ragPipeline;
    @Autowired
    private RedisTemplate<String, String> redisTemplate;

    public String chat(String sessionId, String userInput) {
        // 1. Load session history from Redis
        String historyJson = redisTemplate.opsForValue().get("session:" + sessionId);
        List<Message> history = parseHistory(historyJson);

        // 2. Execute RAG pipeline
        String response = ragPipeline.query(userInput, history);

        // 3. Save session history
        history.add(new UserMessage(userInput));
        history.add(new AssistantMessage(response));
        redisTemplate.opsForValue().set(
            "session:" + sessionId,
            toJson(history),
            3600, TimeUnit.SECONDS
        );
        return response;
    }
}

3.2 Multi-Model Fallback

When an LLM call fails, the system automatically tries the next model in a prioritized list (OpenAI, DashScope, Ollama). A ConcurrentHashMap tracks failure counts per model.

@Service
public class ModelFallbackService {
    private final List<ChatModel> models;
    private final Map<String, Integer> failureCount = new ConcurrentHashMap<>();

    public ModelFallbackService(OpenAiChatModel openAi,
                                DashScopeChatModel dashScope,
                                OllamaChatModel ollama) {
        this.models = List.of(openAi, dashScope, ollama);
    }

    public String chat(String prompt) {
        for (ChatModel model : models) {
            try {
                String result = model.call(prompt);
                failureCount.put(model.getClass().getSimpleName(), 0);
                return result;
            } catch (Exception e) {
                String modelName = model.getClass().getSimpleName();
                failureCount.merge(modelName, 1, Integer::sum);
                log.warn("Model {} call failed, trying next: {}", modelName, e.getMessage());
            }
        }
        throw new RuntimeException("All models unavailable");
    }
}

3.3 Vector Database High Availability

Using Milvus as an example, the article shows a cluster configuration with 3 replicas, S3 storage, and etcd coordination. It emphasizes that the vector database is the RAG bottleneck and must support horizontal scaling.

# Milvus cluster configuration
milvus:
  cluster:
    enabled: true
    replicas: 3
  storage:
    type: s3
    endpoint: s3.amazonaws.com
  etcd:
    endpoints: etcd-cluster:2379

Caching Strategies

4.1 Multi-Level Cache (L1 Local + L2 Redis)

A CachedRagService implements two cache layers: a local Caffeine cache (max 1000 entries, 5-minute TTL) and a Redis cache (30-minute TTL). The flow checks L1, then L2, then executes the RAG pipeline and populates both caches.

@Service
public class CachedRagService {
    private final RagPipeline ragPipeline;
    private final RedisTemplate<String, String> redisTemplate;
    private final Cache<String, String> localCache;

    public CachedRagService(RagPipeline ragPipeline,
                            RedisTemplate<String, String> redisTemplate) {
        this.ragPipeline = ragPipeline;
        this.redisTemplate = redisTemplate;
        this.localCache = Caffeine.newBuilder()
            .maximumSize(1000)
            .expireAfterWrite(5, TimeUnit.MINUTES)
            .build();
    }

    public String query(String userQuestion) {
        // L1: Local cache
        String cached = localCache.getIfPresent(userQuestion);
        if (cached != null) return cached;

        // L2: Redis cache
        String redisKey = "rag:cache:" + md5(userQuestion);
        cached = redisTemplate.opsForValue().get(redisKey);
        if (cached != null) {
            localCache.put(userQuestion, cached);
            return cached;
        }

        // L3: Execute RAG
        String result = ragPipeline.query(userQuestion);

        // Write back to caches
        localCache.put(userQuestion, result);
        redisTemplate.opsForValue().set(redisKey, result, 30, TimeUnit.MINUTES);
        return result;
    }
}

4.2 Semantic Cache

To handle semantically equivalent queries with different wording, a vector-based semantic cache stores query-response pairs. It embeds the incoming query, searches the cache vector store with a similarity threshold of 0.95, and returns the cached response if a match is found. The author reports a 25-35% hit rate, directly saving that portion of LLM costs.

@Service
public class SemanticCache {
    private final VectorStore cacheStore;
    private final EmbeddingModel embeddingModel;
    private static final double THRESHOLD = 0.95;

    public String search(String query) {
        float[] queryVector = embeddingModel.embed(query);
        List<Document> results = cacheStore.similaritySearch(
            SearchRequest.builder()
                .query(query)
                .topK(1)
                .similarityThreshold(THRESHOLD)
                .build()
        );
        return results.isEmpty() ? null
            : results.get(0).getMetadata().get("response").toString();
    }

    public void store(String query, String response) {
        Document doc = new Document(query, Map.of("response", response));
        cacheStore.add(List.of(doc));
    }
}

4.3 Retrieval Result Caching

Since documents change infrequently, retrieval results can be cached using Spring's @Cacheable annotation on the hybrid search method.

@Cacheable(value = "retrieval", key = "#query", unless = "#result.isEmpty()")
public List<Document> retrieve(String query, int topK) {
    return hybridRetriever.hybridSearch(query, topK);
}

Asynchronous Processing

5.1 Why Async?

A table maps scenarios to synchronous problems and async solutions:

Document ingestion : Parsing + embedding takes minutes → background async processing

Bulk index rebuild : Blocks online service → offline batch jobs

Long-document retrieval : User waits too long → async result return

5.2 Async Document Ingestion

An AsyncDocumentService uses @Async with a dedicated thread pool (core 5, max 20, queue 100, caller-runs rejection policy) to process documents without blocking the API.

@Service
public class AsyncDocumentService {
    @Autowired
    private DocumentIngestionPipeline pipeline;

    @Async("documentExecutor")
    public CompletableFuture<String> ingestDocument(String filePath) {
        try {
            List<Chunk> chunks = pipeline.process(filePath);
            vectorStore.add(chunks);
            return CompletableFuture.completedFuture("success");
        } catch (Exception e) {
            log.error("Document ingestion failed", e);
            return CompletableFuture.failedFuture(e);
        }
    }
}

@Configuration
@EnableAsync
public class AsyncConfig {
    @Bean("documentExecutor")
    public Executor documentExecutor() {
        return new ThreadPoolExecutor(
            5, 20, 60L, TimeUnit.SECONDS,
            new ArrayBlockingQueue<>(100),
            new ThreadPoolExecutor.CallerRunsPolicy()
        );
    }
}

5.3 Async Query Processing

For slow queries, an endpoint submits a task to a thread pool, stores the result or error in Redis with a 1-hour TTL, and returns a task ID immediately. The client polls for completion.

@PostMapping("/async-query")
public Map<String, Object> asyncQuery(@RequestBody QueryRequest request) {
    String taskId = UUID.randomUUID().toString();
    CompletableFuture.runAsync(() -> {
        try {
            String result = ragService.query(request.getQuestion());
            redisTemplate.opsForValue().set(
                "task:" + taskId, result, 1, TimeUnit.HOURS);
        } catch (Exception e) {
            redisTemplate.opsForValue().set(
                "task:" + taskId, "FAILED: " + e.getMessage(), 1, TimeUnit.HOURS);
        }
    });
    return Map.of("taskId", taskId, "status", "PENDING");
}

Cost Control

6.1 Layered Cost Structure

┌─────────────────────────────────────────────────────────────────┐
│                      RAG Cost Structure                          │
├─────────────────────────────────────────────────────────────────┤
│                                                                  │
│  Embedding Cost (Offline)                                        │
│  ├── One-time during document ingestion                          │
│  └── Billed per document volume                                  │
│                                                                  │
│  Retrieval Cost (Online)                                         │
│  ├── Vector search: near-zero cost                               │
│  └── Reranking: pay-per-use                                      │
│                                                                  │
│  LLM Cost (Online)                                               │
│  ├── Answer generation                                           │
│  └── Accounts for 80%+ of total cost                             │
│                                                                  │
└─────────────────────────────────────────────────────────────────┘

6.2 Token Optimization

A TokenOptimizer builds context by adding documents until a token budget (2000 tokens) is reached, using a simple heuristic: Chinese characters ≈ 1.5 tokens each.

@Service
public class TokenOptimizer {
    private static final int MAX_CONTEXT_TOKENS = 2000;

    public String buildContext(List<Document> documents) {
        StringBuilder context = new StringBuilder();
        int currentTokens = 0;

        for (Document doc : documents) {
            int docTokens = estimateTokens(doc.getText());
            if (currentTokens + docTokens > MAX_CONTEXT_TOKENS) {
                break; // Budget exceeded, stop adding
            }
            context.append(doc.getText()).append("

");
            currentTokens += docTokens;
        }
        return context.toString();
    }

    private int estimateTokens(String text) {
        // Rough estimate: 1 Chinese char ≈ 1.5 tokens
        return (int) (text.length() * 1.5);
    }
}

6.3 Smart Model Routing

A router selects a cheaper model (GPT-3.5) for simple questions with sufficient context (≥3 documents), and a stronger model (GPT-4) for complex queries. Simplicity is heuristically defined as length < 30 characters and no reasoning requirement.

@Service
public class SmartModelRouter {
    private final ChatModel gpt4;
    private final ChatModel gpt35;

    public ChatModel route(String question, List<Document> contexts) {
        // Simple question + enough context → cheaper model
        if (isSimpleQuestion(question) && contexts.size() >= 3) {
            return gpt35;
        }
        // Complex question → stronger model
        return gpt4;
    }

    private boolean isSimpleQuestion(String question) {
        return question.length() < 30 && !needsReasoning(question);
    }
}

Monitoring and Alerting

7.1 Key Metrics

The article categorizes metrics into four groups:

Performance : Retrieval latency (P50/P95/P99), generation latency (P50/P95/P99), end-to-end latency

Quality : Retrieval hit rate, answer faithfulness, user satisfaction

Cost : Per-request token consumption, daily total tokens, cache hit rate

Errors : Error rate, timeout count, model fallback count

7.2 Metrics Collection Implementation

A RagMetricsCollector uses Micrometer's MeterRegistry to record timers for retrieval and generation durations (publishing 0.5, 0.95, 0.99 percentiles), counters for total requests and cache hits.

@Component
public class RagMetricsCollector {
    private final MeterRegistry meterRegistry;
    private final Timer retrievalTimer;
    private final Timer generationTimer;
    private final Counter requestCounter;
    private final Counter cacheHitCounter;

    public RagMetricsCollector(MeterRegistry meterRegistry) {
        this.meterRegistry = meterRegistry;
        this.retrievalTimer = Timer.builder("rag.retrieval.duration")
            .publishPercentiles(0.5, 0.95, 0.99)
            .register(meterRegistry);
        this.generationTimer = Timer.builder("rag.generation.duration")
            .publishPercentiles(0.5, 0.95, 0.99)
            .register(meterRegistry);
        this.requestCounter = Counter.builder("rag.requests.total")
            .register(meterRegistry);
        this.cacheHitCounter = Counter.builder("rag.cache.hits")
            .register(meterRegistry);
    }

    public void recordRetrieval(long durationMs) {
        retrievalTimer.record(durationMs, TimeUnit.MILLISECONDS);
    }

    public void recordGeneration(long durationMs) {
        generationTimer.record(durationMs, TimeUnit.MILLISECONDS);
    }

    public void recordRequest() {
        requestCounter.increment();
    }

    public void recordCacheHit() {
        cacheHitCounter.increment();
    }
}

Deployment Checklist

A structured checklist groups items by concern:

High Availability : RAG service multi-replica, vector DB cluster, multi-model fallback, Redis HA

Performance : Multi-level cache, async document ingestion, connection pool tuning

Cost : Token budget management, model routing, cost monitoring alerts

Security : Rate limiting, input filtering, access control

Monitoring : Full-chain tracing, key metric collection, alert configuration

The article concludes by previewing the next episode: a complete enterprise knowledge base project with end-to-end implementation and deployment.