How to Build a Production‑Ready AI Service with Spring 7.0 and Spring AI for High‑Concurrency Microservices

Why AI integration is more than a simple API call

When large models are added to enterprise microservices, the real challenges are not just calling the API once but handling four hard problems: high concurrency load, model instability affecting the main transaction, managing knowledge and session state, and ensuring the service can be observed, rate‑limited, and rolled back.

Spring 7.0 together with Spring AI provides a way for Java teams to embed AI within the existing microservice governance framework rather than building a separate demo.

Defining the AI capability layer

Instead of a single "chat endpoint", the article proposes an AI capability layer consisting of four orthogonal concerns:

Access layer – unified entry, authentication, rate limiting, routing, tenant isolation.

Execution layer – prompt orchestration, model calls, tool calls, streaming output.

State layer – session memory, knowledge retrieval, caching, audit records.

Governance layer – timeout, circuit‑breaker, degradation, observability, evaluation.

A textual diagram (shown in the original article) illustrates how these layers sit between the web gateway and downstream components such as Redis, PostgreSQL, Kafka, and the vector store.

Why choose Spring 7.0 × Spring AI over a custom Python service

The advantages stem from alignment with existing Spring ecosystem capabilities: consistent integration with the microservice stack, built‑in observability, mature thread‑pool and connection‑pool management, seamless integration with Spring Security, Gateway, Batch, Kafka, Redis, PostgreSQL, and Kubernetes, and reuse of team engineering skills.

Core value added by Spring 7.0

Both Web MVC and WebFlux stacks are retained, allowing synchronous or reactive access.

Observability is a first‑class citizen, making it easy to include AI calls in metrics and tracing.

Better alignment with modern Java concurrency models for I/O‑heavy, long‑wait workflows.

Improved production‑grade configuration, health checks, native image support, and auto‑configuration.

What Spring AI abstracts

ChatModel

/ ChatClient – unified chat model access. EmbeddingModel – unified vectorization entry. VectorStore – unified vector database abstraction. Advisor – cross‑cutting enhancements such as logging, memory, RAG, prompt rewriting, protection.

Tool Calling – registers business functions as model‑callable tools.

Observability – integrates model calls into metrics and tracing.

ETL / Document Pipeline – consolidates document ingestion, chunking, transformation, and knowledge‑base loading.

These abstractions let developers avoid vendor lock‑in on day one.

Four design principles for high‑concurrency AI services

Never place the large model at the core of the main transaction. Use the model as an asynchronous enhancer; keep the primary business flow deterministic.

Route before generate. Classify requests (FAQ, RAG, tool call, human fallback) and only invoke the model for truly high‑cost cases.

Explicitly model state. Avoid appending raw history to prompts; store session state in a database or cache and inject only the needed context.

Assume model unreliability. Design for vendor rate limits, latency spikes, output drift, tool‑call failures, and irrelevant retrieval results; use shields and fallbacks instead of trusting the model.

Recommended layered implementation

Controller layer

Only handles protocol conversion, basic validation, and propagates tenant, user, trace, and session identifiers.

package com.example.ai.api.dto;

import jakarta.validation.constraints.NotBlank;
import jakarta.validation.constraints.Size;

public record ChatRequest(
        @NotBlank(message = "sessionId cannot be empty") String sessionId,
        @NotBlank(message = "message cannot be empty") @Size(max = 4000, message = "message too long") String message,
        String channel,
        boolean stream) {}

package com.example.ai.api;

import com.example.ai.api.dto.ChatRequest;
import com.example.ai.application.ChatApplicationService;
import com.example.ai.application.ChatReply;
import jakarta.validation.Valid;
import lombok.RequiredArgsConstructor;
import org.springframework.http.MediaType;
import org.springframework.http.codec.ServerSentEvent;
import org.springframework.web.bind.annotation.*;
import reactor.core.publisher.Flux;

@RestController
@RequestMapping("/api/ai/chat")
@RequiredArgsConstructor
public class ChatController {
    private final ChatApplicationService chatApplicationService;

    @PostMapping
    public ChatReply chat(@Valid @RequestBody ChatRequest request,
                         @RequestHeader("X-Tenant-Id") String tenantId,
                         @RequestHeader("X-User-Id") String userId) {
        return chatApplicationService.chat(tenantId, userId, request);
    }

    @PostMapping(value = "/stream", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
    public Flux<ServerSentEvent<String>> stream(@Valid @RequestBody ChatRequest request,
                                                @RequestHeader("X-Tenant-Id") String tenantId,
                                                @RequestHeader("X-User-Id") String userId) {
        return chatApplicationService.stream(tenantId, userId, request)
                .map(token -> ServerSentEvent.builder(token).build());
    }
}

Application (orchestration) layer

Coordinates independent capabilities – cache lookup, intent classification, knowledge retrieval, model invocation, and audit publishing.

package com.example.ai.application;

import com.example.ai.api.dto.ChatRequest;
import reactor.core.publisher.Flux;

public interface ChatApplicationService {
    ChatReply chat(String tenantId, String userId, ChatRequest request);
    Flux<String> stream(String tenantId, String userId, ChatRequest request);
}

package com.example.ai.application;

import com.example.ai.domain.ChatIntent;
import com.example.ai.domain.IntentClassifier;
import com.example.ai.domain.SemanticCache;
import com.example.ai.domain.TenantScopedRetriever;
import com.example.ai.infrastructure.audit.ChatAuditPublisher;
import com.example.ai.infrastructure.model.ChatGateway;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Service;
import reactor.core.publisher.Flux;

import java.time.Duration;
import java.util.List;

@Slf4j
@Service
@RequiredArgsConstructor
public class DefaultChatApplicationService implements ChatApplicationService {
    private final SemanticCache semanticCache;
    private final IntentClassifier intentClassifier;
    private final TenantScopedRetriever retriever;
    private final ChatGateway chatGateway;
    private final ChatAuditPublisher auditPublisher;

    @Override
    public ChatReply chat(String tenantId, String userId, ChatRequest request) {
        ChatExecutionContext context = ChatExecutionContext.of(tenantId, userId, request.sessionId(), request.message());
        ChatReply cached = semanticCache.get(context);
        if (cached != null) {
            auditPublisher.publishCacheHit(context, cached);
            return cached;
        }
        ChatIntent intent = intentClassifier.classify(context.message());
        List<KnowledgeSnippet> snippets = retriever.retrieve(context, intent);
        ChatReply reply = chatGateway.generate(context, intent, snippets);
        semanticCache.put(context, reply, Duration.ofMinutes(30));
        auditPublisher.publishSuccess(context, intent, snippets, reply);
        return reply;
    }

    @Override
    public Flux<String> stream(String tenantId, String userId, ChatRequest request) {
        ChatExecutionContext context = ChatExecutionContext.of(tenantId, userId, request.sessionId(), request.message());
        ChatIntent intent = intentClassifier.classify(context.message());
        List<KnowledgeSnippet> snippets = retriever.retrieve(context, intent);
        return chatGateway.stream(context, intent, snippets)
                .doOnComplete(() -> auditPublisher.publishStreamCompleted(context, intent, snippets))
                .doOnError(ex -> auditPublisher.publishFailure(context, intent, ex));
    }
}

Retrieval (RAG) layer

Implements tenant‑scoped, domain‑filtered vector search.

package com.example.ai.domain;

import com.example.ai.application.ChatExecutionContext;
import com.example.ai.application.KnowledgeSnippet;
import java.util.List;

public interface TenantScopedRetriever {
    List<KnowledgeSnippet> retrieve(ChatExecutionContext context, ChatIntent intent);
}

package com.example.ai.infrastructure.retrieval;

import com.example.ai.application.ChatExecutionContext;
import com.example.ai.application.KnowledgeSnippet;
import com.example.ai.domain.ChatIntent;
import com.example.ai.domain.TenantScopedRetriever;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.ai.document.Document;
import org.springframework.ai.vectorstore.SearchRequest;
import org.springframework.ai.vectorstore.VectorStore;
import org.springframework.stereotype.Component;
import java.util.List;

@Slf4j
@Component
@RequiredArgsConstructor
public class PgVectorTenantScopedRetriever implements TenantScopedRetriever {
    private final VectorStore vectorStore;

    @Override
    public List<KnowledgeSnippet> retrieve(ChatExecutionContext context, ChatIntent intent) {
        SearchRequest searchRequest = SearchRequest.builder()
                .query(context.message())
                .topK(6)
                .similarityThreshold(0.78d)
                .filterExpression("tenantId == '" + context.tenantId() + "' && domain == '" + intent.knowledgeDomain() + "'")
                .build();
        List<Document> documents = vectorStore.similaritySearch(searchRequest);
        return documents.stream()
                .map(d -> new KnowledgeSnippet(d.getId(), d.getText(), d.getMetadata()))
                .toList();
    }
}

Model access layer

Wraps the Spring AI ChatClient with resilience annotations and builds a system prompt that injects intent and retrieved knowledge.

package com.example.ai.infrastructure.model;

import com.example.ai.application.ChatExecutionContext;
import com.example.ai.application.ChatReply;
import com.example.ai.application.KnowledgeSnippet;
import com.example.ai.domain.ChatIntent;
import io.github.resilience4j.circuitbreaker.annotation.CircuitBreaker;
import io.github.resilience4j.timelimiter.annotation.TimeLimiter;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.ai.chat.client.ChatClient;
import org.springframework.ai.chat.prompt.SystemPromptTemplate;
import org.springframework.stereotype.Component;
import reactor.core.publisher.Flux;
import java.util.List;
import java.util.Map;
import java.util.concurrent.CompletableFuture;

@Slf4j
@Component
@RequiredArgsConstructor
public class SpringAiChatGateway implements ChatGateway {
    private final ChatClient chatClient;

    @Override
    @CircuitBreaker(name = "llm")
    @TimeLimiter(name = "llm")
    public ChatReply generate(ChatExecutionContext context, ChatIntent intent, List<KnowledgeSnippet> snippets) {
        String systemPrompt = buildSystemPrompt(intent, snippets);
        String content = chatClient.prompt()
                .system(systemPrompt)
                .user(context.message())
                .call()
                .content();
        return new ChatReply(content, "PRIMARY_MODEL", false);
    }

    @Override
    public Flux<String> stream(ChatExecutionContext context, ChatIntent intent, List<KnowledgeSnippet> snippets) {
        String systemPrompt = buildSystemPrompt(intent, snippets);
        return chatClient.prompt()
                .system(systemPrompt)
                .user(context.message())
                .stream()
                .content();
    }

    private String buildSystemPrompt(ChatIntent intent, List<KnowledgeSnippet> snippets) {
        String knowledge = snippets.stream()
                .map(KnowledgeSnippet::content)
                .reduce((a, b) -> a + "
---
" + b)
                .orElse("");
        return new SystemPromptTemplate("""
                You are an enterprise customer‑service assistant.
                Follow these rules:
                1. Prefer the provided knowledge context.
                2. If unsure, say you don't know – do not hallucinate.
                3. For order‑related actions, give explicit steps.
                4. Never leak information outside the tenant.

                Current intent: {intent}
                Knowledge context:
                {knowledge}
                """)
                .render(Map.of("intent", intent.name(), "knowledge", knowledge));
    }
}

Semantic cache

Stores AI responses keyed by tenant and a fingerprint of the user message, using Redis for fast lookup.

package com.example.ai.domain;

import com.example.ai.application.ChatExecutionContext;
import com.example.ai.application.ChatReply;
import java.time.Duration;

public interface SemanticCache {
    ChatReply get(ChatExecutionContext context);
    void put(ChatExecutionContext context, ChatReply reply, Duration ttl);
}

package com.example.ai.infrastructure.cache;

import com.example.ai.application.ChatExecutionContext;
import com.example.ai.application.ChatReply;
import com.example.ai.domain.SemanticCache;
import lombok.RequiredArgsConstructor;
import org.springframework.data.redis.core.StringRedisTemplate;
import org.springframework.stereotype.Component;
import java.time.Duration;

@Component
@RequiredArgsConstructor
public class RedisSemanticCache implements SemanticCache {
    private final StringRedisTemplate redisTemplate;
    private final SemanticFingerprintService fingerprintService;

    @Override
    public ChatReply get(ChatExecutionContext context) {
        String key = "ai:semantic:" + context.tenantId() + ":" + fingerprintService.fingerprint(context.message());
        String value = redisTemplate.opsForValue().get(key);
        return value == null ? null : ChatReply.deserialize(value);
    }

    @Override
    public void put(ChatExecutionContext context, ChatReply reply, Duration ttl) {
        String key = "ai:semantic:" + context.tenantId() + ":" + fingerprintService.fingerprint(context.message());
        redisTemplate.opsForValue().set(key, reply.serialize(), ttl);
    }
}

High‑concurrency engineering considerations

Thread model: Use standard Spring Web/WebFlux for entry, virtual threads for blocking I/O, and reactive streams for long‑running streaming responses.

Layered timeouts: Separate budgets for gateway (≈10 s), retrieval (≈300 ms), rerank (≈150 ms), model first‑token (≈2 s), full model (≈8 s), and tool calls (≈500 ms).

Rate limiting: Apply limits per tenant, per user, per API, per model, and per token budget to prevent a single noisy tenant from exhausting resources.

Degradation strategy: Four‑level fallback – semantic cache → FAQ/rules → smaller model → human hand‑off.

Observability and audit

Collect AI‑specific metrics (request count, success/error rate, latency percentiles, token usage, cache hit ratio, tool‑call success, per‑tenant cost) and create fine‑grained tracing spans such as:

gateway.auth

chat.intent.classify

chat.semantic.cache

chat.retrieval

chat.rerank

chat.llm.call

chat.tool.invoke

chat.audit.publish

Audit records must capture request summary, session IDs, model used, retrieved document IDs, cache hits, tool‑call details, latency, token counts, response summary, and any error codes. Sensitive user data should be redacted.

Production checklist before launch

Tenant isolation enforced outside prompts.

Layered timeout budgets defined.

Vendor rate‑limit, circuit‑breaker, and fallback policies in place.

Metrics for cache hit rate, token consumption, and latency collected.

Ability to replay a full AI request trace.

Tool calls protected by permission checks, idempotency, and state validation.

Knowledge documents versioned and cleaned up regularly.

Human fallback path ready.

Resource cleanup verified for aborted streaming connections.

Load‑testing performed beyond local demos.

Canary deployment and fast rollback mechanisms prepared.

Sensitive‑information masking and audit compliance in place.

Evolution roadmap

The article outlines four maturity stages: a single‑service proof‑of‑concept, governance‑enhanced service, a shared AI capability platform, and finally an agent‑oriented intelligent orchestration layer. Each stage builds on the previous one’s stability and observability foundations.

Conclusion

Spring 7.0 × Spring AI’s real value lies in letting Java teams keep the familiar microservice engineering discipline—isolating, rate‑limiting, caching, degrading, auditing, and observing—while safely harnessing large‑model capabilities as a production‑grade "AI blade".