Production-Grade Deployment and Best Practices for Java AI Applications

After completing the first five episodes that built a Java AI application from basic calls to an agent framework, the author addresses the remaining problem of running these AI services reliably in production.

1. Core Production Challenges

The author identifies three primary concerns:

Stability : LLM calls may timeout, fail, or return malformed data.

Cost : Token usage directly impacts expenses and must be controlled.

Observability : AI services act as black boxes, requiring insight into their internal behavior.

2. High Availability and Fault‑Tolerance Design

2.1 Timeout and Retry

// config/RetryConfig.java
@Configuration
public class RetryConfig {

    @Bean
    public RetryTemplate aiRetryTemplate() {
        return RetryTemplate.builder()
                .maxAttempts(3)
                .exponentialBackoff(1000, 2, 5000) // 1s → 2s → 4s
                .retryOn(TimeoutException.class)
                .retryOn(IOException.class)
                .build();
    }

    @Bean
    public RestTemplate aiRestTemplate() {
        return new RestTemplateBuilder()
                .setConnectTimeout(Duration.ofSeconds(5))
                .setReadTimeout(Duration.ofSeconds(30))
                .build();
    }
}

2.2 Circuit Breaker and Degradation

Resilience4j is used to protect the service:

# application.yml
resilience4j:
  circuitbreaker:
    instances:
      aiService:
        sliding-window-size: 10
        failure-rate-threshold: 50
        wait-duration-in-open-state: 30s

@Service
public class AiService {

    @CircuitBreaker(name = "aiService", fallbackMethod = "fallback")
    public String chat(String message) {
        return chatClient.prompt(message).call().content();
    }

    public String fallback(String message, Throwable t) {
        return "AI service temporarily unavailable, please retry later";
    }
}

2.3 Multi‑Model Degradation

@Service
public class ModelFallbackService {
    private final List<ChatModel> models = List.of(
        new OpenAiChatModel(),   // primary model
        new DeepSeekChatModel(), // backup 1
        new OllamaChatModel()    // backup 2
    );

    public String chat(String message) {
        for (ChatModel model : models) {
            try {
                return model.call(message);
            } catch (Exception e) {
                log.warn("Model call failed, switching to next: {}", e.getMessage());
            }
        }
        return "All models are unavailable";
    }
}

3. Cost Control

3.1 Token Monitoring and Limiting

@Component
public class TokenCostInterceptor implements RequestInterceptor {
    private final AtomicLong totalTokens = new AtomicLong();

    @Override
    public void apply(RequestTemplate template) {
        // record token usage per request
    }

    @Scheduled(fixedRate = 60000)
    public void checkBudget() {
        long used = totalTokens.get();
        if (used > 100000) {
            log.warn("Token usage exceeds threshold: {}", used);
        }
    }
}

3.2 Caching Strategy

@Service
public class CachedAiService {
    @Cacheable(value = "ai-responses", key = "#message")
    public String chat(String message) {
        return chatClient.prompt(message).call().content();
    }
}

4. Observability

4.1 Distributed Tracing

<dependency>
  <groupId>io.micrometer</groupId>
  <artifactId>micrometer-tracing-bridge-brave</artifactId>
</dependency>

@RestController
public class AiController {
    @GetMapping("/chat")
    public String chat(@RequestParam String message, @CurrentSpan Span span) {
        span.tag("ai.model", "gpt-4");
        span.tag("ai.tokens", "1234");
        return aiService.chat(message);
    }
}

4.2 Custom Metrics

@Component
public class AiMetrics {
    private final Counter requestCounter;
    private final Timer requestTimer;

    public AiMetrics(MeterRegistry registry) {
        this.requestCounter = Counter.builder("ai.requests.total")
                .description("Total AI requests")
                .register(registry);
        this.requestTimer = Timer.builder("ai.request.duration")
                .publishPercentiles(0.5, 0.95, 0.99)
                .register(registry);
    }

    public void recordRequest(long duration, boolean success) {
        requestCounter.increment();
        requestTimer.record(duration, TimeUnit.MILLISECONDS);
    }
}

5. Containerized Deployment

FROM openjdk:17-jdk-slim

WORKDIR /app
COPY target/*.jar app.jar

ENV JAVA_OPTS="-Xms512m -Xmx512m -XX:+UseG1GC"

ENTRYPOINT ["sh", "-c", "java $JAVA_OPTS -jar app.jar"]

6. Best‑Practice Checklist

Timeout Settings : Configure reasonable timeouts for each LLM call.

Retry Mechanism : Use exponential back‑off to avoid cascade failures.

Circuit‑Breaker Degradation : Gracefully fall back when dependencies are unavailable.

Cost Monitoring : Track token consumption in real time.

Observability : Enable tracing and custom metrics for request latency and counts.

Caching : Cache frequent queries to reduce token usage.

Security : Store API keys encrypted, preferably via environment variables.

The series concludes with this sixth episode, completing the Java AI application development guide.