Building a Pluggable LLM Gateway with Spring Boot

Why a Unified LLM Gateway?

Large language models (LLMs) are now standard in enterprise systems for chatbots, content generation, data analysis, code assistance, and knowledge‑base Q&A. Most teams integrate them by directly embedding each vendor’s SDK, hard‑coding calls, and duplicating common concerns like timeout, retry, circuit‑breaker, logging, and token accounting. This creates tightly coupled, "siloed" code that is hard to maintain and switch.

Key Pain Points

Vendor lock‑in : Changing a model requires replacing dependencies, rewriting call logic, and full regression testing.

Repeated generic capabilities : Every service re‑implements timeout, retry, fallback, rate‑limit, logging, and token statistics.

Scattered secret management : API keys are stored in each service, risking leaks and making unified permission control impossible.

No global traffic governance : A single vendor outage can bring down all dependent services.

Core Design Idea: Interface‑Driven + Strategy Pattern for True Plug‑ability

The gateway follows the Open‑Closed Principle: business code depends only on abstract interfaces, while the gateway encapsulates vendor‑specific implementations. Adding or switching a model never touches business code.

Overall Architecture (Four Layers)

Access Layer : Exposes a standard HTTP API; request/response formats are fixed.

Gateway Core Layer : Strategy factory for routing, generic governance (retry, circuit‑breaker, rate‑limit, logging, billing).

Vendor Adaptation Layer : One implementation class per vendor handling parameter conversion, protocol adaptation, and API calls.

Base Configuration Layer : Centralized management of API keys, service URLs, and environment‑specific settings.

Design Patterns

Strategy Pattern : Define a top‑level LlmClient interface; each vendor provides a concrete strategy selected by a factory.

Template Method : Common logging, statistics, and exception handling are implemented once in the gateway; vendor code only deals with protocol conversion.

Interface‑Driven Programming : Business modules depend solely on the abstract interface, achieving true model replaceability.

Step‑by‑Step Extensibility

Adding a new model provider requires only two actions:

Create a new class implementing LlmClient and annotate it with @Component to handle request conversion and response mapping.

Add the provider’s api-key and base-url to application.yml. No business code changes are needed.

Key Code Snippets

pom.xml (partial)

<project xmlns="http://maven.apache.org/POM/4.0.0">
  <modelVersion>4.0.0</modelVersion>
  <parent>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-parent</artifactId>
    <version>3.3.0</version>
  </parent>
  <properties>
    <java.version>21</java.version>
  </properties>
  <dependencies>
    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-web</artifactId>
    </dependency>
    <!-- OpenFeign for HTTP calls -->
    <dependency>
      <groupId>org.springframework.cloud</groupId>
      <artifactId>spring-cloud-starter-openfeign</artifactId>
      <version>4.1.0</version>
    </dependency>
    <!-- Resilience4j for retry/circuit‑breaker -->
    <dependency>
      <groupId>io.github.resilience4j</groupId>
      <artifactId>resilience4j-spring-boot3</artifactId>
      <version>2.2.0</version>
    </dependency>
  </dependencies>
</project>

Unified Request/Response DTOs

@Data
public class LlmChatRequest {
    /** optional provider, defaults to global setting */
    private String modelProvider;
    /** e.g., gpt-4o, qwen-plus, deepseek-chat */
    private String modelName;
    private List<Message> messages;
    private Double temperature; // 0‑1
    private Integer maxTokens;
    private boolean stream = false;

    @Data
    public static class Message {
        private String role; // system / user / assistant
        private String content;
    }
}

@Data
public class LlmChatResponse {
    private boolean success;
    private String content;
    private Integer promptTokens;
    private Integer completionTokens;
    private Integer totalTokens;
    private String usedProvider;
    private String errorMsg;
}

Top‑Level Interface

public interface LlmClient {
    /** Returns a unique provider identifier, e.g., "openai", "deepseek" */
    String getProviderName();

    /** Non‑streaming chat */
    LlmChatResponse chat(LlmChatRequest request);

    /** Optional streaming chat (default throws UnsupportedOperationException) */
    default void chatStream(LlmChatRequest request) {
        throw new UnsupportedOperationException("Current model does not support streaming");
    }
}

Strategy Factory (dynamic routing)

@Component
@RequiredArgsConstructor
public class LlmClientFactory {
    private final List<LlmClient> allLlmClients;
    private final LlmProperties llmProperties;
    private final Map<String, LlmClient> clientMap = new HashMap<>();

    @PostConstruct
    public void init() {
        for (LlmClient client : allLlmClients) {
            clientMap.put(client.getProviderName(), client);
        }
    }

    public LlmClient getClient(String provider) {
        String realProvider = (provider == null) ? llmProperties.getDefaultProvider() : provider;
        LlmClient client = clientMap.get(realProvider);
        if (client == null) {
            throw new IllegalArgumentException("Unknown model provider: " + realProvider);
        }
        return client;
    }
}

DeepSeek Vendor Implementation (example)

@Component
@RequiredArgsConstructor
public class DeepSeekLlmClient implements LlmClient {
    private final DeepSeekFeignClient deepSeekFeignClient;

    @Override
    public String getProviderName() { return "deepseek"; }

    @Override
    @Retry(name = "llmRetry")
    @CircuitBreaker(name = "llmCircuit")
    public LlmChatResponse chat(LlmChatRequest request) {
        // 1. Convert to vendor‑specific payload
        Map<String, Object> body = new HashMap<>();
        body.put("model", request.getModelName());
        body.put("messages", request.getMessages());
        body.put("temperature", request.getTemperature());
        body.put("max_tokens", request.getMaxTokens());
        body.put("stream", false);
        try {
            // 2. Call vendor API via Feign
            Map<String, Object> resp = deepSeekFeignClient.chatCompletions(body);
            LlmChatResponse result = new LlmChatResponse();
            result.setSuccess(true);
            result.setUsedProvider(getProviderName());
            // 3. Map response fields
            List<?> choices = (List<?>) resp.get("choices");
            if (!choices.isEmpty()) {
                Map<?, ?> choice = (Map<?, ?>) choices.get(0);
                Map<?, ?> msg = (Map<?, ?>) choice.get("message");
                result.setContent(msg.get("content").toString());
            }
            Map<?, ?> usage = (Map<?, ?>) resp.get("usage");
            result.setPromptTokens((Integer) usage.get("prompt_tokens"));
            result.setCompletionTokens((Integer) usage.get("completion_tokens"));
            result.setTotalTokens((Integer) usage.get("total_tokens"));
            return result;
        } catch (Exception e) {
            // 4. Uniform error handling
            LlmChatResponse err = new LlmChatResponse();
            err.setSuccess(false);
            err.setUsedProvider(getProviderName());
            err.setErrorMsg(e.getMessage());
            return err;
        }
    }

    @FeignClient(name = "deepseek", url = "${llm.providers.deepseek.base-url}", configuration = DeepSeekFeignConfig.class)
    public interface DeepSeekFeignClient {
        @PostMapping("/chat/completions")
        Map<String, Object> chatCompletions(@RequestBody Map<String, Object> body);
    }

    public static class DeepSeekFeignConfig {
        private final LlmProperties llmProperties;
        public DeepSeekFeignConfig(LlmProperties llmProperties) { this.llmProperties = llmProperties; }
        @Bean
        public RequestInterceptor authInterceptor() {
            return template -> {
                String key = llmProperties.getProviders().get("deepseek").getApiKey();
                template.header("Authorization", "Bearer " + key);
            };
        }
    }
}

Gateway Controller (single entry point)

@RestController
@RequestMapping("/api/llm/gateway")
@RequiredArgsConstructor
public class LlmGatewayController {
    private final LlmClientFactory llmClientFactory;

    @PostMapping("/chat")
    public LlmChatResponse chat(@RequestBody LlmChatRequest request) {
        log.info("[LLM Gateway] provider={}, model={}", request.getModelProvider(), request.getModelName());
        LlmClient client = llmClientFactory.getClient(request.getModelProvider());
        LlmChatResponse resp = client.chat(request);
        log.info("[LLM Gateway] success={}, totalTokens={}", resp.isSuccess(), resp.getTotalTokens());
        return resp;
    }
}

application.yml (central configuration)

llm:
  default-provider: deepseek
  providers:
    openai:
      api-key: sk-xxx
      base-url: https://api.openai.com/v1
    deepseek:
      api-key: sk-xxx
      base-url: https://api.deepseek.com/v1
    qwen:
      api-key: sk-xxx
      base-url: https://dashscope.aliyuncs.com/compatible-mode/v1

resilience4j:
  retry:
    instances:
      llmRetry:
        max-attempts: 3
        wait-duration: 1000
        enable-exponential-backoff: true
  circuitbreaker:
    instances:
      llmCircuit:
        sliding-window-size: 20
        failure-rate-threshold: 50
        wait-duration-in-open-state: 5000

Production‑Grade Enhancements

Retry & Circuit‑Breaker : Configured via Resilience4j; automatic exponential back‑off up to three attempts, and circuit‑breaker to prevent cascading failures.

Automatic Failover : If the primary model fails, the gateway can fall back to a secondary provider, raising overall availability from ~99.9% to ~99.99%.

Global AOP Logging & Token Accounting : A custom aspect records request parameters, response, latency, token usage, provider, and model for auditing and cost allocation.

Auth & Rate Limiting : AppKey‑based authentication hides raw vendor keys; per‑client QPS and daily quota limits protect model quotas.

SSE Streaming Support : Optional chatStream method returns Server‑Sent Events for real‑time token‑by‑token output.

Response Caching : Frequently repeated prompts are cached at the gateway level, reducing cost and latency.

When to Use This Gateway

✅ Recommended for enterprises with multiple business lines, frequent model comparisons, strict stability requirements, and the need for unified cost and permission management.

❌ Not necessary for simple, single‑model use‑cases where direct SDK integration is sufficient.

Conclusion

Integrating LLMs is more than a single API call; as usage grows, siloed implementations become unsustainable. A pluggable Spring Boot gateway centralizes vendor adaptation, traffic governance, and operational metrics, allowing business code to stay focused on domain logic while supporting scalable, reliable, and cost‑effective AI services.