Deploy and Test the Latest DeepSeek and Qwen Open‑Source LLMs Locally (Beginner to Advanced)

Local deployment rationale

Model inference runs on the PC or mobile device, eliminating cloud compute costs.

API calls stay within the local network, avoiding token‑based usage fees.

Sensitive data never leaves the local environment.

Why choose the DeepSeek R1 distilled model

Local hardware can only accommodate a small‑parameter model; the full‑size Qwen model is larger, while DeepSeek R1 provides a 7B distilled variant that fits modest devices.

The DeepSeek R1 open‑source license explicitly permits model distillation, and a Qwen‑based distilled build is available for direct download.

Install Ollama and run the models

# Install Ollama
curl -fsSL https://ollama.com/install.sh | sh

# Run DeepSeek R1 distilled model
ollama run deepseek-r1:7b

Ollama supports DeepSeek R1 and Qwen2.5; Qwen2.5‑Max will be added once Ollama updates.

Spring AI Alibaba application

Add the Alibaba starter and the Ollama starter to the Maven project:

<dependency>
  <groupId>com.alibaba.cloud.ai</groupId>
  <artifactId>spring-ai-alibaba-starter</artifactId>
  <version>1.0.0-M5.1</version>
</dependency>
<dependency>
  <groupId>org.springframework.ai</groupId>
  <artifactId>spring-ai-ollama-spring-boot-starter</artifactId>
  <version>1.0.0-M5</version>
</dependency>

If the Spring AI artifacts are not yet in the central repository, add the Spring Milestones repository to pom.xml:

<repositories>
  <repository>
    <id>spring-milestones</id>
    <name>Spring Milestones</name>
    <url>https://repo.spring.io/milestone</url>
    <snapshots>
      <enabled>false</enabled>
    </snapshots>
  </repository>
</repositories>

Inject ChatClient and expose a simple REST endpoint:

@RestController
public class ChatController {
  private final ChatClient chatClient;

  public ChatController(ChatClient.Builder builder) {
    this.chatClient = builder.build();
  }

  @GetMapping("/chat")
  public String chat(String input) {
    return this.chatClient.prompt()
        .user(input)
        .call()
        .content();
  }
}

Configure the model URL and name in application.properties:

spring.ai.ollama.base-url=http://localhost:11434
spring.ai.ollama.chat.model=deepseek-r1:7b

Full source code: https://github.com/springaialibaba/spring-ai-alibaba-examples/tree/main/spring-ai-alibaba-chat-example/ollama-chat/ollama-chat-client

Production scenario with Higress gateway

Higress is a cloud‑native API gateway built on Istio and Envoy, enhanced for AI workloads.

Install Higress with a single command:

curl -sS https://higress.cn/ai-gateway/install.sh | bash

After installation, the gateway can route requests to any major LLM provider, including DeepSeek and Qwen. API keys are managed centrally, and the gateway offers key pooling, consumer management, fallback models, and gray‑release (model rollout).

Python client that talks to an OpenAI‑compatible endpoint behind Higress:

import json
from openai import OpenAI

client = OpenAI(
    api_key='xxxxx',  # Higress can generate consumer‑level keys
    base_url="http://127.0.0.1:8080/v1"
)

completion = client.chat.completions.create(
    model="deepseek-chat",  # any model name is routed by Higress
    messages=[{"role": "user", "content": "你好"}],
    stream=True
)
for chunk in completion:
    print(chunk.choices[0].delta)

When using Spring AI Alibaba, add the OpenAI starter and point it to the Higress gateway:

<dependency>
  <groupId>org.springframework.ai</groupId>
  <artifactId>spring-ai-openai-spring-boot-starter</artifactId>
  <version>1.0.0-M5</version>
</dependency>

spring.ai.openai.base-url=http://127.0.0.1:8080/v1
spring.ai.openai.chat.model=deepseek-chat
spring.ai.openai.chat.api-key=xxxxx

Higress monitoring shows per‑model token consumption and latency. Additional built‑in plugins include:

API Key governance : configure an API‑Key pool; unavailable keys are automatically bypassed.

Consumer management : create consumers to distribute vendor keys without exposing them, and control per‑consumer quotas.

Fallback model : specify a secondary model (e.g., OpenAI) that is invoked when the primary model fails.

Model gray release : smooth, proportion‑based rollout of new model versions.

References

https://github.com/deepseek-ai/DeepSeek-R1

https://qwenlm.github.io/blog/qwen2.5-vl/

https://github.com/deepseek-ai/Janus?tab=readme-ov-file

https://qwenlm.github.io/blog/qwen2.5-1m

https://qwenlm.github.io/zh/blog/qwen2.5-max/

https://github.com/alibaba/higress