Full Spring Boot Example: Integrating Spring AI with Local Ollama for Fast AI Chat

Scenario

Running large language models locally enables rapid development, data privacy, cost‑free operation, offline usage, and custom model selection. Ollama provides an open‑source, OpenAI‑compatible API that can be integrated with Spring AI.

Installation and Configuration of Ollama

Step 1: Download and install Ollama

https://ollama.com/

On Windows the service runs at http://localhost:11434.

Step 2: Start the Ollama service

# Windows PowerShell
ollama list   # list downloaded models
ollama ps      # view active models

Step 3: Pull a local model

# Recommended model (good compatibility with Alibaba Cloud Bailei)
ollama pull qwen2.5
# Alternative models
ollama pull llama3.2
ollama pull phi3
ollama pull mistral

Step 4: Verify the model works

# Simple conversation test
ollama run qwen2.5 "你好，请介绍一下你自己"
/bye

Environment Preparation: JDK 17

Spring AI and Spring Boot 3.x require at least JDK 17. If the system default is JDK 8, configure the project to use JDK 17 without changing the global JDK.

Configure JDK 17 for the project

Download a JDK 17 zip (e.g., D:\SoftWare\jdk\jdk17) and extract it. Do not add it to PATH or modify system variables.

Create a startup script that sets JAVA_HOME and updates PATH before invoking Maven.

@echo off
chcp 65001 >nul
echo ========================================
echo   Spring AI Ollama Manager
echo ========================================
set MVN=D:\SoftWare\Maven\apache-maven-3.6.3\bin\mvn.cmd
set JAVA_HOME=D:\SoftWare\jdk\jdk17
set PATH=%JAVA_HOME%\bin;%PATH%

Create Spring Boot Project

pom.xml

<parent>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-parent</artifactId>
    <version>3.2.5</version>
</parent>

<groupId>com.example</groupId>
<artifactId>spring-ai-ollama-demo</artifactId>
<version>1.0</version>
<properties>
    <java.version>17</java.version>
    <spring-ai.version>1.0.0-M5</spring-ai.version>
</properties>

<dependencies>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-web</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.ai</groupId>
        <artifactId>spring-ai-openai-spring-boot-starter</artifactId>
        <version>1.0.0-M5</version>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-test</artifactId>
        <scope>test</scope>
    </dependency>
</dependencies>

<repositories>
    <repository>
        <id>spring-milestones</id>
        <name>Spring Milestones</name>
        <url>https://repo.spring.io/milestone</url>
    </repository>
</repositories>

application.yml

server:
  port: 885
  servlet:
    context-path: /
  tomcat:
    uri-encoding: UTF-8
    max-threads: 800
    min-spare-threads: 30
logging:
  level:
    com.example: info
    org.springframework: info
spring:
  ai:
    openai:
      api-key: ollama   # field cannot be empty, but no real key is required
      base-url: http://localhost:11434   # local Ollama address
      chat:
        options:
          model: qwen2.5   # name must match the model pulled with <code>ollama pull</code>
          temperature: 0.7
          max-tokens: 2048

Create Chat Controller

package com.badao.ai.controller;

import org.springframework.ai.chat.client.ChatClient;
import org.springframework.web.bind.annotation.*;
import java.util.Map;

@RestController
public class ChatController {
    private final ChatClient chatClient;
    public ChatController(ChatClient.Builder builder) {
        this.chatClient = builder.build();
    }

    @GetMapping("/ai/generate")
    public String generate(@RequestParam(value = "message", defaultValue = "你好") String message) {
        return chatClient.prompt()
                .user(message)
                .call()
                .content();
    }

    @PostMapping("/ai/chat")
    public Map<String, String> chat(@RequestBody Map<String, String> request) {
        String message = request.get("message");
        String response = chatClient.prompt()
                .user(message)
                .call()
                .content();
        return Map.of(
                "message", message,
                "response", response,
                "model", "ollama-qwen2.5");
    }

    @GetMapping("/ai/health")
    public Map<String, String> health() {
        return Map.of(
                "status", "running",
                "model", "ollama-qwen2.5",
                "type", "local");
    }
}

Launch Script and Testing

A batch file builds, runs, or packages the project. Maven commands can be used directly:

# Build (skip tests)
mvn clean install -DskipTests
# Run
mvn spring-boot:run

Test the API:

GET:

http://localhost:885/ai/generate?message=你好，请介绍下你自己

POST (curl):

curl -X POST http://localhost:885/ai/chat \
  -H "Content-Type: application/json" \
  -d '{"message": "请解释机器学习"}'

Health endpoint returns:

{
  "status": "running",
  "model": "ollama-qwen2.5",
  "type": "local"
}

Comparison: Alibaba Cloud Bailei vs Local Ollama

Cost : Bailei offers a free tier then charges; Ollama is completely free.

Network : Bailei requires internet; Ollama runs locally without external network.

Speed : Bailei performance depends on network latency; Ollama performance depends on local hardware.

Privacy : Bailei sends data to the cloud; Ollama keeps data on the local machine.

Configuration : Bailei needs an API key; Ollama only requires a placeholder value (e.g., ollama).

Model options : Bailei provides cloud models such as qwen‑max; Ollama supports local models like qwen2.5, llama3.2, phi3, mistral.

Use case : Bailei is suited for production; Ollama is ideal for development, testing, and offline scenarios.

Common Issues

Q1: Ollama fails to start

# Check if the process is running
tasklist | findstr ollama
# Start manually
ollama serve
# Verify service
curl http://localhost:11434/api/tags

Q2: Slow model download

# Use a domestic mirror or proxy, or pull a smaller model
ollama pull phi3

Q3: Out‑of‑memory

Ollama uses GPU if available; otherwise CPU.

On low‑memory machines choose smaller models (e.g., phi3, qwen2.5:1.5b).

Check model size with ollama list.

Q4: Compilation fails – missing test dependency

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-test</artifactId>
    <scope>test</scope>
</dependency>

Q5: Port 885 occupied

# Change the port in application.yml
server:
  port: 8080

Q6: Model response slow

First request loads the model from disk to memory; subsequent requests are fast.

Adjust max-tokens to limit response length.

Advanced: Streaming Responses

@GetMapping("/ai/stream")
public SseEmitter stream(@RequestParam String message) {
    SseEmitter emitter = new SseEmitter();
    chatClient.prompt()
        .user(message)
        .stream()
        .content()
        .subscribe(
            content -> {
                try { emitter.send(content); }
                catch (IOException e) { emitter.completeWithError(e); }
            },
            error -> emitter.completeWithError(error),
            () -> emitter.complete()
        );
    return emitter;
}

Conclusion

The example demonstrates a complete workflow for integrating a local Ollama LLM into a Spring Boot application using Spring AI. Core steps:

Install Ollama and pull the desired model.

Configure the project to use JDK 17 (required by Spring AI and Spring Boot 3.x).

Create a Spring Boot project and add the spring-ai-openai-spring-boot-starter dependency.

Set base-url to http://localhost:11434 and specify the model name in application.yml.

Implement chat endpoints with ChatClient (simple GET, JSON POST, health check, optional streaming).

Build and run the application, then verify functionality via HTTP requests.

This approach provides a cost‑free, privacy‑preserving, offline‑ready solution for AI prototyping and development.