Spring AI Day 5: Enabling a Multimodal ChatClient to Process Text and Images

1. Multimodal: Text and Image together

Multimodal means a model can accept text, image, audio, and other inputs. Spring AI represents media attachments with the Media type attached to a UserMessage.

⚠️ DeepSeek is a pure‑text model and does not support image input. To work with multimodal data you need a visual model such as OpenAI gpt-4o , Alibaba qwen-vl , Zhipu GLM-4V , or a local Ollama llava model. Switching models follows the same starter and API pattern as Day 1; the .media(...) code remains unchanged, illustrating the abstraction value of Spring AI.

2. Media API: One image + one sentence

Place an image under src/main/resources and call .media(...) together with the prompt text.

String response = ChatClient.create(chatModel).prompt()
        .user(u -> u
                .text("这张图里有什么？")
                // specify image type + resource; image placed in resources
                .media(MimeTypeUtils.IMAGE_PNG, new ClassPathResource("/fruit.png")))
        .call()
        .content(); // Output: 图中是一个铁丝水果篮，里面有香蕉、苹果……

MimeTypeUtils

comes from org.springframework.util and ClassPathResource from org.springframework.core.io. Current mainstream visual models support image/jpeg, image/png, image/gif, and image/webp.

3. Using the low‑level ChatModel

If you work directly with ChatModel, you must build a UserMessage that includes a Media attachment.

var imageResource = new ClassPathResource("/chart.png");
UserMessage userMessage = UserMessage.builder()
        .text("总结这张图表的关键数据")
        .media(new Media(MimeTypeUtils.IMAGE_PNG, imageResource))
        .build();

ChatResponse response = chatModel.call(new Prompt(userMessage));

A single message can carry multiple images, and the model will analyse them jointly.

4. Real‑world scenarios

Invoice / document OCR : upload an image, let the model extract key fields and directly produce an Invoice object.

Chart understanding : feed a data‑chart image and ask the model to summarise trends.

Product image moderation : determine whether an image complies with policy or contains prohibited content.

Screenshot troubleshooting : a user sends an error screenshot; the model reads the image to locate the problem.

Practical tip : In the invoice scenario, combine .media() with .entity(Invoice.class) so a single call returns a structured Invoice object, eliminating the traditional OCR‑plus‑regex pipeline.

5. Day 5 recap

Multimodal – model processes text + image.

Media – Spring AI’s unified type for media attachments. .media(MimeType, Resource) – attaches an image to a ChatClient request.

Visual models – DeepSeek does not support images; switch to gpt‑4o, qwen‑vl, GLM‑4V, or llava.

Change model, keep code – the Media code stays unchanged when swapping starters.