Quarkus Meets LangChain4j: A Guide to Integrating Large Language Models

Large language models (LLMs) are reshaping software development, and Quarkus now offers a dedicated extension built on the LangChain4j library to make LLM integration straightforward for Java developers.

Overview

When integrating an LLM into a Quarkus application you first need to describe the desired AI behavior in natural language. The extension provides two mechanisms to extend the model’s knowledge: a tool set that the model can invoke, and a document store for retrieval‑augmented generation (RAG).

Tool set – allows the model to perform actions such as sending email, calling REST endpoints, or querying a database. The model decides when to use a tool, which parameters to pass, and how to handle the result.

Document store – lets the model retrieve relevant documents because LLMs have limited context size and poor memory. The extension can fetch documents from a store and prepend them to the model’s prompt.

The diagram below (originally in the blog) illustrates the interaction between the LLM, tools, and the document store.

Show Me the Code!

First add the OpenAI dependency to your project:

<dependency>
  <groupId>io.quarkiverse.langchain4j</groupId>
  <artifactId>quarkus-langchain4j-openai</artifactId>
  <version>0.1.0</version> <!-- Update to the latest version -->
</dependency>

Define an AI service interface using the @RegisterAiService annotation. The annotation works like Quarkus REST client configuration and models the interaction with the LLM.

@RegisterAiService
public interface TriageService {
    @SystemMessage("""
        You are working for a bank, processing reviews about financial products.
        Triage reviews into positive and negative ones, responding with a JSON document.
        """)
    @UserMessage("""
        Your task is to process the review delimited by ---.
        Apply sentiment analysis to determine if it is positive or negative, considering various languages.
        For example:
        - `I love your bank, you are the best!` is a 'POSITIVE' review
        - `J'adore votre banque` is a 'POSITIVE' review
        - `I hate your bank, you are the worst!` is a 'NEGATIVE' review
        Respond with a JSON document containing:
        - the 'evaluation' key set to 'POSITIVE' or 'NEGATIVE'
        - the 'message' key with a polite thank‑you or apology in the review's language.
        ---
        {review}
        ---
        """)
    TriagedReview triage(String review);
}

Inject the service and call it from a JAX‑RS resource:

@Path("/review")
public class ReviewResource {
    @Inject
    TriageService triage;

    record Review(String review) {}

    @POST
    public TriagedReview triage(Review review) {
        return triage.triage(review.review());
    }
}

Benefits of this declarative approach include testability (you can mock the interface), observability (Quarkus metrics can monitor the AI calls), and fault‑tolerance (Quarkus fault‑tolerance annotations handle timeouts and errors).

Tools and Document Loading

Real‑world scenarios often require the model to call custom business logic or fetch data. Declare a bean method as a tool with @Tool:

@ApplicationScoped
public class CustomerRepository implements PanacheRepository<Customer> {
    @Tool("get the customer name for the given customerId")
    public String getCustomerName(long id) {
        return find("id", id).firstResult().name;
    }
}

Expose the allowed tools in the @RegisterAiService annotation:

@RegisterAiService(
    tools = { TransactionRepository.class, CustomerRepository.class },
    chatMemoryProviderSupplier = RegisterAiService.BeanChatMemoryProviderSupplier.class
)
public interface FraudDetectionAi {
    // …
}

Configure chat memory to keep track of multi‑turn interactions:

@RequestScoped
public class ChatMemoryBean implements ChatMemoryProvider {
    Map<Object, ChatMemory> memories = new ConcurrentHashMap<>();

    @Override
    public ChatMemory get(Object memoryId) {
        return memories.computeIfAbsent(memoryId, id ->
            MessageWindowChatMemory.builder()
                .maxMessages(20)
                .id(memoryId)
                .build()
        );
    }

    @PreDestroy
    public void close() { memories.clear(); }
}

Only OpenAI models currently support tool invocation.

Document Store (RAG)

RAG (Retrieval‑Augmented Generation) extends the LLM’s knowledge with your own data. It consists of two steps:

Extraction – documents are embedded and stored as vectors.

Retrieval – before calling the LLM, relevant vectors are fetched and added to the prompt.

The extension provides utilities for both steps. Example of ingesting documents into a Redis store:

@ApplicationScoped
public class IngestorExample {
    /** The embedding store (the database). */
    @Inject
    RedisEmbeddingStore store;

    /** The embedding model that creates vectors. */
    @Inject
    EmbeddingModel embeddingModel;

    public void ingest(List<Document> documents) {
        var ingestor = EmbeddingStoreIngestor.builder()
            .embeddingStore(store)
            .embeddingModel(embeddingModel)
            .documentSplitter(recursive(500, 0))
            .build();
        ingestor.ingest(documents);
    }
}

Create a retriever bean that queries the store:

@ApplicationScoped
public class RetrieverExample implements Retriever<TextSegment> {
    private final EmbeddingStoreRetriever retriever;

    RetrieverExample(RedisEmbeddingStore store, EmbeddingModel model) {
        retriever = EmbeddingStoreRetriever.from(store, model, 20);
    }

    @Override
    public List<TextSegment> findRelevant(String s) {
        return retriever.findRelevant(s);
    }
}

Wire the retriever into the AI service:

@RegisterAiService(
    retrieverSupplier = RegisterAiService.BeanRetrieverSupplier.class
)
public interface MyAiService {
    // …
}

Final Thoughts

The quarkus‑langchain4j 0.1 extension is the first release that brings a Quarkus‑friendly way to work with LLMs. It supports tool invocation, document‑store‑backed RAG, and declarative service definitions that make AI code testable, observable, and resilient. Future versions will expand model support and add more integration patterns.

Thanks to Dmytro Liubarskyi’s work on LangChain4j, this extension could be built, offering native compilation and seamless bean interaction. Developers can now connect any bean as a tool or feed custom data into an LLM, opening many possibilities for AI‑enhanced Java applications.