How to Build a Retrieval‑Augmented Generation (RAG) System with Langchain4j and Ollama 3

Retrieval‑Augmented Generation (RAG) combines external document retrieval with language‑model generation, reducing hallucinations and injecting up‑to‑date domain knowledge. The article first discusses RAG’s importance for accuracy and contextual relevance.

Key Components in Langchain4j + Ollama3

EmbeddingStore : Manages vectors extracted from documents.

EmbeddingStoreIngestor : Ingests documents and creates their embeddings.

OllamaEmbeddingModel : Generates embeddings from text for retrieval.

OllamaLanguageModel : Produces precise, context‑aware responses using retrieved data.

Step‑by‑Step Example

1. Ensure the Ollama 3 engine is running.

2. Add the Maven dependency:

<dependency>
  <groupId>dev.langchain4j</groupId>
  <artifactId>langchain4j-ollama</artifactId>
  <version>0.33.0</version>
</dependency>

3. Create a text file (e.g., dictionary.txt) containing a fantasy creature description.

4. Load and split the document, then ingest it into an in‑memory embedding store:

EmbeddingModel embeddingModel = OllamaEmbeddingModel.builder()
    .baseUrl("http://localhost:11434")
    .modelName("llama3")
    .build();

EmbeddingStore<Embedding> embeddingStore = new InMemoryEmbeddingStore();
URL fileUrl = RAGIngestor.class.getResource("/dictionary.txt");
Path path = Paths.get(fileUrl.toURI());
Document document = FileSystemDocumentLoader.loadDocument(path, new TextDocumentParser());
DocumentSplitter splitter = DocumentSplitters.recursive(600, 0);
EmbeddingStoreIngestor ingestor = EmbeddingStoreIngestor.builder()
    .documentSplitter(splitter)
    .embeddingModel(embeddingModel)
    .embeddingStore(embeddingStore)
    .build();
ingestor.ingest(document);

5. Generate an embedding for the user query and retrieve the most relevant chunk:

Embedding queryEmbedding = embeddingModel.embed("What is the Shadowmire ?").content();
List<EmbeddingMatch<Embedding>> relevant = embeddingStore.findRelevant(queryEmbedding, 1);
EmbeddingMatch<Embedding> match = relevant.get(0);
String information = match.embedded().text();

6. Build a prompt that injects the retrieved information and invoke the language model:

Prompt prompt = PromptTemplate.from("""
    Tell me about {{name}}? Use the following information to answer the question:
    {{information}}
""")
    .apply(Map.of("name", "Shadowmire", "information", information));

OllamaLanguageModel model = OllamaLanguageModel.builder()
    .baseUrl("http://localhost:11434")
    .modelName("llama3")
    .timeout(Duration.ofSeconds(900))
    .build();
String answer = model.generate(prompt).content();
System.out.println("Answer:" + answer);

The code comments explain each step: initializing the embedding model, setting up the in‑memory store, loading and parsing the document, splitting it, ingesting embeddings, creating a query embedding, performing similarity search, preparing the prompt, initializing the language model, and finally generating the answer.

Running the program yields a response that incorporates the custom creature description, demonstrating how RAG with Langchain4j and Ollama 3 can enhance NLP tasks.

Conclusion

By merging retrieval‑based and generative models, the Langchain4j‑Ollama 3 RAG pipeline offers a powerful method to improve accuracy and relevance in natural‑language applications, and the presented framework can be further customized for different use cases and datasets.