Building Enterprise‑Grade AI Agents in Java in 3 Days

Enterprise Agent vs. Demo

Data source : Demo uses hard‑coded or public APIs; Enterprise agents query internal databases and a vector knowledge base.

Knowledge base : Demo has none; Enterprise uses a vector DB with document retrieval.

Permission control : Demo none; Enterprise filters results by user identity.

Conversation storage : Demo keeps it in memory; Enterprise persists it in Redis or a relational DB.

Observability : Demo logs to console; Enterprise provides full trace chains.

Deployment : Demo runs locally; Enterprise is containerised with Docker/K8s and Spring Boot.

Project: Internal Knowledge‑Base Assistant

The assistant answers internal company questions, retrieves technical documents, fetches employee ticket records, and enforces role‑based data visibility.

Step 1: Connect a Vector Store for RAG

RAG (Retrieval‑Augmented Generation) enables the model to answer questions based on internal documents.

# application.yml
spring:
  datasource:
    url: jdbc:postgresql://localhost:5432/enterprise_ai
    username: ${DB_USER}
    password: ${DB_PASS}
  ai:
    vectorstore:
      pgvector:
        index-type: HNSW
        distance-type: COSINE_DISTANCE
        dimensions: 1536  # text‑embedding‑3‑small dimension

Ingest documents (one‑off script or scheduled job):

@Service
public class KnowledgeBaseService {
    @Autowired
    private VectorStore vectorStore;
    @Autowired
    private TokenTextSplitter textSplitter;

    // Split and store document chunks
    public void ingestDocument(String content, String docTitle, String category) {
        Document document = new Document(
                content,
                Map.of(
                        "title", docTitle,
                        "category", category,
                        "createdAt", LocalDate.now().toString()
                )
        );
        List<Document> chunks = textSplitter.apply(List.of(document));
        vectorStore.add(chunks);
        log.info("Document [{}] ingested, split into {} chunks", docTitle, chunks.size());
    }
}

Querying uses QuestionAnswerAdvisor to perform similarity search automatically:

@Service
public class EnterpriseAgentService {
    @Autowired
    private ChatClient chatClient;
    @Autowired
    private VectorStore vectorStore;

    public String ask(String question, UserContext userCtx) {
        SearchRequest searchRequest = SearchRequest.defaults()
                .withTopK(5)
                .withSimilarityThreshold(0.7)
                .withFilterExpression("category in " + userCtx.getAllowedCategories());
        return chatClient.prompt()
                .system(buildSystemPrompt(userCtx))
                .user(question)
                .advisors(new QuestionAnswerAdvisor(vectorStore, searchRequest))
                .call()
                .content();
    }
}

Step 2: Wrap Internal Database Access as Tools

Expose JPA/MyBatis queries as Spring‑AI @Tool methods so the agent can fetch real‑time data.

@Component
public class EnterpriseDataTools {
    @Autowired
    private TicketRepository ticketRepository;
    @Autowired
    private SecurityContextHolder securityContextHolder;

    @Tool(description = "Query current user's tickets, optional status filter. Values: OPEN/IN_PROGRESS/CLOSED")
    public List<TicketSummary> getMyTickets(String status) {
        String currentUser = getCurrentUser();
        TicketStatus ticketStatus = status != null ? TicketStatus.valueOf(status) : null;
        return ticketRepository.findByAssigneeAndStatus(currentUser, ticketStatus)
                .stream()
                .map(TicketSummary::from)
                .collect(Collectors.toList());
    }

    @Tool(description = "Get ticket detail and processing history by ticket ID")
    public TicketDetail getTicketDetail(String ticketId) {
        String currentUser = getCurrentUser();
        Ticket ticket = ticketRepository.findById(ticketId)
                .orElseThrow(() -> new TicketNotFoundException(ticketId));
        if (!ticket.isAccessibleBy(currentUser)) {
            throw new AccessDeniedException("No permission to view this ticket");
        }
        return TicketDetail.from(ticket);
    }

    private String getCurrentUser() {
        return SecurityContextHolder.getContext()
                .getAuthentication()
                .getName();
    }
}

Permission checks live inside the tool, exactly as in a regular service.

Step 3: Persist Conversations with Redis

Replace the default in‑memory chat memory with a Redis‑backed implementation.

@Configuration
public class ChatConfig {
    @Autowired
    private RedisTemplate<String, Object> redisTemplate;

    @Bean
    public ChatMemory chatMemory() {
        // Replace default InMemoryChatMemory
        return new RedisChatMemory(redisTemplate, Duration.ofHours(24));
    }
}

public class RedisChatMemory implements ChatMemory {
    private final RedisTemplate<String, Object> redisTemplate;
    private final Duration ttl;
    private static final String KEY_PREFIX = "chat:memory:";

    @Override
    public void add(String conversationId, List<Message> messages) {
        String key = KEY_PREFIX + conversationId;
        // Serialize each message into a Redis List
        messages.forEach(msg -> redisTemplate.opsForList().rightPush(key, serializeMessage(msg)));
        redisTemplate.expire(key, ttl);
    }

    @Override
    public List<Message> get(String conversationId, int lastN) {
        String key = KEY_PREFIX + conversationId;
        long size = redisTemplate.opsForList().size(key);
        long start = Math.max(0, size - lastN);
        return redisTemplate.opsForList().range(key, start, -1)
                .stream()
                .map(this::deserializeMessage)
                .collect(Collectors.toList());
    }
    // serializeMessage / deserializeMessage omitted for brevity
}

Step 4: Add Observability

Spring AI integrates Micrometer; enable tracing and record prompts/completions.

# application.yml
management:
  tracing:
    enabled: true
    sampling:
      probability: 1.0
spring:
  ai:
    chat:
      observations:
        include-prompt: true   # record input prompt
        include-completion: true # record model output

Optional LangSmith integration for dedicated AI tracing:

@Configuration
public class LangSmithConfig {
    @Bean
    public ChatClientCustomizer langSmithCustomizer() {
        return builder -> builder.defaultAdvisors(new SimpleLoggerAdvisor()); // basic logging
    }
}

With Micrometer + Prometheus + Grafana you can monitor latency distribution, token consumption trends, tool‑call success rate, and most frequent question types.

Step 5: Containerised Deployment

# Dockerfile
FROM openjdk:21-jre-slim
WORKDIR /app
COPY target/enterprise-ai-0.0.1-SNAPSHOT.jar app.jar
ENV OPENAI_API_KEY=""
ENV DB_PASSWORD=""
EXPOSE 8080
ENTRYPOINT ["java","-jar","app.jar"]

# docker-compose.yml
version: '3.8'
services:
  app:
    build: .
    ports:
      - "8080:8080"
    environment:
      - OPENAI_API_KEY=${OPENAI_API_KEY}
      - SPRING_DATASOURCE_URL=jdbc:postgresql://db:5432/enterprise_ai
    depends_on:
      - db
      - redis
  db:
    image: pgvector/pgvector:pg16
    environment:
      POSTGRES_DB: enterprise_ai
      POSTGRES_PASSWORD: ${DB_PASSWORD}
  redis:
    image: redis:7-alpine

Start the stack with a single command:

docker-compose up -d

Key Takeaways

Permission control uses Spring Security unchanged.

Database access uses Spring Data JPA / MyBatis unchanged.

Conversation persistence uses Redis for durability.

Deployment and operations remain Docker/K8s based.

Monitoring and alerts use Micrometer + Prometheus unchanged.

AI capabilities are additive, not a replacement. Spring AI plugs large‑model intelligence into the existing Java stack without rebuilding the ecosystem.