Skip to content
Connecting LLM, RAG, Agent & MCP: A Java Engineer's Production AI Engineering Guide

Connecting LLM, RAG, Agent & MCP: A Java Engineer's Production AI Engineering Guide

This article uses a real e-commerce customer service request to thread together LLM, RAG, Agent, MCP, vector search, and cosine similarity, providing five-layer architecture diagrams, production-grade Java code, 16 pitfall tables, and 13 interview Q&As to show how to embed uncertain model capabilities into deterministic engineering systems.

Rain's Path to Java Mastery
Rain's Path to Java Mastery
Rain's Path to Java Mastery

Overview

The article addresses a common interview challenge: connecting LLM, RAG, Agent, MCP, vectors, and cosine similarity into a coherent narrative. It uses a real customer query — "Why hasn't my order shipped?" — as a thread to walk through every technical component, Java implementation, and production pitfall.

One-Sentence Metaphor & Five-Layer Architecture

LLM is the engine; Embedding is the coordinate system; Cosine similarity is the ruler; Vector database is the library index; RAG is the external reference room; Tools are hands and feet; MCP is the USB-C port; Skills are the job manual; Agent is the digital employee that plans, acts, and reflects. Java engineers build the car, highway, and gas stations — not the engine.

The five layers:

Model Layer — Provides understanding/generation compute (HTTP/SSE, Spring AI ChatModel).

Context Layer — Turns semantics into computable distance (MilvusVectorStore, PgVectorStore, ES dense_vector).

Capability Layer — Lets model see knowledge and touch systems (LangChain4j ContentRetriever, Spring AI ToolCallback).

Orchestration Layer — Lets model decide next steps (AiServices, Spring StateMachine, ToolProvider).

Engineering Governance Layer — Makes AI stable, secure, cheap (WebFlux+SSE, Redis token bucket, Micrometer, gateway).

Key insight: Layers 1–2 are bought, 3–4 come from frameworks, layer 5 is the enterprise moat.

LLM: A Probabilistic Next-Token Predictor

Three core recognitions:

It generates plausible text, not ground truth.

Knowledge freezes at training cutoff.

Hallucination stems from optimizing fluency over factuality.

LLM has no memory (history must be fed each turn) and no external access. In Java terms, it's just a slow, expensive, occasionally hallucinating third-party REST service. Engineering focus: timeouts, retries, circuit breakers, observability.

Token, Context, Temperature as Budgets

Token ≈ 1–1.5 Chinese chars; cost and latency scale with tokens.

Prompt is code — version, evaluate, regression test.

Context window is a budget, not capacity; allocate for retrieval, history, output.

Temperature 0–0.3 for extraction/classification; 0.7+ for creative writing.

Structured output via BeanOutputConverter + validation + retry.

Vectors & Cosine Similarity: Making Semantics Computable

Why Vectors

Keyword search matches characters; vectors match meaning. Embedding models compress text into high-dim floats (768/1024/1536/3072). Semantically close texts have small angles.

Cosine vs Euclidean vs Dot Product

Cosine ignores magnitude (text length), range [-1,1]. Euclidean suffers from curse of dimensionality. For normalized vectors, cosine == dot product, and ranking equivalence holds, but absolute thresholds are not interchangeable.

Three Mathematical Relations (Interview Essentials)

Normalized: cosine == dot product (saves sqrt/division). FAISS IndexFlatIP with normalized vectors computes cosine.

Normalized: higher cosine ⇔ lower L2 distance (‖a-b‖² = 2(1-cosθ)).

Ranking equivalence ≠ threshold universality — mixing normalized/unnormalized breaks fixed thresholds.

Production Code Details

public final class CosineSimilarity {  private CosineSimilarity() {}  /**  ① double accumulation prevents float precision loss over 1536 dims  ② zero-vector short-circuit returns 0 (avoids NaN poisoning Comparator)  ③ dimension validation ensures same coordinate system  */  public static double cosine(float[] a, float[] b) {    if (a.length != b.length) throw new IllegalArgumentException(...);    double dot = 0.0, na = 0.0, nb = 0.0;    for (int i = 0; i < a.length; i++) {      dot += (double) a[i] * b[i];      na += (double) a[i] * a[i];      nb += (double) b[i] * b[i];    }    if (na == 0.0 || nb == 0.0) return 0.0;    return dot / (Math.sqrt(na) * Math.sqrt(nb));  }  /** L2 normalize at write time; query becomes dot product */  public static float[] l2Normalize(float[] v) { ... }  /** Dot product for normalized vectors (hot path) */  public static double dot(float[] a, float[] b) { ... } }

SIMD acceleration via JDK Vector API (incubator) for offline indexing.

Vector Databases & ANN: Why Not Brute Force

1M vectors × 1024 dims ≈ 2B FLOPs/query; 4GB memory scan per query. ANN trades ~1% recall for orders-of-magnitude speed.

Flat : exact, <100k vectors.

IVF : k-means buckets, search nprobe clusters; scalable, boundary misses.

HNSW : hierarchical navigable small world; best recall/latency, high memory, slow build, delete-unfriendly. Default choice.

PQ : product quantization compresses vectors 10x; precision loss, needs rerank.

HNSW = skip list for long jumps + small-world graph for precision. Target recall@K 95–99%.

Java Selection Guide

Existing ES cluster → ES 8.x dense_vector (BM25 + vector hybrid).

Existing PostgreSQL → pgvector (reuse transactions, backups).

10M+ vectors, low latency → Milvus/Qdrant (tunable HNSW/PQ).

Tests/small data → in-memory Map + cosine.

Don't implement ANN yourself; expertise is in index selection, parameter tuning (nprobe, efSearch), recall@K validation. For pgvector, push computation to DB:

SELECT id, content, 1 - (embedding <=> :qvec) AS score FROM doc_chunk WHERE tenant_id = :tenantId ORDER BY embedding <=> :qvec LIMIT 5;

RAG: Open-Book Exam, Bottleneck Is Retrieval Not Generation

RAG = external knowledge management + retrieval system + generation model. Solves: stale knowledge, private knowledge unknown, hallucination, untraceable answers.

Naive RAG Failure Modes & Fixes

Missed recall ("Apple CEO" vs "Tim Cook") → Hybrid search (Dense+BM25), query rewrite.

Semantic ≠ exact (ID "AG-2024-001") → BM25 exact match fallback.

Context fragmentation (fixed 512-char chunks) → Structure-aware chunking, parent-child chunks.

Multi-hop reasoning fails → Query decomposition, Agentic RAG.

Global questions ("core theme") → GraphRAG / global summary.

Noisy top-K → Cross-encoder rerank, keep top-5–8.

Production RAG formula: Advanced parsing (OCR/layout) + Hybrid search (BM25+Dense, RRF fusion) + Neural rerank (Cross-Encoder) + Agentic routing (dynamic depth) + GraphRAG (global/multi-hop) + Continuous eval (RAGAS/TruLens: recall@k, faithfulness, hallucination rate).

Five Key Design Decisions

Chunking : 500–1000 tokens, 10–20% overlap, structure-aware, parent-child (child for vector match, parent fed to model). Metadata for filtering & citation.

Embedding consistency : Query and doc same model/dimension.

Hybrid search + RRF : score = Σ 1/(k+rank); no score normalization needed.

Rerank : Cross-encoder (Q+D joint) accurate but slow; apply only to top-50.

Generation constraints : Prompt hard-coded "answer only from sources, else say unknown" + forced citation.

Production RAG Code (Spring AI)

@Configuration class RagConfig {  @Bean VectorStore vectorStore(EmbeddingModel em) { return SimpleVectorStore.builder(em).build(); }  @Bean ChatClient chatClient(ChatClient.Builder builder, VectorStore vs) {    return builder      .defaultSystem("You are enterprise knowledge assistant. Answer ONLY from [retrieved docs], cite [source: file#chunk]. If not in docs, reply 'not covered'.")      .defaultAdvisors(QuestionAnswerAdvisor.builder(vs)        .searchRequest(SearchRequest.builder().topK(8).similarityThreshold(0.6).build())        .build())      .build();  }}

Two-stage retrieval + streaming:

public Flux<String> streamAnswer(String query) {  List<Document> docs = retrieve(query); // coarse top-20 → rerank top-5  String context = docs.stream().map(d -> "["+d.getId()+"] "+d.getText()).collect(joining("
---
"));  String sys = "Rules: 1. Only from [docs]... 2. Cite [DOC-xxx] 3. No fabrication
[docs]
"+context;  return chatModel.stream(new Prompt(new SystemMessage(sys), new UserMessage(query)))    .mapNotNull(r -> r.getResult().getOutput().getText()); }

Parallel multi-source retrieval with virtual threads + RRF fusion.

Ingestion: Chunking, Async Embedding, Caching

public void ingest(List<Document> docs) {  List<Document> chunks = new TokenTextSplitter(800, 200, 80, 5000, true).apply(docs); // 800 tokens, 200 overlap  vectorStore.add(chunks); }  @Async public CompletableFuture<List<Float>> embedAsync(String text) {  String key = DigestUtils.md5Hex(text);  List<Float> cached = cache.get(key);  if (cached != null) return CompletableFuture.completedFuture(cached);  List<Float> vec = embeddingClient.embed(text).getResult().getOutput();  cache.put(key, vec, Duration.ofHours(24));  return CompletableFuture.completedFuture(vec); }

Async non-blocking; MD5 cache saves 30%+ embedding cost.

Ops: Sync & Evaluation

Knowledge updates → event-driven (Canal/CDC) + version compare + transactional vector update. Prove improvement: golden eval set (100–500 QA pairs), track Recall@K, Faithfulness, Answer Relevance; gate every chunking/model/threshold change in CI. "RAG without auto-eval is faith, not engineering."

Tool Calling: Model Names, Java Executes

Model outputs "I need to call queryOrderStatus with orderId=SO2026…"; Java executes (auth, rate-limit, audit), returns result to model for final phrasing. All real business actions stay in Java code — safety boundary.

@Tool: Description Is Model-Facing API Doc

@Tool("Query order status & logistics by order number. Read-only.") public String queryOrderStatus(@P("Order ID like SO202610150001") String orderId) {  if (!orderId.matches("SO\d{8,}")) return "Invalid format, e.g. SO202610150001";  return JsonUtils.toJson(orderService.queryStatus(orderId)); }

Defensive validation returns readable error text so model can self-correct. Retry loop with feedback injection:

public ToolResult executeWithRetry(ToolCall call, int maxRetries) {  for (int i=0; i<maxRetries; i++) {    try { Tool tool = registry.get(call.name()); validateParams(tool.getInputSchema(), call.arguments()); return tool.execute(call.arguments());    } catch (ToolValidationException e) { call.setFeedback("Param error: "+e.getMessage()); }  }  return new ToolResult("error", "Multiple retries failed, rephrase."); }

Runtime Tool Convergence: 28 → 5–8 Tools

Problems with full tool set: token explosion (800→4200), model confusion ("order" appears 6x), permission leakage. Solution: ToolProvider dynamically serves tools per tenant/role/intent.

@Bean ToolProvider scopedToolProvider(OrderTools ot, RefundTools rt) {  return req -> {    UserContext ctx = UserContextHolder.current();    ToolProviderResult.Builder b = ToolProviderResult.builder();    if (ctx.hasPerm("order:read") && looksLikeOrderQ(req.userMessage()))      b.add(toolSpec("queryOrderStatus"), executor(ot));    if (ctx.hasPerm("order:refund")) b.add(toolSpec("applyRefund"), executor(rt));    return b.build();  }; }

Result: tokens 4200→1200, mis-invocation dropped significantly.

Skills: Don't Stuff SOPs into System Prompt

Skills = task-type playbooks (SKILL.md + scripts + refs) loaded on demand via progressive disclosure .

Load time : System Prompt always resident, consumes budget; Skills on-demand, zero token when unused.

Scale : System Prompt ~10 processes blow context; Skills 100 skills, only summary overhead.

Maintainability : System Prompt one change breaks all, no versioning; Skills one folder per responsibility, independent evolution.

Rules: SKILL.md ≤500 lines; refs only one level deep; >100-line refs need TOC. Skill definition YAML includes name, version, description, prompt template, tools (MCP), inputs/outputs, eval dataset.

Clarifications

Skills vs MCP: Skills = "know how"; MCP = "can touch". Skill steps call MCP tools.

Skills vs Agent: Skill = passive tool (input→output); Agent = active planner (loops, multi-skill).

Skills vs Prompt: Prompt = "this task"; Skills = "this class of tasks".

Trend: Agent Skills becoming open standard (Claude Code, Cursor, Copilot, Codex, Gemini CLI). JVM abstraction emerging (LangChain4j dev.langchain4j.skills).

MCP: AI's USB-C Port

MCP (Model Context Protocol, Anthropic 2024) standardizes how AI apps securely access external data/tools. HTTP unified Web APIs; MCP unifies tool discovery (tools/list) and invocation (tools/call). Solves M×N integration hell.

Three Primitives (JSON-RPC 2.0)

Tools (model-controlled): executable functions (query order, write file).

Resources (app-controlled): context data (file content, config, git history).

Prompts (user-controlled): predefined templates (slash commands, menu items).

Plus Sampling: server requests LLM inference for recursive tasks.

Transport Evolution

STDIO: local process, no network.

SSE+HTTP: remote, but long-lived connections (~80KB/conn), flaky, deprecated in Spring AI 2.0.

Streamable HTTP (preferred): request-response + streaming in one HTTP semantics; works with gateways, LB, auth.

Stateless Streamable HTTP : <5KB/req, serverless-friendly.

2026-07-28 Spec v5: fully stateless (no handshake/session), versioned extensions (MCP Apps, Tasks, server/discover), production hardening (OAuth2/OIDC, JSON Schema 2020-12, W3C Trace Context).

MCP vs Function Calling

Function Calling = model-side verb (decide which function); MCP = engineering-side USB-C (standardized tool supply). Complementary.

Why Java Engineers Must Care

Enterprise core systems (order, payment, risk, ERP, supply chain) are Java. MCP exposes them to AI with zero business logic changes.

<dependency>  <groupId>org.springframework.ai</groupId>  <artifactId>spring-ai-starter-mcp-server-webmvc</artifactId></dependency>  # application.yml  spring.ai.mcp.server.name=order-mcp-server  spring.ai.mcp.server.version=1.0.0  spring.ai.mcp.server.protocol=STREAMABLE  spring.ai.mcp.server.streamable-http.mcp-endpoint=/api/mcp  @SpringBootApplication class OrderMcpServerApplication {  @Bean ToolCallbackProvider orderTools(OrderService svc) {    return MethodToolCallbackProvider.builder().toolObjects(svc).build();  }}

Client side: ToolProvider aggregates multiple MCP servers, dynamic discovery, hot-plug without restart.

Security Red Lines & Write Discipline

HTTP MCP endpoints default to NO AUTH — naked exposure = naked run. Must front with OAuth2/OIDC, gateway rate-limit, tool-level permissions, audit logs. Write tools require: permission check, human-in-the-loop (HITL) confirmation, idempotency.

@Tool("Submit expense, write op, needs user confirmation") public ExpenseResult submitExpense(SubmitExpenseRequest req) {  return idempotentService.execute(req.idempotentKey(), () -> expenseService.submit(req)); }

Idempotency key + business outbox + unique index; tool whitelist, read-only first, high-risk writes default closed.

Agent: From Thinking to Delivering

Agent = LLM (brain) + Planning + Memory + Tools. Core: autonomously decides next step. Unlike LLM's Q&A, Agent = "given goal, self-completes".

ReAct Loop: Thought → Action → Observation

while (not done) {  think nextStep;  select tool;  execute tool.call(params);  observe toolResult;  update context; }

Resembles Java's Chain of Responsibility + Strategy.

Three-Tier Paradigm

Inner: ReAct — single agent reasoning; termination by model; risks: infinite loops, premature stop.

Middle: Loop Engineering — graph state machine / multi-agent loops; structured rules + state checks; no fault tolerance/security.

Outer: Harness Engineering — guardrails, brakes, ops foundation; global policies.

Production Multi-Agent = Harness (outer) + Loop Engineering (middle) + ReAct (inner).

ReAct with Four Guards

public Answer run(String input, int maxSteps, long budgetMs) {  List<Message> msgs = ...;  long deadline = System.currentTimeMillis() + budgetMs;  for (int step=0; step<maxSteps; step++) {    if (System.currentTimeMillis() > deadline) return new Answer("Timeout, returning partial.");    AiMessage ai = chatModel.generate(msgs).content();    if (!ai.hasToolExecutionRequests()) return new Answer(ai.text());    msgs.add(ai);    for (ToolExecutionRequest req : ai.toolExecutionRequests()) {      String result;      try { result = toolExecutor.execute(req, tenantId()); result = truncate(result, 2000); } catch (Exception e) { result = "TOOL_ERROR: "+e.getMessage()+", try another way."; }      msgs.add(ToolExecutionResultMessage.from(req, result));    }    msgs = contextCompactor.compact(msgs, TOKEN_BUDGET);  }  return new Answer("Step limit exceeded. Break problem smaller."); }

Extra: tool call deduplication (same tool+params → short-circuit), response truncation (10k-line SQL blows context). Context compaction: summarize old half, keep recent turns.

Framework: AiServices Minimal Wiring

@Bean CustomerServiceAgent customerServiceAgent(ChatLanguageModel lm, OrderQuerySkill os, LogisticsQuerySkill ls) {  return AiServices.builder(CustomerServiceAgent.class)    .chatLanguageModel(lm)    .tools(os, ls)    .chatMemory(MessageWindowChatMemory.withMaxMessages(20))    .maxIterations(5)    .timeout(Duration.ofSeconds(30))    .build(); }

Short-term memory via sliding window; long-term via vector store keyed by user profile.

Agent vs Workflow

Workflow = fixed path; Agent = model-decided path. Use Workflow for deterministic flows (stable, cheap). Agent only when path truly unpredictable — cost/stability tradeoff, not tech showoff. Evaluate: task completion rate, avg steps, tool accuracy, latency, cost/token, trace replay.

End-to-End: One Customer Request

User: "Why hasn't order SO202610280001 shipped?" Flow: Skill determines process → LLM intent recognition → Tool decides action → MCP connects legacy systems → RAG supplies biz knowledge → Vector+cosine handles recall → Engineering governance ensures stability/security/cost. Agent decides what to query, call, answer — not dumping all knowledge into LLM.

Same skeleton works for travel expense assistant (RAG for policy, MCP for reimbursement system) and ops assistant (RAG for runbooks, MCP for monitoring).

Engineering Governance: Java's Home Turf

Interviewer asks: "1000 concurrent users, how does Spring Boot handle it?" Three shields:

@GetMapping(value="/chat/stream", produces=TEXT_EVENT_STREAM_VALUE) public Flux<String> stream(@RequestParam String q, @RequestHeader("X-Tenant-Id") String tid) {  return redisRateLimiter.acquire(tid)    .filter(RateLimitResult::allowed)    .switchIfEmpty(Flux.error(new BizException("TOO_MANY_REQUESTS")))    .flatMapMany(ok -> chatClient.prompt().user(q).stream().content()      .timeout(Duration.ofSeconds(30))      .onErrorResume(e -> Flux.just("Model busy, returning raw KB search."))); }

Key metrics: streaming cuts first-token latency 8s→500ms; semantic cache skips full RAG chain (40–70% cost reduction); model routing (small model for intent/format, flagship for reasoning) saves 40–60%; unified ChatModel abstraction makes model swap a config change.

Other must-haves: multi-tenant isolation (metadata tenantId/acl/version, pre-filter at retrieval, vector partition per tenant); observability (full trace: prompt/token/latency/tool calls, Micrometer+OpenTelemetry+W3C Trace Context across MCP); three-tier model fallback (multi-key/region retry → backup model → KB raw output); data security (gateway PII scrubbing, sensitive fields never reach model, private deployment option).

16 Pitfall Table (Condensed)

Hallucination : hard prompt constraints + forced citation + low temp + threshold refuse + human fallback. RAG reduces probability, cannot eliminate.

Retrieval inaccuracy : hybrid (Dense+BM25→RRF) → Cross-Encoder rerank → top-5–8; Multi-Query rewrite.

Chunk semantic break : structure-aware + overlap + parent-child + metadata enrichment.

Context explosion/cost : sliding window + history summary + tool result truncation + Skills lazy-load + semantic cache + model routing + stable prefix for KV cache.

Agent runaway : hard step limit + time/token budget fuse + call deduplication + state machine instead of while(true).

Tool mis-invocation : better @Tool descriptions + few-shot + JSON Schema validation + error feedback retry + runtime tool convergence.

Tool security/escalation : permission push-down to retrieval + tool whitelist + least privilege + audit + HITL for high-risk.

MCP production : OAuth2/OIDC front + gateway limit + tool perms + audit; prefer Streamable/Stateless HTTP.

Concurrency/latency : WebFlux+SSE + virtual threads + parallel retrieval + tenant token bucket + timeout fallback to KB.

Multi-tenant leakage : metadata tenantId pre-filter + partition/collection isolation.

Vector staleness : CDC/binlog incremental sync + version compare + scheduled full reconciliation + force-refresh API.

Write idempotency : idempotencyKey + outbox + unique index + double-confirm.

Unmeasurable effectiveness : golden QA set, quantify recall@k/faithfulness/hallucination; Prompt/chunk changes must pass regression gate.

Brittle stack : isolate volatility behind adapters: unified ChatModel abstraction, MCP standard protocol, Prompt versioning.

Multi-turn context loss : query rewrite from history to resolve references before retrieval.

Data security : gateway desensitization + permission filter + private deployment.

13 Rapid-Fire Interview Answers

Cosine not Euclidean: text semantics care about direction, not length; magnitude adds noise; normalized they rank same.

Normalized → dot product: saves two sqrts + one division on hot path, direct P99 gain.

Vector DB vs ES: vector DB specializes ANN/high-dim; ES shines at unified text+vector, hybrid search simplest. Choose by ops cost & scale.

1M context still need RAG: long context is budget not replacement; more docs = more noise, higher cost/latency; RAG filters first, long context deep-reads.

RAG vs Fine-tune: frequent updates + citation needed → RAG; fixed style/format + prompt compression → fine-tune. Prod often combines: small model fine-tuned + RAG + Agent tools.

Agent vs Workflow: Workflow fixed path, Agent model-decided path. Deterministic → Workflow. Don't use Agent for sake of it.

MCP vs Function Calling: FC = model decides which function; MCP = engineering standardizes tool supply. MCP enables write-once, reuse everywhere.

Skills vs Prompt: Prompt = this task; Skills = this class of tasks. Progressive disclosure = many skills, low context cost.

Why Rerank: vector retrieval = bi-encoder (independent encoding), no deep interaction; Cross-encoder = joint encoding, accurate but slow → only rerank top-50.

Vector dimension: common 768/1024/1536/3072; higher = finer but linear cost; enterprise often 1024 + PQ quantization.

Model down: three-tier fallback — multi-key/region retry → backup model → KB raw output. Availability via architecture, not model.

Prove RAG works: golden QA set, measure recall@k/faithfulness/hallucination; Prompt/chunk changes must pass CI regression gate.

Java vs Python for AI: Python better at model layer; Java stronger at engineering layer — virtual threads for concurrency, strong typing fits structured output, mature microservice/resilience/observability stack, and 90% of enterprise systems already Java — MCP Server plugs directly, no rewrite.

Closing: Cheat Sheet & Three Lessons

Concept cheat sheet with one-liner essence, key params, Java touchpoints (LLM, Token, Embedding, Cosine, Vector DB, RAG, Tool, Agent, MCP, Skills).

Treat LLM as unreliable third-party service — timeout, retry, circuit breaker, fallback, observability; same old discipline, now for probabilistic output.

Anchor answers in facts via RAG, turn effectiveness into numbers via eval sets — these two steps graduate demo to production.

Skills, MCP, Agent are OO thinking in AI era — encapsulation, abstraction, standardized interfaces. Grasp this, new buzzwords won't faze you.

AI application essence: put "uncertain model capabilities" into "deterministic engineering systems". That's exactly what Java engineers excel at. Few build engines; most build cars, highways, gas stations.
Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

JavaLLMMCPRAGVector DatabaseAgentSpring AIAI EngineeringCosine SimilarityTool CallingSkillsProduction RAG
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.