Skip to content
Beyond API Calls: Master LLM Internals, KV Cache, RAG & Function Calling with Java

Beyond API Calls: Master LLM Internals, KV Cache, RAG & Function Calling with Java

This comprehensive guide teaches backend engineers LLM fundamentals—from token prediction and attention mechanisms to KV Cache optimization—and provides production-ready Java code for token budgeting, SSE streaming, RAG, function calling, and agent loops, plus 14 real-world pitfalls and interview preparation.

Rain's Path to Java Mastery
Rain's Path to Java Mastery
Rain's Path to Java Mastery

The article opens by reframing LLMs as "text continuation machines" that predict the next token's probability distribution, sample one, append it, and repeat until an end token. This core insight connects hallucination, streaming, KV Cache, RAG, and function calling.

1. LLM Training Pipeline

Three stages turn a base model into a chat product:

Pre-training : Unsupervised next-token prediction on trillion-token corpora → learns grammar, logic, world knowledge.

SFT (Supervised Fine-Tuning) : Trains on human-written instruction–response pairs → learns instruction following and dialogue format.

RLHF / DPO : Aligns with human preferences (helpful, harmless, honest). DPO skips the explicit reward model and PPO. Alignment creates a "sycophancy" bias —models prefer inventing answers over saying "I don't know" because training rewards answers.

2. Forward Pass: Token → Embedding → Attention → Sampling

2.1 Tokenization & Embedding

Text is split into tokens (BPE). Each token maps to a high-dimensional vector. jtokkit is used for exact token counting in Java; Chinese ≈ 0.6–1.5 tokens/char, English ≈ 1.3 tokens/word. UUIDs, Base64, signed URLs are "token assassins" (a signed URL can consume 300+ tokens).

2.2 Self-Attention (Scaled Dot-Product)

Formula: Attention(Q,K,V) = softmax(Q·Kᵀ / √d_k) · V. Analogy: weighted HashMap lookup —Query = search term, Key = book labels, Value = book content. Steps: Q·Kᵀ: every token asks every other token for relevance.

Divide by √d_k: prevents dot-product variance from pushing softmax into saturation (gradient vanishing). softmax: converts relevance to probability weights.

Multiply by V: weighted sum of values.

Example: generating "它" in "小明把苹果吃了,它很开心" attends 0.62 to "小明", 0.28 to "苹果", 0.10 to "吃了".

2.3 Critical Design Choices

Causal Mask : future positions set to -inf so decoder only sees left context. This is why GPT generates and BERT only understands.

Multi-Head : parallel subspaces capture syntax, coreference, entity relations.

GQA (Grouped Query Attention) : multiple Q heads share K/V heads → cuts KV memory several-fold with negligible quality loss ; used in Llama3, Qwen.

2.4 Sampling

Temperature sharpens/flattens the distribution; Top-p (nucleus) truncates the long tail adaptively; Top-k keeps fixed k candidates; Penalty suppresses repetition. Production rule: tune only one (usually top_p) . Parameter cheat sheet:

Customer service: T=0.1–0.3, top_p=0.9

Code generation: T=0.2–0.4, top_p=0.9

Marketing copy: T=0.7–1.0, top_p=0.95

Data extraction: T=0.0–0.2, top_p=0.9

3. Performance: Prefill, Decode & KV Cache

Inference splits into two phases:

Prefill : processes entire prompt in parallel → compute-bound → determines TTFT (Time To First Token).

Decode : generates tokens one by one, reusing cached K/V → memory-bandwidth-bound → determines TPOT (Time Per Output Token).

KV Cache stores past K/V so each new token only computes its own Q and attends to cached K/V. Without it, complexity is O(n²); with it, O(n). Memory formula: 2 × layers × seq_len × hidden_dim × bytes_per_float. In long-context scenarios KV Cache can exceed 70% of inference memory .

Optimization chain targeting KV Cache:

GQA/MQA : reduce K/V heads at source.

KV Quantization : lower precision.

PagedAttention (vLLM): OS-style virtual memory for KV blocks → eliminates fragmentation from reserving contiguous memory for max sequence length.

Sliding Window / Sparse Attention : reduce O(n²) compute.

Continuous Batching : dynamic request packing for higher throughput.

Speculative Decoding : small model drafts, large model verifies → 3–5× speedup observed .

Prefix Cache : reuse common system prefixes → cost drops ~90% .

Streaming uses SSE (single-direction, HTTP, browser-native) rather than WebSocket.

4. Terminology Reference (25+ Terms)

Token / BPE : model's minimal processing unit; Chinese ~1 char ≈ 0.6–1.5 tokens, English ~1 word ≈ 1.3 tokens; UUIDs, Base64, signed URLs are "token assassins".

Embedding : maps token/text to high-dim vector; semantic similarity ≈ vector proximity; embedding model and generation model are separate.

Transformer / Decoder-only : universal backbone; self-attention core; solves long-range dependency and enables parallel compute; causal mask enables autoregressive generation.

Self-Attention / QKV : Q=query, K=key, V=value; weighted sum via Q·K similarity; scaling by √d_k prevents softmax saturation.

RoPE : rotary position encoding injects order into vectors; extrapolation ability determines long-context scalability.

Causal Mask : decoder sees only left context; fundamental reason GPT generates, BERT only understands.

Multi-head / GQA : multiple heads = parallel subspaces for syntax, coreference, entities; GQA shares K/V heads across Q heads → memory reduction.

Context Window : total tokens (input+output) per forward pass; model has no memory—history resent each turn; larger window ≠ better (attention dilution, linear cost growth).

KV Cache : caches historical K/V to avoid recomputation; reduces decode from O(n²) to O(n); cost = memory grows linearly with sequence length.

Prefill / Decode : prefill parallel prompt processing (TTFT), decode sequential token generation (TPOT).

PagedAttention : virtual memory for KV blocks (fixed-size pages) → solves contiguous memory reservation waste.

Temperature / Top-p / Top-k : temperature reshapes distribution sharpness; top-p adaptively truncates cumulative probability tail; top-k keeps fixed k candidates; low temp for stability, high for creativity; tune only one.

Penalty : frequency/presence penalty suppresses repetition.

Hallucination : model fabricates plausible but false answers when precise facts absent in parameters; root cause: probability completion ≠ factual retrieval.

Prompt / Context Engineering : all role, background, examples, constraints fed to model; 2026 term "Context Engineering" emphasizes what, how much, and order in window.

RAG : retrieval-augmented generation = open-book exam; enterprise first choice for knowledge updates without weight changes; lowers but cannot eliminate hallucination.

SFT / LoRA : domain adaptation; LoRA trains two low-rank matrices (≈0.1% params), freezes base weights; teaches style/format, not facts (catastrophic forgetting).

Quantization : FP16 → INT8/INT4; halves memory, boosts throughput, minor accuracy drop; must evaluate on own test set.

MoE : mixture of experts; multiple FFNs per layer, each token activates 1–2; total params large but active params small; memory heavy, compute lighter; load balancing challenge.

RLHF / DPO / Alignment : makes model helpful/harmless/honest; DPO removes explicit reward model and PPO; alignment creates sycophancy bias (prefers answering over "I don't know").

Reasoning Models / Test-time Compute : longer chain-of-thought trades latency and token cost for accuracy; not suitable for all scenarios.

Function Calling : model emits structured JSON (function name + args); execution, validation, permissions, idempotency stay in backend; "model = brain, Java = hands".

MCP : Model Context Protocol = unified tool/resource integration protocol (USB-C for AI); turns N×M adapters into N+M.

Agent / ReAct : loop of Thought → Action(tool) → Observation → … → Final; stability = backend timeouts, idempotency, retries, state persistence.

SSE / TTFT / TPOT : server-sent events, time-to-first-token, time-per-output-token; SSE unidirectional, HTTP, browser-native, better than WebSocket for chat.

5. Java Implementation of Core Primitives

5.1 Scaled Dot-Product Attention (Java)

Hand-written SelfAttention class with numerically stable softmax (subtract max before exp), causal masking ( Float.NEGATIVE_INFINITY), and explicit loops matching the math. Extensible to multi-head via head splitting and parallel threads.

5.2 Multi-Head Causal Self-Attention (PyTorch)

Interview-standard implementation: fused qkv linear layer, view+transpose for head splitting, masked_fill(-inf) for causal mask (multiplying by zero would produce uniform softmax).

5.3 KV Cache Incremental Decoding

KVCache

class stores keys/values in lists; step() computes only the new token's Q, reads all cached K/V, returns output. bytes() method calculates memory footprint: 2L * layers * seq_len * dim * bytesPerFloat.

5.4 Temperature + Top-P Sampler

Pure Java LlmSampler: temperature scaling with max-logit subtraction for numerical stability, dynamic Top-p truncation (sort by probability, accumulate until ≥ top_p), renormalization inside nucleus, then random draw. Three discussion points: max-logit subtraction, adaptive Top-p, nucleus renormalization.

6. Production-Grade Java Engineering

6.1 Token Budgeting ( ContextBudget )

Uses jtokkit (dependency com.knuddels:jtokkit:1.1.0) for exact token counts. Reserves RESERVED_FOR_ANSWER = 1024 tokens for output. Truncation strategy: system prompt never cut; iterate history from newest to oldest; stop when budget exhausted; fallback binary-search truncation of the last user message (subtracting truncation marker tokens first). Hard guards: no branch exceeds window ; throw exception if system prompt alone overflows. Rough estimator provided for gateway pre-checks (Chinese ~1.6 char/token, English ~4 char/token). Additional production notes: keep tool_call / tool_result pairs together; asynchronously summarize dropped history into structured state stored in Redis.

6.2 SSE Streaming with Spring AI

Controller returns Flux<ServerSentEvent<String>> with produces = TEXT_EVENT_STREAM_VALUE. Key practices: full reactive chain (no blockFirst() / collectList()); Nginx proxy_buffering off; proxy_cache off; doOnCancel to cancel upstream model request; timeout + fallback event; TTFT monitoring (warn if >30s). Consumer-side WebClient example parses SSE frames, stops at [DONE], enforces first-packet timeout (TTFT). Token-budget circuit breaker ( AtomicInteger) aborts stream when budget exhausted.

6.3 RAG: Open-Book Exam

Pipeline: embed query → vector search (topK=5, minScore=0.72) → concatenate docs → LLM with strict system prompt ("answer only from knowledge, else say 'I don't know'"), forced JSON output with references array, low temperature (0.2), 8s timeout, fallback to human handoff. Chunking: token-budget-aware, break at sentence boundaries, 10–20% overlap. Cosine similarity helper with 1e-10 epsilon. Three-layer hallucination defense : (1) Retrieval: parent-child chunking + BM25/vector hybrid RRF + rerank + similarity threshold hard-reject; (2) Generation: constrained prompt + forced citations; (3) Post-processing: verify every cited sentence, delete unsourced claims.

6.4 Function Calling: Model = Brain, Java = Hands

Model only emits structured JSON (function name + arguments). Execution, validation, permissions, idempotency, rate limits stay in Java. LangChain4j @Tool with descriptive docstrings; Spring AI registers Function beans with @Description. Both enforce allow-lists.

6.5 Agent Loop (ReAct)

Minimal loop: Thought → Action(tool) → Observation → … → Final. Three hardening patterns: hard step limit ( MAX_STEPS) prevents infinite loops; tool guardrails (allow-list + JSON Schema validation + timeout + idempotency key + result truncation); structured-output self-healing (parse failure → feed error back to model for repair, max 2 retries).

6.6 Pure Java Inference with DJL

Load Hugging Face models (e.g., uer/gpt2-chinese-cluecorpussmall) via ZooModel, configure TextGenerationTranslator (max tokens, temperature, sampling), run prediction. Acknowledges Java tooling lags Python; recommends quantization (AWQ/GPTQ 4-bit), service split (Java biz + Ollama/vLLM inference via gRPC/HTTP), mature frameworks (DJL, LangChain4j).

7. 14 Production Pitfalls & Solutions

Context overflow : 400 errors, truncated answers → Token budget + sliding window + summarization + chunked truncation → jtokkit pre-calc, system never cut, reserve output, gateway intercept.

Long-context amnesia : early preferences lost by turn 30 → Sliding window + persist structured state → Redis session state, inject summary each turn.

Hallucination : fabricates return policy → RAG + forced citations + threshold reject + structured output + low temp + eval set → Post-process citation verification, delete unsourced sentences.

Poor retrieval : irrelevant answers → Parent-child chunks + overlap + BM25/vector RRF + rerank → Chunking & fusion in service, never dump raw long docs.

Slow first token / low throughput : user waits 10s+ → SSE + smaller prefix + prefix cache + small-model routing + parallel retrieval → Flux end-to-end, Nginx buffering off.

Token cost runaway : monthly bill explosion → Context compression + response caching + tiered routing + prompt cache hits → Redis question_fingerprint → answer, skip model on repeat.

Concurrency & rate limits : 1000 concurrent → 429 cascade → Distributed token bucket + multi-tenant quota + queue/degrade + exponential backoff + multi-vendor failover → Redis+Lua token bucket, virtual threads for I/O, separate inference queue.

Stream abort / resource leak : user closes tab, backend keeps burning tokens → Listen cancel signal, cascade cancel upstream → doOnCancel, share() reuse upstream.

Uncontrolled tool calls : wrong tool, bad args, infinite loop → Allow-list + JSON Schema validation + step limit + idempotency key → Validation failure → feed error to model for self-correct, hard stop after N rounds.

Unstable output format : wants JSON, gets markdown fence → Constrained decoding / JSON Mode + tolerant parse + one retry → Parse fail → regex fallback → lower temperature retry.

Non-deterministic output : same prompt → different answers, model drift → temperature=0 + seed + prompt versioning + golden eval set + canary deploy → Eval set runs per version, regression blocks release.

Prompt injection / data leak : user extracts secrets, bypasses rules → Input guardrails + delimiter isolation + least-privilege tools + HITL for sensitive ops + I/O sanitization + output audit → Security boundary & execution rights stay in backend.

Knowledge update / model swap cost : policy changed yesterday, model unaware; vendor switch rewrites all tools → Decouple knowledge mgmt (update/delete/perm/audit) from model; unified abstraction + MCP → Doc changes → MQ → re-embed; code against ChatClient interface, not vendor SDK.

Effectiveness measurement : "feels better" not measurable → Golden test set + RAGAS/LLM-as-Judge + A/B + human spot-check + live feedback → Track accuracy / recall / refuse-rate continuously.

Combined impact: ~30% hallucination reduction, ~60% cost reduction, TTFT <500ms, 99.9% availability (varies by business).

8. Interview Deep-Dive (12 High-Frequency Questions)

Each answer contrasts "crash" vs. "bonus" responses. Key takeaways:

KV Cache : saves recompute; cost = memory bandwidth bound.

RAG cannot eliminate hallucination—three-layer defense needed.

Million-token windows don't replace RAG: cost, accuracy (lost-in-the-middle), governance, multi-tenant isolation.

1000 concurrent users : WebFlux+SSE, Redis token bucket, fingerprint cache, timeout/fallback—all backend fundamentals.

MCP vs Function Calling : FC = model capability (emits JSON); MCP = tool integration protocol (USB-C for AI).

RAG vs Fine-tune : knowledge changes → RAG; style/format changes → LoRA. Fine-tune teaches style, not facts (catastrophic forgetting).

Temperature vs top_p : temperature reshapes distribution; top_p truncates tail. Only tune one.

Temperature=0 still non-deterministic : FP non-determinism, dynamic batching, MoE routing jitter.

Multi-head attention : parallel experts for syntax, coreference, position.

Inference acceleration : quantization, speculative decoding, PagedAttention, operator fusion, sparse attention.

Embedding selection : dimension, language, retrieval scenario; separate from generation model.

Prompt Engineering = Context Engineering : every character shifts probability distribution; 2026 term is Context Engineering (what, how much, order in window).

9. Interview Cheat Sheets

3-Minute Oral Script

"I view LLM as a probabilistic continuation engine , not a database. Forward pass: tokenize → embed + positional encoding → N Transformer layers (Self-Attention: Q·K similarity, softmax weights, weighted V) → logits over vocab → temperature/top-p sampling → append → repeat. Three engineering pillars: (1) Context window is scarce —Java-side token budget with jtokkit, reserve output quota, evict oldest, system prompt immutable. (2) Hallucination is inherent —convert closed-book to open-book via RAG: retrieve evidence, hard-constrain 'say I don't know', force citations, similarity threshold. (3) KV Cache defines the bottleneck —prefill compute-bound, decode memory-bandwidth-bound; long-context optimizations (PagedAttention, quantization, prefix cache) all target KV Cache. Summary: LLM provides language understanding; Java backend makes it controllable, observable, and concurrent."

Crash vs. Bonus Answers Table

Examples: "Attention lets model focus" (crash) vs. "Q·K dot-product, softmax, weighted V—weighted sum" (bonus). "We just call APIs" (crash) vs. "I own token budget, SSE streaming, distributed rate-limiting, model fallback, agent state persistence" (bonus).

Overnight Review Checklist (6 items)

Self-Attention QKV & √d_k (derive manually)

Context window + token math (explain per-turn history resend)

KV Cache + prefill/decode (explain TTFT vs TPOT)

Hallucination cause & RAG layered defense

Function Calling decision/execution separation

Quantization, MoE, Alignment (know trade-offs)

8 Self-Test Questions

Checklist: √d_k reason; GPT vs BERT difference; model memory & token growth; KV Cache savings/cost; Temperature vs Top-P; hallucination inevitability; RAG limitation; who executes function.

30-Second Memorization Version

LLM = Transformer autoregressive next-token prediction
Training = Pre-train(know) → SFT(follow) → RLHF(safe)
Inference = Tokenize → Embed → Attention(√dk scale + causal mask) → Sample(T/top_p)
Accelerate = KV Cache + GQA + Quantization + Continuous Batching
Deploy = RAG(knowledge) + LoRA(style) + Function Calling(hands) + Agent(loop)
Risks = Hallucination / Cost / Latency / Safety → Eval sets & guardrails

10. Closing Principles

"Saying 'KV Cache is memory-bandwidth bound' is 10× more valuable than 'KV Cache speeds things up'; saying 'I reserved 1024 tokens for output' is 100× more valuable than 'I know context windows'."
"LLM handles 'speaking like a human'; engineering handles 'behaving like a reliable system'."
"Engineering value = taming a 'hallucinating genius' into a production-grade service."

Rollout order: (1) Token budget + SSE streaming (lowest cost, fastest win); (2) Build golden eval set before hallucination mitigation; (3) Treat model as unreliable external service—timeouts, retries, circuit breakers, fallbacks, cost metering.

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

JavaLLMRAGAgentinterview-preparationFunction CallingAttention MechanismKV CacheToken BudgetingSSE Streaming
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.