LLM Autoregressive Generation Deep Dive: From Next-Token Prediction to Java Streaming Implementation
This article explains LLM autoregressive generation principles including next-token prediction, KV Cache optimization, sampling strategies (temperature, top-k, top-p), and provides complete Java implementations for streaming token generation with Spring AI, WebFlux, and virtual threads.
Core Insight: LLM as a Next-Token Predictor
LLMs are fundamentally massive next-token predictors . Given preceding tokens, the model computes a probability distribution over the vocabulary (typically 30K–150K tokens) and samples one token. This token is appended to the input, and the process repeats until an EOS token is generated. Mathematically: P(next_token | all_previous_tokens). The model does not "understand" — it compresses statistical patterns from trillion-token corpora via maximum likelihood estimation during pre-training.
Inference Pipeline: Tokenizer → Embedding + RoPE → Self-Attention → Causal Mask → FFN Stack → LM Head
Tokenizer : Subword algorithms (BPE/WordPiece/SentencePiece) split text into token IDs. Example: unhappiness → un + happi + ness.
Embedding + RoPE : Tokens map to high-dimensional vectors; Rotary Positional Encoding injects order information into vectors so attention can distinguish "cat chases mouse" from "mouse chases cat".
Self-Attention (Q/K/V) : Each token queries all others. Attention(Q,K,V) = Softmax(Q·Kᵀ/√d) · V. Example: in "Xiao Ming gave apple to Xiao Hong because she was hungry", the token "she" attends to "Xiao Hong" with weight 0.87 vs 0.02 for "apple".
Causal Mask : During generation of token t, positions >t are masked to -inf preventing "cheating" by seeing future tokens.
FFN + LayerNorm + Residual stacked across dozens of layers, then LM Head projects final hidden state to vocabulary logits → Softmax → probability distribution.
Two-Stage Training: Pre-training → Alignment
Pre-training : Goal — Learn universal language & world knowledge. Key Techniques — Transformer Decoder, self-attention, positional encoding, trillion-token corpora.
Alignment (SFT + RLHF/DPO) : Goal — Follow instructions & human values. Key Techniques — Supervised fine-tuning on high-quality QA pairs; Reinforcement Learning from Human Feedback / Direct Preference Optimization.
Inference : Goal — Generate text from prompt. Key Techniques — Autoregressive generation, KV Cache, sampling strategies.
Autoregressive Generation: The Token-by-Token Loop
Definition : Generate one token, append to input, predict next — repeat until EOS . "Auto" = output becomes next input; "Regressive" = same model/parameters reused each step.
Mathematically: chain rule factorization of joint probability P(x₁…xₙ) = ∏ P(xᵢ | x₁…xᵢ₋₁). Each token depends only on predecessors — causal, irreversible.
Round 1: Input "今天" → Predict "天气"
Round 2: Input "今天天气" → Predict "真"
Round 3: Input "今天天气真" → Predict "好"
Round 4: Input "今天天气真好" → Predict "!"
Round 5: Input "今天天气真好!" → Predict <EOS>, stopTraining vs Inference: Same Model, Different Execution
Input source : Training — Ground-truth context (Teacher Forcing); Inference — Model's own generated context (Free Running).
Mask : Both use Causal Mask.
Execution : Training — Full parallel (all positions computed simultaneously); Inference — True serial (one token at a time).
Throughput : Training — Highest efficiency; Inference — Token-level latency, linear in sequence length.
Java focus : Training — Data pipeline, corpus cleaning; Inference — TTFT, TPOT, SSE streaming.
Causal Mask is used in training to match inference distribution — preventing the model from "cheating" by seeing future tokens during training, which would cause train/inference mismatch.
Architecture note: BERT (bidirectional denoising autoencoder) suits understanding, not generation. GPT/LLaMA (Decoder-only + autoregressive) are native to continuation and dialogue.
Autoregressive vs Non-Autoregressive
Generation order : Autoregressive — Serial, one-by-one; Non-Autoregressive — Parallel, all at once.
Strength : Autoregressive — Coherent context, high quality; Non-Autoregressive — Speed.
Weakness : Autoregressive — Latency grows linearly with length; Non-Autoregressive — Poor coherence.
Use case : Autoregressive — Chat, writing, reasoning; Non-Autoregressive — Image generation, text infilling.
Two Inference Phases: Prefill & Decode
Prefill (Pre-fill) : Process all input tokens in parallel — compute-bound . Builds KV Cache.
Decode : Generate tokens serially — memory-bound (repeatedly moving large parameters + cache).
Key metrics: TTFT (Time To First Token) dominated by Prefill; TPOT (Time Per Output Token) determined by Decode. Decode optimization centers on KV Cache.
KV Cache: The Performance Lifeline
Without caching, generating token 1000 requires re-computing Attention over all 999 previous tokens — O(n²) compute. KV Cache stores historical Key/Value tensors in GPU memory; each new step computes only its own Query and attends to cached K/V — reducing complexity to O(n).
No KV Cache : Per-step Compute — Recompute full sequence; Note — Slow, wasteful.
With KV Cache : Per-step Compute — Compute only new token; Note — Standard in production engines.
Why cache K/V but not Q? Q belongs to the current token only — used once, no reuse value. K/V are the "business cards and content" that every future token must query, so they are worth caching. This explains why KV Cache grows linearly with context length — the root cause of memory pressure.
Java Pseudocode for KV Cache
/** KV Cache: inference acceleration key */
public class KVCache {
private final Map<Integer, Tensor> keyCache = new HashMap<>();
private final Map<Integer, Tensor> valueCache = new HashMap<>();
public void append(Token newToken, int position) {
// Compute K, V for new token only; history untouched
keyCache.put(position, computeKey(newToken));
valueCache.put(position, computeValue(newToken));
}
public Tensor attentionWithCache(int currentPos) {
Tensor query = computeQuery(currentToken); // Q only for current position
// Reuse historical K/V, avoid O(n²) recomputation
return attention(query, keyCache, valueCache, currentPos);
}
}Industrial Solutions for KV Cache Memory Pressure
PagedAttention (vLLM core) : Manage KV Cache in "pages" like OS virtual memory — on-demand allocation, reclamation, and prefix sharing across requests. Dramatically improves GPU memory utilization.
K/V Quantization : Compress cache with FP8/INT8.
Engineering eviction : Session-level LRU, maxContext limits, proactive release on request cancellation.
Key insight : PagedAttention's paging mirrors OS virtual memory — paging, on-demand allocation, fragmentation control. This is a backend engineer's knowledge-transfer advantage in AI interviews.
Sampling Strategies: Beyond Greedy Decoding
Greedy (argmax) is stable but repetitive. Production uses sampling with ordered pipeline:
temperature : Role — Smooths/sharpens distribution; Effect — <1 = peaked (conservative); >1 = flat (creative).
top_k : Role — Restrict to top-K candidates; Effect — Controls divergence.
top_p : Role — Nucleus sampling by cumulative probability; Effect — Dynamic; 0.9 balances diversity & coherence.
repetition_penalty : Role — Penalize recent tokens; Effect — Typical 1.1–1.3.
stop_words : Role — Halt on specific tokens; Effect — Business-level stop control.
Order matters : Temperature → Top-K → Top-P → Random sample.
Two Production Gotchas
Softmax must subtract max first . Direct exp(logits) overflows float → NaN. Subtracting max is mathematically equivalent and numerically stable — a favorite big-tech interview detail.
temperature=0 is not fully deterministic . Theoretically reduces to greedy, but GPU floating-point non-determinism means exact reproducibility requires fixed seed + low temperature.
Repetition Loop: Three-Layer Defense
Sampling layer : Top-P 0.8–0.9 + Temperature 0.7 — truncate low-quality tail tokens.
Penalty layer : Apply repetition_penalty to logits of recent N tokens.
Product layer : Detect repeated n-grams → trigger regeneration or hard stop.
If training access exists, add unlikelihood training to explicitly penalize repetition.
Remember: LLM generation is probabilistic sampling , not "answer retrieval" — this is the root of all output non-determinism.
Java Engineering: Wiring the Streaming Pipeline
Positioning : Real inference runs on GPU (vLLM/Triton). Java owns the serial orchestration layer — low latency, governance, streaming.
1. Generation Config (Sampling Parameters as Data)
public record GenerateConfig(
int maxNewTokens,
float temperature,
int topK,
float topP,
int eosTokenId
) {
public static GenerateConfig ofDefault() {
// Common combo: temperature 0.7 + top-p 0.9
return new GenerateConfig(256, 0.7f, 50, 0.9f, 2);
}
}2. Autoregressive Main Loop
public class AutoregressiveGenerator {
private final LlmModel model; // Backed by vLLM / ONNX Runtime / inference gateway
private final Tokenizer tokenizer;
private final TokenSampler sampler;
public String generate(String prompt, GenerateConfig config) {
// 1. Encode: text → token IDs
List<Integer> tokenIds = new ArrayList<>(tokenizer.encode(prompt));
// 2. Prefill: parallel process full prompt, build KV cache
KVCache kvCache = model.prefill(tokenIds);
StringBuilder result = new StringBuilder();
for (int step = 0; step < config.maxNewTokens(); step++) {
// 3. Decode: feed only last token, reuse KV cache
int lastToken = tokenIds.get(tokenIds.size() - 1);
float[] logits = model.forwardOneStep(lastToken, kvCache);
// 4. Sample: temperature → top-k → top-p
int nextToken = sampler.sample(logits, config);
// 5. Stop conditions: EOS / max length (add timeout/cancel in prod)
if (nextToken == config.eosTokenId()) break;
// 6. Autoregressive essence: append generated token back to sequence
tokenIds.add(nextToken);
result.append(tokenizer.decode(nextToken));
}
return result.toString();
}
}3. Streaming Version: Typewriter Effect = Autoregressive + SSE
public Flux<String> generateStream(String prompt, GenerateConfig config) {
List<Integer> ids = tokenizer.encode(prompt);
KVCache cache = model.prefill(ids);
return Flux.generate(
() -> new StepState(ids.size(), cache, 0, lastOf(ids), false),
(state, sink) -> {
// Triple stop: max length / EOS / external timeout + cancel
if (state.step() >= config.maxNewTokens() || state.done()) {
sink.complete();
return state;
}
// Feed only last token + cache, no full-prefix recompute
float[] logits = model.forwardOneStep(state.lastToken(), state.cache());
int next = sampler.sample(logits, config);
if (next == config.eosTokenId()) {
sink.complete();
return state.markDone();
}
sink.next(tokenizer.decode(next)); // SSE push to frontend, first token not blocked
return state.advance(next);
}
).doOnCancel(cache::release); // Critical: release cache on client disconnect
}4. Sampler: Order & Numerical Stability
public class TokenSampler {
public int sample(float[] logits, GenerateConfig config) {
// 1. Temperature scaling + Softmax: T<1 sharpens, T>1 flattens
float[] probs = softmax(logits, config.temperature());
// 2. Top-K: keep only K highest-prob candidates
List<Integer> candidates = topKFilter(probs, config.topK());
// 3. Top-P (nucleus): truncate at cumulative probability p
candidates = nucleusFilter(candidates, probs, config.topP());
// 4. Roulette-wheel: random pick from normalized candidate probs
return rouletteWheelSelect(candidates, probs);
}
private float[] softmax(float[] logits, float temperature) {
// Highlight: subtract max to prevent exp overflow, numerically stable
float max = Float.NEGATIVE_INFINITY;
for (float v : logits) max = Math.max(max, v);
float temp = Math.max(temperature, 1e-5f);
float sum = 0f;
float[] probs = new float[logits.length];
for (int i = 0; i < logits.length; i++) {
probs[i] = (float) Math.exp((logits[i] - max) / temp);
sum += probs[i];
}
for (int i = 0; i < probs.length; i++) probs[i] /= sum;
return probs;
}
private List<Integer> nucleusFilter(List<Integer> cand, float[] probs, float topP) {
cand.sort((a, b) -> Float.compare(probs[b], probs[a]));
float cum = 0f;
List<Integer> nucleus = new ArrayList<>();
for (int id : cand) {
nucleus.add(id);
cum += probs[id];
if (cum >= topP) break; // Truncate when cumulative prob reaches nucleus threshold
}
return nucleus;
}
// topKFilter / rouletteWheelSelect omitted — logic matches names
}5. Spring AI Quick Integration
@RestController
public class ChatController {
private final ChatModel chatModel; // Spring AI unified abstraction
public ChatController(ChatModel chatModel) { this.chatModel = chatModel; }
@GetMapping("/stream")
public Flux<String> stream(@RequestParam String prompt) {
return chatModel.stream(new Prompt(prompt))
.map(resp -> resp.getResult().getOutput().getText())
.filter(Objects::nonNull); // Stream tokens
}
}Streaming (SSE/WebFlux) essence: push each autoregressive token to frontend in real time — principle and engineering unified.
6. WebClient Calling OpenAI-Compatible API
@Service
public class ChatStreamService {
private final WebClient webClient = WebClient.builder()
.baseUrl("https://open.bigmodel.cn/api/paas/v4")
.build();
public Flux<String> streamChat(String userMessage) {
return webClient.post()
.uri("/chat/completions")
.header(HttpHeaders.AUTHORIZATION, "Bearer " + apiKey)
.bodyValue(Map.of(
"model", "glm-4",
"messages", List.of(Map.of("role", "user", "content", userMessage)),
"stream", true // Streaming switch, incremental token push
))
.retrieve()
.bodyToFlux(String.class)
.filter(line -> !"[DONE]".equals(line))
.map(this::extractDeltaContent) // Parse incremental token per frame
.timeout(Duration.ofSeconds(60))
.retryWhen(Retry.backoff(2, Duration.ofMillis(500))); // Exponential backoff
}
}Retry must use exponential backoff . Blind retries after LLM timeout can instantly saturate GPUs — retry storms crashing inference services are a real incident.
7. Java 21 Virtual Threads: Low-Cost High-Concurrency Streaming
// Java 21+: Virtual threads for low-cost high-concurrency LLM streaming
try (var executor = Executors.newVirtualThreadPerTaskExecutor()) {
IntStream.range(0, 10_000).forEach(i -> executor.submit(() -> {
var req = HttpRequest.newBuilder(URI.create(llmUrl))
.header("Content-Type", "application/json")
.POST(HttpRequest.BodyPublishers.ofString(json))
.build();
var resp = httpClient.send(req, HttpResponse.BodyHandlers.ofLines());
resp.body()
.filter(line -> line.startsWith("data:")) // OpenAI / vLLM SSE format
.forEach(this::pushToClient); // Push each token to frontend
return null;
}));
}Tokenizer Consistency Trap
Inference often runs in Python; if Java uses a different tokenizer, same text encodes differently → garbled output. Fix: unify tokenizer (e.g., HuggingFace Tokenizers Java binding) and cache encode results.
KV Cache cuts compute, Top-p controls diversity, SSE enables streaming, Virtual threads / WebFlux handle concurrency — these four pillars let Java integrate LLMs cleanly. But production brings tougher challenges.
Eight Hard Problems & Solutions
Performance & Cost
Serial generation slow : Root Cause — Each token needs a forward pass; inherently non-parallelizable. Solutions — KV Cache reuse, Continuous Batching, Speculative Decoding, INT4/INT8 quantization.
Attention compute/memory explosion : Root Cause — Attention is O(n²). Solutions — FlashAttention, Sliding Window Attention (Mistral), PagedAttention.
KV Cache memory bloat : Root Cause — Cache grows linearly with context; high concurrency → OOM. Solutions — PagedAttention paging + prefix sharing, K/V quantization (FP8), session LRU, maxContext limits, release on cancel.
Low throughput : Root Cause — Single request under-utilizes GPU. Solutions — Continuous/Dynamic Batching, Prefix Caching, Request queuing.
High deployment cost : Root Cause — Model too large for commodity servers. Solutions — Quantization (4/8-bit), Pruning, Knowledge Distillation, LoRA, Tensor/Pipeline Parallelism.
Latency-Throughput-Cost triangle : Root Cause — Three goals conflict. Solutions — PD Separation : Deploy Prefill/Decode on separate hardware, each optimized.
Speculative Decoding : Small model "guesses" N tokens; large model verifies in parallel — 2–3× speedup . Cheap compute bets on expensive serial. Medusa adds multiple prediction heads to large model for parallel prediction — similar idea.
Continuous Batching : Static batching waits for slowest request; fast ones idle. Continuous batching groups at iteration granularity — completed requests exit immediately, new ones join anytime, GPU stays saturated. Standard in vLLM/SGLang.
Generation Quality
Hallucination : Root Cause — Models "plausibility", not "truth". Solutions — RAG, Chain-of-Thought, Citation tracing, Post-generation fact-check, Low temperature, Function Calling, SFT+DPO, Cognitive steady-state governance (model knows what it doesn't know).
Repetition degeneration : Root Cause — Exposure bias + sampling collapse. Solutions — Repetition Penalty (1.1–1.3), Top-P/Temp tuning, Product-layer n-gram detection + regeneration, Unlikelihood training.
Uncontrollable output : Root Cause — Free text drifts. Solutions — System Prompt constraints, Function Calling, JSON Schema validation, Grammar/JSON constrained decoding.
Long-context degradation : Root Cause — Beyond trained length: RoPE extrapolation failure, attention defocus. Solutions — RoPE scaling/NTK extrapolation, Sliding Window Attention, RAG chunk retrieval instead of stuffing full doc , Summarization compression, Vector memory, Support 128K+ context.
Train-inference mismatch : Root Cause — Training sees ground truth; inference sees own outputs. Solutions — Scheduled Sampling, RLHF preference alignment.
Output instability : Root Cause — Sampling is stochastic. Solutions — Lower temperature, Fixed seed, Top-P/Top-K constraints.
On hallucination : Never let the model answer naked . Factual workloads need RAG or Function Calling + citation tracing — probabilistic models don't guarantee facts; that's mechanism, not bug.
Java Engineering
User stares at blank screen : Root Cause — High TTFT. Solutions — SSE streaming first-token priority, First-packet optimization, Reconnection.
Backpressure & disconnects : Root Cause — Slow generation, frontend close, thread pool exhaustion. Solutions — Flux backpressure, Semaphore rate-limiting, Async gRPC (non-blocking Tomcat), doOnCancel cache release, SSE heartbeat.
High concurrency GPU pressure : Root Cause — GPU expensive, easily saturated. Solutions — Token-level rate limiting, Request queue, Tenant quotas, Fallback small model, Resilience4j circuit breaker.
Session context bloat : Root Cause — Multi-turn history grows unbounded. Solutions — Sliding window truncation, Summarization compression, Vector retrieval memory.
Java ↔ Inference stack integration : Root Cause — Inference lives in Python/GPU services. Solutions — gRPC/HTTP/SSE calls; DJL/ONNX Runtime; Unified Tokenizer to avoid cross-language drift.
Inference Framework Selection
vLLM : Core Strength — PagedAttention → high memory utilization. Best For — High-throughput online serving.
TensorRT-LLM : Core Strength — NVIDIA hardware deep optimization. Best For — Max performance on NVIDIA GPUs.
SGLang : Core Strength — Structured generation optimization. Best For — Complex multi-turn, structured output.
Java ecosystem: DJL, ONNX Runtime , or call vLLM's OpenAI-compatible API / Java SDK — treat model service as black box, keep governance in Java.
Interview Cheat Sheet: High-Frequency Follow-ups
Q1: Why Self-Attention not RNN? Training parallelizable; long-range dependency path length O(1) vs RNN's sequential propagation.
Q2: Is temperature=0 fully deterministic? Theoretically greedy, but GPU float non-determinism prevents strict reproducibility.
Q3: Can autoregressive be accelerated? Speculative Decoding (small model guesses, large verifies); Medusa multi-head parallel prediction.
Q4: Prefill vs Decode difference? Prefill parallel/compute-bound; Decode serial/memory-bound. Optimization focuses on reducing memory traffic.
Q5: Why Causal Mask in training? Match train/inference distribution; prevent "cheating" on future tokens.
Q6: Why cache K/V not Q? Q is current-token only, single-use; K/V are queried by all future tokens.
Q7: How Continuous Batching improves throughput? Iteration-level dynamic batching — done requests exit instantly, new ones join, GPU always full.
Q8: Design a Java LLM Gateway layering? Routing + Auth + Streaming + Batching + Rate-limit + Cache + Observability; inference delegated to vLLM/Triton; gateway owns governance.
Interviewers evaluate four dimensions: Conceptual precision (Token/Attention/Probability), Depth (Formulas, KV Cache, O(n²)→O(n)), Engineering delivery (Sampling params, Streaming, Runnable code), Problem awareness (Hallucination/Latency/Repetition with solution catalog). Steering talk toward TTFT, KV Cache, backpressure release separates you from "blog-memorizers".
Mnemonic: Tokenize → Vector → Transformer; Causal mask looks left only; Predict next token; Feed back, predict again — that's autoregression.
Closing Summary
LLM Essence : Transformer compresses massive corpora into conditional probability distributions — given context, predict next token.
Autoregressive Generation : Chain-rule factorization; output fed back as input; token-by-token; serial is its Achilles' heel and optimization starting point.
Engineering Keys : Prefill parallel + Decode serial; powered by KV Cache, Sampling Strategies, Streaming Output, Batching — these four pillars sustain production LLM experience.
Four Production Recommendations
Stream first — TTFT drops from seconds to milliseconds, UX jumps a tier.
Parameterize sampling — Code generation: low temp for rigor; Creative copy: high temp for divergence; Per-scenario profiles.
Treat LLM as unreliable downstream — Timeout, Exponential backoff retry, Rate-limit, Fallback, Disconnect → Cache release. None optional.
No naked model answers — Factual tasks: RAG or Tool Calling + Citation tracing.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
