Skip to content
Tagged articles

TTFT

2 articles · Page 1 of 1
Code Archaeology
Code Archaeology
Jan 19, 2026 · Artificial Intelligence

LLM Inference Optimization: Metrics, Model & Service Techniques

This article systematically covers LLM inference optimization across performance metrics (TTFT, TPOT, throughput, goodput, MFU, MBU), model-level techniques (quantization, distillation, pruning, speculative decoding, attention optimizations), and service-level strategies (continuous batching, prefill/decode separation, prompt caching, parallelism) to improve latency, throughput, and cost-efficiency.

Inference OptimizationKV cacheLLM inference
0 likes · 14 min read
LLM Inference Optimization: Metrics, Model & Service Techniques
Wu Shixiong's Large Model Academy
Wu Shixiong's Large Model Academy
Nov 16, 2025 · Artificial Intelligence

How to Slash RAG First‑Token Latency: Practical Engineering Strategies

This guide breaks down the three layers of a RAG pipeline—embedding, vector retrieval, and system architecture—and provides concrete engineering tactics such as batch embedding, async concurrency, caching, ANN indexing, partitioning, connection pooling, and async pipelines to dramatically reduce Time‑to‑First‑Token latency.

Async PipelineEmbeddingRAG
0 likes · 10 min read
How to Slash RAG First‑Token Latency: Practical Engineering Strategies