Skip to content
Tagged articles

speculative decoding

89 articles · Page 1 of 1
Lao Guo's Learning Space
Lao Guo's Learning Space
Nov 16, 2026 · Artificial Intelligence

CPU-Only 284B LLM at 9 Tokens/sec: Why Memory Capacity Now Trumps GPU

The llama-modchip project demonstrates running a 284B parameter MoE model on a 64GB RAM CPU-only machine via aggressive quantization to 56.6GB and kernel optimizations, but achieves only 9 tokens/sec decoding speed, revealing that memory capacity—not GPU—is becoming the key bottleneck for local large language model inference, with practical implications for Mac memory sizing.

CPU inferenceMac hardwareMoE
0 likes · 11 min read
CPU-Only 284B LLM at 9 Tokens/sec: Why Memory Capacity Now Trumps GPU
AI Frontier Lectures
AI Frontier Lectures
Nov 16, 2026 · Artificial Intelligence

NextLat: World Model Constrains LLM Reasoning, 3.3x Faster Speculative Decoding

Microsoft Research and MIT propose Next Latent Prediction Transformer (NextLat), which adds a latent state prediction loss during training to compress history into belief states, achieving 98.7% valid route generation on Manhattan taxi data, 54.8% accuracy on Countdown arithmetic, and up to 3.3x speculative decoding speedup on a 1.3B model.

Language ModelLatent DynamicsMIT
0 likes · 15 min read
NextLat: World Model Constrains LLM Reasoning, 3.3x Faster Speculative Decoding
Lao Guo's Learning Space
Lao Guo's Learning Space
Nov 15, 2026 · Artificial Intelligence

4× DGX Spark vs M5 Ultra: Why Higher Bandwidth Mac Loses on GLM-5.3-Flash

A developer's benchmark shows four NVIDIA DGX Spark units interconnected via 200Gb/s networking outperform a single 512GB Mac Studio M5 Ultra on GLM-5.3-Flash inference across prefill, code generation, and concurrent workloads, despite the Mac's higher 1.23TB/s memory bandwidth, due to software stack differences, KV cache compression, and batching efficiency.

GLM-5.3-FlashKV CacheLLM inference
0 likes · 13 min read
4× DGX Spark vs M5 Ultra: Why Higher Bandwidth Mac Loses on GLM-5.3-Flash
AI Frontier Lectures
AI Frontier Lectures
Nov 14, 2026 · Backend Development

llama.cpp 1.0 Drops 35 Deprecated APIs, Adopts Semantic Versioning

llama.cpp 1.0.0 removes 35 deprecated functions from its public C API, adopts semantic versioning where only breaking API changes trigger major versions, adds FP8 KV cache support, MoE CPU offloading, new model integrations including Kimi-K3.1-Pro and drama-3, and provides pre-built binaries for 150+ architecture variants across major platforms.

C APIFP8KV Cache
0 likes · 5 min read
llama.cpp 1.0 Drops 35 Deprecated APIs, Adopts Semantic Versioning
Lao Guo's Learning Space
Lao Guo's Learning Space
Nov 12, 2026 · Artificial Intelligence

ANE on Mac LLMs: 1.14x–1.49x Only for Prefill, M6 Dual 16-Core Changes Game

The article analyzes three ways to use Apple's Neural Engine (ANE) for LLM inference on Macs, revealing that 1.14x–1.49x speedups only apply to first-token latency (prefill), not token generation; M5 chips trade speed for power efficiency, while M6's dual 16-core ANE enables both speed and efficiency gains but is limited by 32GB RAM.

Apple Neural EngineCore AILLM inference
0 likes · 25 min read
ANE on Mac LLMs: 1.14x–1.49x Only for Prefill, M6 Dual 16-Core Changes Game
Lao Guo's Learning Space
Lao Guo's Learning Space
Nov 9, 2026 · Artificial Intelligence

48GB M5 Pro Mac mini: 27B Is the Coding Threshold, 4-bit Quantization the Sweet Spot

The article analyzes the M5 Pro Mac mini with 48GB unified memory as the cost-effective sweet spot for local LLM coding, showing that 27B models like Qwen3.8-27B at 4-bit quantization (Q4_K_M) achieve ~25 tokens/second, while smaller models lack capability and 70B models exceed memory capacity; 4-bit optimizes context window over higher quantization.

4-bit quantization48GB unified memoryM5 Pro Mac mini
0 likes · 8 min read
48GB M5 Pro Mac mini: 27B Is the Coding Threshold, 4-bit Quantization the Sweet Spot
Machine Heart
Machine Heart
Nov 8, 2026 · Artificial Intelligence

DSpine: Depth-Wise Causal Injection for Parallel Speculative Decoding, 4x SGLang Throughput

Researchers from Tencent Youtu Lab, Shanghai Jiao Tong University, and Xiamen University introduce DSpine, a parallel speculative decoding method that embeds causal dependencies layer-by-layer in the draft network, achieving state-of-the-art acceptance length and up to 4x throughput on SGLang, with open-source code and models.

DSpineLLM inference accelerationSGLang
0 likes · 17 min read
DSpine: Depth-Wise Causal Injection for Parallel Speculative Decoding, 4x SGLang Throughput
Lao Guo's Learning Space
Lao Guo's Learning Space
Nov 6, 2026 · Artificial Intelligence

M6 Launch Day: 9.1 vs 10.6 tok/s — Should You Upgrade 16GB to 32GB for Local LLMs?

Real-world tests show M6 MacBook Pro hits 9.1 tok/s on Qwen3.8-27B 4-bit, reaching 86% of the 10.6 tok/s bandwidth ceiling; the article explains why the dual neural engine doesn't accelerate token generation, debunks the 8-model limit as a software filter, and provides a three-tier buying guide based on memory bandwidth and model size.

Apple SiliconBuying GuideLLM inference
0 likes · 13 min read
M6 Launch Day: 9.1 vs 10.6 tok/s — Should You Upgrade 16GB to 32GB for Local LLMs?
DeepHub IMBA
DeepHub IMBA
Nov 6, 2026 · Artificial Intelligence

Soft-MTP: Fixing Gradient Noise in Multi-Token Prediction via Cross-Entropy Decomposition

Soft-MTP decomposes the cross-entropy gradient in multi-token prediction into a distribution-alignment term and a residual term, then cancels the residual from gradients flowing to MTP heads and the shared backbone while preserving cross-entropy training on the shared output projection, improving language modeling and speculative decoding without extra parameters or full KL computation.

Cross-Entropy LossGradient DecompositionKL Divergence
0 likes · 17 min read
Soft-MTP: Fixing Gradient Noise in Multi-Token Prediction via Cross-Entropy Decomposition
AI Frontier Lectures
AI Frontier Lectures
Nov 4, 2026 · Artificial Intelligence

SpecPilot: 10.87x LLM Inference Speedup via Cost-Model Guided Speculative Decoding

NVIDIA, UCB, Princeton, and Stanford introduce SpecPilot, a cost-model guided speculative decoding system that achieves 10.87x speedup over autoregressive decoding for Qwen3.8-27B on RTX PRO 6000, using 12-bit weight format, tree verification, and two-level verification to optimize per-token verification and batch scheduling.

12-bit weightsLLM inferenceNVIDIA
0 likes · 5 min read
SpecPilot: 10.87x LLM Inference Speedup via Cost-Model Guided Speculative Decoding
SmartApe AI
SmartApe AI
Nov 4, 2026 · Artificial Intelligence

SpecPilot: AI Agent Framework Boosts Speculative Decoding 10.87x for Qwen3.8-27B

Researchers from UC Berkeley, Princeton, Stanford, and NVIDIA open-sourced SpecPilot, an AI agent framework that autonomously optimizes speculative decoding by iteratively identifying bottlenecks and applying techniques like W12 weight compression, best-first tree verification, and two-level verification, achieving 10.87x speedup for Qwen3.8-27B over autoregressive decoding.

AI agentBatching OptimizationBest-first Tree Verification
0 likes · 15 min read
SpecPilot: AI Agent Framework Boosts Speculative Decoding 10.87x for Qwen3.8-27B
Lao Guo's Learning Space
Lao Guo's Learning Space
Nov 2, 2026 · Artificial Intelligence

Why 70B Struggles on 64GB MacBook Pro: 5 Deployment Iron Laws

This guide details five iron laws for deploying large language models on a 64GB MacBook Pro with M5 Pro chip, covering memory budgeting (45GB usable), model tier recommendations (32B smooth, 70B borderline), inference engine selection, speedup techniques like speculative decoding, and API serving best practices.

Apple SiliconLLM DeploymentMacBook Pro M5 Pro
0 likes · 10 min read
Why 70B Struggles on 64GB MacBook Pro: 5 Deployment Iron Laws
Lao Guo's Learning Space
Lao Guo's Learning Space
Nov 1, 2026 · Artificial Intelligence

Two RTX 3090s Hit 3145 tok/s on 180B Model: Strata Engine Brings Giant LLMs to Gaming PCs

The Strata inference engine enables Qwen3.8-Flash-Next (180B total params, 6B active) to achieve 3145 tokens/sec prefill and 120 tokens/sec decode at 400K context on dual RTX 3090s by offloading experts to CPU/SSD, using speculative decoding and chunked prefill, offering a low-cost alternative to Mac Studio for long-context workloads.

LLM inferenceQwen3.8-Flash-NextRTX 3090
0 likes · 11 min read
Two RTX 3090s Hit 3145 tok/s on 180B Model: Strata Engine Brings Giant LLMs to Gaming PCs
Lao Guo's Learning Space
Lao Guo's Learning Space
Oct 31, 2026 · Artificial Intelligence

552B DeepSeek on M5 Ultra 256GB: 3883 Prefill, 92 Decode Tok/s – Local Full-Scale LLM Achieved

Community developers benchmarked the 552B DeepSeek V4.1-Flash model at 3bpw quantization on a Mac Studio M5 Ultra with 256GB unified memory, achieving 3883 tok/s prefill, 92 tok/s decode, and up to 163 tok/s with speculative decoding, demonstrating that full-scale LLMs can run locally on consumer hardware.

Apple SiliconDeepSeekLLM inference
0 likes · 8 min read
552B DeepSeek on M5 Ultra 256GB: 3883 Prefill, 92 Decode Tok/s – Local Full-Scale LLM Achieved
PaperAgent
PaperAgent
Oct 29, 2026 · Artificial Intelligence

Lafa: Block-Level Asynchronous Flow LLMs Achieve 2.5x Throughput Over Autoregressive Models

CMU, UC Berkeley, and Amazon introduce Lafa, a continuous flow language model that converts pretrained autoregressive models into block-level asynchronous denoisers, achieving 2.02–2.54x throughput over AR baselines at 8B scale with matched quality, using under 30B tokens for adaptation.

Block-Level Asynchronous FlowCMUContinuous Flow Models
0 likes · 8 min read
Lafa: Block-Level Asynchronous Flow LLMs Achieve 2.5x Throughput Over Autoregressive Models
Lao Guo's Learning Space
Lao Guo's Learning Space
Oct 26, 2026 · Artificial Intelligence

M5 Max 128GB: Running 125B Qwen3.8-Flash-Next at 60 tok/s with MLX & MTP

This guide details running Qwen3.8-Flash-Next 125B MoE on Apple M5 Max 128GB, covering hardware bandwidth limits, memory constraints, framework selection (MLX vs llama.cpp), command-level optimizations including MTP speculative decoding and GPU memory limit tuning, and realistic performance expectations of 45–60 tok/s with MTP enabled.

Apple SiliconLLM inferenceM5 Max
0 likes · 9 min read
M5 Max 128GB: Running 125B Qwen3.8-Flash-Next at 60 tok/s with MLX & MTP
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Oct 21, 2026 · Artificial Intelligence

DeepSeek Open-Sources DeepInfer Inference Engine & deepseek-recipe API Adapter

DeepSeek has open-sourced DeepInfer, a high-performance inference framework supporting distributed inference, speculative decoding, and SWA Bounded Replay for MoE models, alongside deepseek-recipe, a production-grade token encoding/decoding component that standardizes API request handling for consistent deployment.

API adapterDeepInferDeepSeek
0 likes · 9 min read
DeepSeek Open-Sources DeepInfer Inference Engine & deepseek-recipe API Adapter
Machine Heart
Machine Heart
Oct 21, 2026 · Artificial Intelligence

DeepSeek Open-Sources DeepInfer: Inference Framework Reveals 82% Gross Margin on AI Serving

DeepSeek releases DeepInfer inference framework and deepseek-recipe tokenizer under Apache 2.0, disclosing 155 trillion tokens/day throughput, $38/GPU/hour revenue at list price, and an 82% gross margin assuming $7/GPU/hour rental, with technical details on KV cache retention, speculative decoding, and deployment optimizations.

AI inferenceDeepInferDeepSeek
0 likes · 9 min read
DeepSeek Open-Sources DeepInfer: Inference Framework Reveals 82% Gross Margin on AI Serving
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Oct 19, 2026 · Artificial Intelligence

AdaFlash: Adaptive Speculative Decoding with Online Self-Improvement for High-Concurrency LLM Inference

Nanjing University and Huawei's Foundation Model team introduce AdaFlash, an adaptive speculative decoding framework that enables diffusion draft models to self-improve online via on-policy distillation and dynamic verification length prediction, achieving up to 5.32x end-to-end speedup on Qwen3-8B and 66% higher throughput under high concurrency.

AdaFlashLLM inference accelerationOn-Policy Distillation
0 likes · 17 min read
AdaFlash: Adaptive Speculative Decoding with Online Self-Improvement for High-Concurrency LLM Inference
Baidu Intelligent Cloud Tech Hub
Baidu Intelligent Cloud Tech Hub
Oct 15, 2026 · Artificial Intelligence

TB-Scale LLM Inference: Zero-Redundancy Weight Deployment & 80% KV Cache Compression

Baidu Baige presents extreme VRAM optimization for TB-scale model inference: zero-redundancy weight deployment via DP Attn-sharding (releasing 16–30 GiB/card) and full-route CPP (50% throughput gain), plus KV Cache compression via LU-KV (80% reduction, 0.52% accuracy loss), Draft SWA, and PD handshake optimization (20% VRAM release).

KV Cache compressionLLM inferenceLU-KV
0 likes · 27 min read
TB-Scale LLM Inference: Zero-Redundancy Weight Deployment & 80% KV Cache Compression
Machine Heart
Machine Heart
Oct 15, 2026 · Artificial Intelligence

SpecTr-GBV: Unifying Multi-Draft and Block Verification for Faster Speculative Decoding

SpecTr-GBV unifies multi-draft generation and greedy block verification into a single speculative decoding framework using optimal transport, achieving optimal expected acceptance length under conditionally independent couplings and outperforming SpecTr and GBV across five benchmarks with higher block efficiency and speedup.

Block VerificationConditionally Independent CouplingsGreedy Block Verification
0 likes · 14 min read
SpecTr-GBV: Unifying Multi-Draft and Block Verification for Faster Speculative Decoding
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Oct 10, 2026 · Artificial Intelligence

Relay-OPD: Teacher Steps In at Critical Moments to Fix Student Reasoning Errors in Online Distillation

Zhejiang University and Alibaba propose Relay-OPD, an online distillation method that detects student reasoning errors via teacher-student token divergence and briefly hands control to the teacher for targeted correction, achieving +5.73% accuracy on math benchmarks while halving training trajectory length.

Mathematical ReasoningOn-Policy DistillationPrefix Failure
0 likes · 17 min read
Relay-OPD: Teacher Steps In at Critical Moments to Fix Student Reasoning Errors in Online Distillation
Architecture Digest
Architecture Digest
Oct 9, 2026 · Artificial Intelligence

How Strata Runs a 125B Model on a 12GB Gaming GPU

Strata is an open-source inference engine that enables running the 125B-parameter Qwen3.8-Flash-Next MoE model on consumer gaming PCs with 12GB VRAM by tiering experts across GPU, RAM, and SSD, using speculative decoding for 1.6-1.8x speedup, and providing OpenAI/Anthropic-compatible APIs for local coding agents.

GPU memory offloadingMoEQwen3.8-Flash-Next
0 likes · 14 min read
How Strata Runs a 125B Model on a 12GB Gaming GPU
Tencent Technical Engineering
Tencent Technical Engineering
Sep 22, 2026 · Backend Development

Tencent Backend Veteran's Reset: Mastering LLM Inference Engineering from Zero

A Tencent senior backend engineer with 15 years experience shares his transition to LLM inference engineering, detailing learning paths for model architecture, inference optimization, and self-assessment exercises, while arguing that traditional infrastructure skills remain crucial for AI Infra and Agent Infra development.

AI infrastructureBackend EngineeringKV Cache
0 likes · 38 min read
Tencent Backend Veteran's Reset: Mastering LLM Inference Engineering from Zero
Data STUDIO
Data STUDIO
Sep 11, 2026 · Artificial Intelligence

Running 744B MoE on 25GB RAM: Int4, Disk Streaming & MLA Compression

This article details how a custom C inference engine runs the 744B-parameter GLM 5.2 MoE model on 25GB RAM by quantizing routed experts to int4, streaming them from disk via pread, compressing KV cache 57x with Multi-head Latent Attention, and employing a five-tier memory hierarchy with speculative decoding.

AVX2 kernelsC inference engineGLM-5.2
0 likes · 48 min read
Running 744B MoE on 25GB RAM: Int4, Disk Streaming & MLA Compression
DeepHub IMBA
DeepHub IMBA
Sep 3, 2026 · Artificial Intelligence

FlashSpec: Adaptive LLM Inference with Speculative Decoding — Six Hard-Won Lessons

FlashSpec implements speculative decoding using GPU-native Triton kernel verification and online bandit-based draft model selection, sharing six practical lessons on specification-first development, hidden temperature bugs, cross-platform packaging pitfalls, kernel performance trade-offs, property-based testing value, and adaptive algorithm prerequisites.

CI/CDLLM inferenceThompson sampling
0 likes · 15 min read
FlashSpec: Adaptive LLM Inference with Speculative Decoding — Six Hard-Won Lessons
Xiaomi Tech
Xiaomi Tech
Aug 28, 2026 · Artificial Intelligence

Xiaomi's 8 EMNLP 2026 Papers: Mobile Agents, Efficient LLMs, Proactive AI

Xiaomi had eight papers accepted at EMNLP 2026, covering mobile GUI agents with self-evolving rewards and process reward agents, diffusion LLM training with informed masking, speculative decoding via early-layer collaboration, token-budgeted reasoning, neural architecture search with code semantics, and proactive mobile agents with pre-reasoning perception.

EMNLP 2026LLM efficiencyProactive AI
0 likes · 16 min read
Xiaomi's 8 EMNLP 2026 Papers: Mobile Agents, Efficient LLMs, Proactive AI
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Aug 25, 2026 · Artificial Intelligence

When Online Distillation Goes Off‑Track: Relay‑OPD Lets the Teacher Take the Baton

The paper identifies the “prefix failure” problem in on‑policy distillation, proposes Relay‑OPD—a handoff‑triggered, budget‑controlled teacher takeover mechanism that corrects early mistakes, and demonstrates across eight math‑reasoning benchmarks that it improves accuracy by up to 7.3% while halving training trajectory length.

On-Policy DistillationPrefix FailureRelay-OPD
0 likes · 13 min read
When Online Distillation Goes Off‑Track: Relay‑OPD Lets the Teacher Take the Baton
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Aug 24, 2026 · Artificial Intelligence

When Online Distillation Goes Off‑Track: How Relay‑OPD Lets the Teacher Take Over at Critical Moments

The article analyzes the prefix‑failure problem in on‑policy distillation, introduces Relay‑OPD with a handoff trigger that lets a teacher model intervene locally, and shows through eight math‑reasoning benchmarks that this approach improves accuracy by up to 7.3% while cutting training trajectory length by more than half.

On-Policy DistillationPrefix FailureRelay-OPD
0 likes · 13 min read
When Online Distillation Goes Off‑Track: How Relay‑OPD Lets the Teacher Take Over at Critical Moments
ThinkingAgent
ThinkingAgent
Aug 24, 2026 · Artificial Intelligence

Why Quantization and KV‑Cache Are Key to High‑Performance LLM Inference

The article analyzes why the same LLM can exhibit vastly different cost, speed, and concurrency across inference systems, showing that KV‑cache memory management, continuous batching, PagedAttention, quantization trade‑offs, and speculative decoding together determine real‑world throughput and latency.

KV CacheLLM inferencecontinuous batching
0 likes · 39 min read
Why Quantization and KV‑Cache Are Key to High‑Performance LLM Inference
DataFunSummit
DataFunSummit
Aug 12, 2026 · Artificial Intelligence

How Meta’s Open‑Source 30B Muse Glimmer Fits Into 24 GB VRAM for Always‑On Local Agents

Meta’s Muse Glimmer is a 30B dense transformer with a visual encoder that, after 4‑bit quantization, runs within 24‑32 GB VRAM, achieves agentic benchmark strengths, and uses DFlash speculative decoding to reach 233 tok/s, enabling always‑on, high‑frequency local AI agents on consumer hardware.

AI ModelMuse Glimmerbenchmark
0 likes · 9 min read
How Meta’s Open‑Source 30B Muse Glimmer Fits Into 24 GB VRAM for Always‑On Local Agents
DataFunTalk
DataFunTalk
Aug 11, 2026 · Artificial Intelligence

Meta Open‑Sources Muse Glimmer: Packing a 30B Model into 24 GB VRAM for Always‑On Local Agents

Meta’s Muse Glimmer is a 30‑billion‑parameter, open‑source dense model engineered to run continuously on consumer‑grade hardware with as little as 24 GB VRAM, using 4‑bit quantization and DFlash speculative decoding to retain multimodal, long‑context and agentic capabilities while achieving up to 233 tok/s throughput.

Muse Glimmeragentic benchmarkslocal agents
0 likes · 10 min read
Meta Open‑Sources Muse Glimmer: Packing a 30B Model into 24 GB VRAM for Always‑On Local Agents
Data Party THU
Data Party THU
Aug 5, 2026 · Artificial Intelligence

How GPT‑5.6 Rewrote Its Own Kernel to Slash Service Costs by 20%

GPT‑5.6’s Sol model autonomously rewrote OpenAI’s production GPU kernel, optimizing load balancing, KV cache and speculative decoding, which cut service costs by 20% and boosted token generation efficiency by over 15%, illustrating a closed‑loop self‑optimization but not full recursive self‑improvement.

AI modelsGPT-5.6inference efficiency
0 likes · 9 min read
How GPT‑5.6 Rewrote Its Own Kernel to Slash Service Costs by 20%
AI Engineer Programming
AI Engineer Programming
Jul 22, 2026 · Artificial Intelligence

Is Prompt Engineering Dead? A Deep Dive into Harness, Context Assembly, and Token Generation

The article examines why traditional prompt engineering is no longer sufficient in production AI systems, detailing how harness layers, context reassembly, tool orchestration, token generation methods, training objectives, and architecture choices transform a simple prompt into a complex, multi‑stage workflow that demands robust, system‑level design.

Context ManagementHarnessLLM
0 likes · 16 min read
Is Prompt Engineering Dead? A Deep Dive into Harness, Context Assembly, and Token Generation
Xiaohongshu Tech REDtech
Xiaohongshu Tech REDtech
Jul 10, 2026 · Artificial Intelligence

PIPO: Pair‑In Pair‑Out Architecture Cuts Input Length in Half and Doubles Token Output for LLMs

The PIPO (Pair‑In, Pair‑Out) architecture compresses two input tokens into one latent vector and expands each hidden state to emit an extra draft token, halving effective input length and doubling per‑step output, achieving up to +7.15 pass@4 improvement and 2.64× TTFT / 2.07× TPOT speedups on benchmarks such as AIME 2025, GPQA‑Diamond, LiveCodeBench v6 and LongBench v2 with Qwen3.5‑4B/9B backbones.

EfficiencyMulti-Token PredictionOn-Policy Distillation
0 likes · 14 min read
PIPO: Pair‑In Pair‑Out Architecture Cuts Input Length in Half and Doubles Token Output for LLMs
Machine Heart
Machine Heart
Jun 30, 2026 · Artificial Intelligence

Beyond DeepSeek: Open‑Source JetSpec and Other Projects Accelerate Large‑Model Decoding Up to 10×

The article compares DSpark and JetSpec, two recent open‑source speculative decoding frameworks that tackle inference efficiency from system‑level verification reduction and algorithmic token‑acceptance improvements, respectively, showing up to 9.64× end‑to‑end speedup on Qwen3‑8B and significant gains across math, code, and dialogue benchmarks.

DSparkJetSpeccausal consistency
0 likes · 14 min read
Beyond DeepSeek: Open‑Source JetSpec and Other Projects Accelerate Large‑Model Decoding Up to 10×
DataFunTalk
DataFunTalk
Jun 29, 2026 · Artificial Intelligence

DSpark Explained: 10 Key Concepts You Need to Know

The DSpark system from DeepSeek combines batch decoding, speculative decoding, draft‑model tricks, Eagle‑MTP, DFlash parallelism, variable‑length scheduling and online confidence calibration to deliver up to 85% speedup and four‑fold throughput gains while maintaining generation quality.

Batch DecodingDFlashDSpark
0 likes · 12 min read
DSpark Explained: 10 Key Concepts You Need to Know
PaperAgent
PaperAgent
Jun 29, 2026 · Artificial Intelligence

DeepSeek Opens DSpark: A New Speculative Decoding Framework for Large Language Models

DeepSeek releases DSpark, an open‑source speculative decoding system that combines semi‑autoregressive generation with confidence‑scheduled verification, delivering 60‑85% per‑user speed gains, lower latency, and superior acceptance rates compared with Eagle3 and DFlash across multiple LLM benchmarks.

Confidence SchedulingLLM inferenceSemi-Autoregressive Generation
0 likes · 14 min read
DeepSeek Opens DSpark: A New Speculative Decoding Framework for Large Language Models
Geek Labs
Geek Labs
Jun 29, 2026 · Artificial Intelligence

DeepSpec Boosts Large-Model Inference Speed by 2–5× with Speculative Decoding

DeepSpec, an open‑source framework from DeepSeek, accelerates large‑language‑model inference by 2–5× through speculative decoding, where a lightweight draft model generates candidate tokens that the target model validates in parallel, reducing the serial bottleneck of autoregressive decoding and offering a full‑stack pipeline from data preparation to evaluation.

DeepSpecGPUPython
0 likes · 6 min read
DeepSpec Boosts Large-Model Inference Speed by 2–5× with Speculative Decoding
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Jun 28, 2026 · Artificial Intelligence

DSpark Explained in 10 Essential Concepts: System‑Level Engineering Insights

DSpark, DeepSeek’s new LLM inference framework, combines batch processing, speculative decoding, Eagle‑style draft models and DFlash‑style parallel generation with a lightweight sequential head and hardware‑aware scheduling, delivering 60‑85% speedups while preserving model quality.

DeepSeekGPU optimizationLLM inference
0 likes · 12 min read
DSpark Explained in 10 Essential Concepts: System‑Level Engineering Insights
Machine Heart
Machine Heart
Jun 27, 2026 · Artificial Intelligence

DSpark in DeepSeek V4 Cuts LLM Inference Latency by Up to 85%

DeepSeek V4’s DSpark adds a speculative decoding framework that combines a lightweight draft model, semi‑autoregressive generation, and confidence‑scheduled verification, delivering 60‑85% faster inference for Qwen3 and Gemma models while providing an open‑source DeepSpec toolkit for training and evaluation.

Confidence-Scheduled VerificationDSparkDeepSeek
0 likes · 7 min read
DSpark in DeepSeek V4 Cuts LLM Inference Latency by Up to 85%
Old Zhang's AI Learning
Old Zhang's AI Learning
Jun 19, 2026 · Artificial Intelligence

Gemma‑4‑12B‑v2 (Fable 5 Clone) Achieves 3.5× Telecom Benchmark Boost

The author reproduces Anthropic’s Fable 5 using Gemma‑4‑12B‑v2, showing a 3.5× improvement on the telecom tau2‑bench versus the base model, details the agentic, coding, and general training data, compares quantization sizes, provides llama.cpp launch commands, and notes speed gains from speculative MTP decoding and current limitations.

Agentic AIFable 5Gemma-4-12B
0 likes · 9 min read
Gemma‑4‑12B‑v2 (Fable 5 Clone) Achieves 3.5× Telecom Benchmark Boost
Xiaomi Tech
Xiaomi Tech
Jun 9, 2026 · Artificial Intelligence

How Xiaomi’s MiMo‑V2.5‑Pro UltraSpeed Achieves 1000 TPS on a 1‑Trillion‑Parameter Model

Xiaomi’s MiMo‑V2.5‑Pro UltraSpeed mode breaks the 1000 tokens‑per‑second barrier for a 1‑trillion‑parameter model by combining FP4 expert‑only quantization, DFlash block‑masked speculative decoding, and TileRT’s ultra‑low‑latency GPU system, and the API is now available through a limited‑time trial.

AI inferenceDFlashFP4 quantization
0 likes · 13 min read
How Xiaomi’s MiMo‑V2.5‑Pro UltraSpeed Achieves 1000 TPS on a 1‑Trillion‑Parameter Model
AI Engineer Programming
AI Engineer Programming
Jun 8, 2026 · Artificial Intelligence

When to Use Small Models: A System Design Perspective

Small models are chosen based on deployment constraints rather than absolute parameter counts; the article outlines how resource limits, latency, cost, privacy, and task characteristics define their suitability, compares their strengths and weaknesses to large models, and offers system‑level design patterns for effective use.

Inference OptimizationLLM DeploymentRAG
0 likes · 20 min read
When to Use Small Models: A System Design Perspective
Architect Practice
Architect Practice
Jun 4, 2026 · Artificial Intelligence

Why Cursor Is So Fast: Engineering Beats Model in Agent Systems (Four‑Layer Breakdown)

The article dissects Cursor's Agentic programming platform, revealing that performance gains stem from four engineering layers—architecture, inference, transport, and Agent loop—rather than merely larger models, and offers concrete optimization techniques and lessons for building fast, reliable cloud agents.

Agentic ProgrammingLinux kernel tuningSpeculative execution
0 likes · 17 min read
Why Cursor Is So Fast: Engineering Beats Model in Agent Systems (Four‑Layer Breakdown)
Old Zhang's AI Learning
Old Zhang's AI Learning
May 24, 2026 · Artificial Intelligence

LM Studio Adds MTP Support, Boosting Qwen3.6‑35B to ~130 Tokens/s

LM Studio 0.4.14+ now implements Multi‑Token Prediction (MTP) speculative decoding, eliminating the need for a separate draft model and delivering roughly double the token throughput—e.g., Qwen3.6‑35B reaches about 130 tokens/s on RTX 3090—while providing a six‑step activation guide and a list of known pitfalls.

LM StudioMTPQwen3.6
0 likes · 6 min read
LM Studio Adds MTP Support, Boosting Qwen3.6‑35B to ~130 Tokens/s
Machine Heart
Machine Heart
May 22, 2026 · Artificial Intelligence

Nvidia’s First Tri‑Mode LLM Boosts Token Throughput 4× and Promises Second‑Second Long‑Text Generation

Nvidia introduces a tri‑mode large language model that can switch among autoregressive, diffusion and self‑speculation decoding, delivering up to four times higher token throughput, achieving state‑of‑the‑art accuracy on benchmarks, and showing significant speed gains on DGX Spark, RTX 6000 Pro and GB200 hardware.

LLMNVIDIATri-mode
0 likes · 8 min read
Nvidia’s First Tri‑Mode LLM Boosts Token Throughput 4× and Promises Second‑Second Long‑Text Generation
Old Zhang's AI Learning
Old Zhang's AI Learning
May 16, 2026 · Artificial Intelligence

vLLM 0.21.0 Arrives: Speculative Decoding Now Supports Reasoning Models

The vLLM 0.21.0 release brings five major updates—including Transformers v4 deprecation, a C++20 build requirement, KV offload with hybrid memory, speculative decoding that respects thinking budgets, and a Blackwell token‑speed backend—while offering detailed upgrade guidance for different user groups.

C++20KV CacheTransformers
0 likes · 12 min read
vLLM 0.21.0 Arrives: Speculative Decoding Now Supports Reasoning Models
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
May 14, 2026 · Artificial Intelligence

Elastic Speculative Decoding Breaks Large‑Model Inference Bottlenecks

The paper introduces ECHO, an elastic speculative decoding framework that treats token verification as a global budget‑scheduling problem, uses sparse confidence gating and a two‑level priority scheduler, and demonstrates up to 14.4% throughput gains for high‑concurrency LLM serving.

Inference Optimizationelastic budgetlarge language models
0 likes · 14 min read
Elastic Speculative Decoding Breaks Large‑Model Inference Bottlenecks
Didi Tech
Didi Tech
May 14, 2026 · Artificial Intelligence

Accelerating Training and Inference of EAGLE-3 for Multi‑Round Agent Workflows

This article analyzes the latency bottlenecks of large language models in multi‑round AI Agent scenarios, introduces SpecForge‑based speculative decoding and Unified Sequence Parallelism (USP) techniques applied to the EAGLE-3 model, and presents benchmark results showing over two‑fold Accept‑Len gains and 35‑44% reductions in P95 token‑level latency while enabling 128K context training on an 8‑GPU node.

Agent AIEAGLE-3Unified Sequence Parallelism
0 likes · 26 min read
Accelerating Training and Inference of EAGLE-3 for Multi‑Round Agent Workflows
Lao Guo's Learning Space
Lao Guo's Learning Space
May 7, 2026 · Artificial Intelligence

Gemma 4 MTP Deep Dive: Speculative Decoding & KV‑Cache Sharing for 3× Faster Inference

The article explains why large‑language‑model inference is bottlenecked by memory‑bandwidth, then details Google’s Gemma 4 MTP technique—using a small draft model with speculative decoding and shared KV‑Cache—to parallelize token prediction, achieving up to three‑fold speed gains without any loss in output quality, and provides step‑by‑step local deployment instructions.

Gemma 4Inference OptimizationKV Cache
0 likes · 11 min read
Gemma 4 MTP Deep Dive: Speculative Decoding & KV‑Cache Sharing for 3× Faster Inference
AI Engineer Programming
AI Engineer Programming
May 7, 2026 · Artificial Intelligence

How Cursor Turned Its Coding Agent from Demo to Production

The article examines Cursor's journey of shipping its Composer coding agent, detailing the agentic AI model, system architecture, and the three major production challenges—diff handling, latency accumulation, and sandbox scaling—along with the engineering solutions that enabled reliable, fast, and adoptable AI‑driven code generation.

Agentic AICoding AgentCursor
0 likes · 16 min read
How Cursor Turned Its Coding Agent from Demo to Production
Old Zhang's AI Learning
Old Zhang's AI Learning
May 6, 2026 · Artificial Intelligence

Google Boosts Gemma 4 Inference Speed Up to 3× with MTP Drafter and Day‑0 vLLM Support

Google’s new Multi‑Token Prediction (MTP) drafter for Gemma 4 delivers up to three‑fold inference speedups across hardware and frameworks—validated by official benchmarks and independent DGX Spark tests—while preserving identical output quality, and is immediately usable via Hugging Face, vLLM, MLX, Ollama and edge‑device runtimes.

Apple SiliconGemma 4LLM inference
0 likes · 9 min read
Google Boosts Gemma 4 Inference Speed Up to 3× with MTP Drafter and Day‑0 vLLM Support
Linyb Geek Road
Linyb Geek Road
May 5, 2026 · Artificial Intelligence

Optimizing Retrieval and Generation Latency in High‑Concurrency RAG Agents

The article dissects latency in high‑concurrency RAG Agent pipelines, showing how retrieval, re‑ranking, and LLM generation each contribute milliseconds of delay, and presents system‑level tactics—from ANN index tuning and partitioned search to vLLM PagedAttention, continuous batching, speculative decoding, model quantization, routing, semantic caching, and pipeline parallelism—to dramatically cut end‑to‑end response time.

ANNLLMRAG
0 likes · 15 min read
Optimizing Retrieval and Generation Latency in High‑Concurrency RAG Agents
Old Zhang's AI Learning
Old Zhang's AI Learning
May 1, 2026 · Artificial Intelligence

DeepSeek‑V4 Local Deployment: How SGLang Overcomes the Architecture Challenges

The article analyzes DeepSeek‑V4's architectural innovations—including mixed sparse attention, mHC, and native FP4 weights—explains SGLang's ShadowRadix, HiSparse, and in‑graph speculative decoding solutions, presents benchmark gains, provides Docker deployment steps, and warns of key pitfalls for long‑context inference.

HiSparseLong-context inferenceSGLang
0 likes · 15 min read
DeepSeek‑V4 Local Deployment: How SGLang Overcomes the Architecture Challenges
Old Zhang's AI Learning
Old Zhang's AI Learning
Apr 17, 2026 · Artificial Intelligence

How DFlash Achieves 8× Lossless Acceleration for Large‑Model Inference (Qwen3.5‑27B Example)

The article explains how DFlash’s block‑diffusion draft model and KV Injection boost speculative decoding speed by 5‑8× without sacrificing output quality, and how DDTree further raises the gain to over 8×, backed by benchmark results and integration guides for major inference frameworks.

DDTreeDFlashacceleration
0 likes · 7 min read
How DFlash Achieves 8× Lossless Acceleration for Large‑Model Inference (Qwen3.5‑27B Example)
Old Zhang's AI Learning
Old Zhang's AI Learning
Apr 14, 2026 · Artificial Intelligence

Qwen3.5-27B-DFlash Delivers Up to 5× Faster Inference Without Quality Loss

The DFlash approach replaces speculative decoding’s autoregressive drafter with a block diffusion model and injects target‑model hidden features into every KV‑cache layer, achieving up to 5× speed‑up for Qwen3.5‑27B on single‑GPU and 1.5–1.9× on high‑concurrency workloads while preserving output quality.

DFlashQwen3.5SGLang
0 likes · 12 min read
Qwen3.5-27B-DFlash Delivers Up to 5× Faster Inference Without Quality Loss
Old Zhang's AI Learning
Old Zhang's AI Learning
Apr 7, 2026 · Artificial Intelligence

vLLM 0.19.0: HuggingFace v5 Support, Multimodal Boosts, and CPU KV Cache Offload

The vLLM 0.19.0 release adds first‑day Gemma 4 support, merges zero‑bubble asynchronous scheduling with speculative decoding, matures Model Runner V2, introduces full‑CUDA‑graph acceleration for ViT, generalizes DBO, brings CPU KV cache offload, and expands hardware and Transformers compatibility, offering substantial performance and flexibility gains for production LLM inference.

CPU KV offloadGPUGemma 4
0 likes · 18 min read
vLLM 0.19.0: HuggingFace v5 Support, Multimodal Boosts, and CPU KV Cache Offload
DeepHub IMBA
DeepHub IMBA
Apr 2, 2026 · Artificial Intelligence

Speculative Decoding Explained: Small Draft Model + One‑Shot Verification

The article details how speculative decoding—using a fast small model to draft tokens and a large model to verify them—overcomes the memory‑bandwidth bottleneck of autoregressive inference, introduces SSD’s self‑draft and tree‑verification stages, presents real‑world benchmark gains, and shows how to enable it in vLLM.

GPU memory bandwidthInference OptimizationSSD
0 likes · 14 min read
Speculative Decoding Explained: Small Draft Model + One‑Shot Verification
Machine Heart
Machine Heart
Apr 1, 2026 · Artificial Intelligence

SSD Framework Doubles Inference Speed Over Top Engines, Breaking the Serial Bottleneck

The SSD framework and its SAGUARO optimization, developed by researchers from Stanford, Princeton, and Together AI, parallelize drafting and verification in speculative decoding, eliminating serial dependencies and achieving up to 2× faster inference than the world’s strongest engines and up to 5× speedup over standard autoregressive generation, while addressing challenges such as prediction accuracy, acceptance‑rate trade‑offs, and fallback strategies.

SAGUAROSSDinference acceleration
0 likes · 7 min read
SSD Framework Doubles Inference Speed Over Top Engines, Breaking the Serial Bottleneck
Old Zhang's AI Learning
Old Zhang's AI Learning
Mar 27, 2026 · Artificial Intelligence

vLLM’s Four Major 2026 Updates: Semantic Router Athena, Nemotron 3 Super, P‑EAGLE, and Model Runner V2

The March 2026 vLLM release bundle introduces four substantial upgrades—Semantic Router v0.2 Athena, NVIDIA Nemotron 3 Super, the parallel speculative decoding P‑EAGLE, and a completely re‑architected Model Runner V2—each backed by concrete benchmarks, architectural diagrams, and code examples that demonstrate how the engine evolves from a pure inference engine to a full‑stack AI serving platform.

GPU accelerationModel Runner V2Nemotron 3 Super
0 likes · 17 min read
vLLM’s Four Major 2026 Updates: Semantic Router Athena, Nemotron 3 Super, P‑EAGLE, and Model Runner V2
Old Zhang's AI Learning
Old Zhang's AI Learning
Mar 23, 2026 · Artificial Intelligence

How Large‑Model Research Is Shifting: Insights from 120 Top Papers

The article reveals that large‑model research has moved from sheer scale to deeper capabilities and multimodal integration, highlighting ten hot directions and summarizing 120 recent top‑conference papers—including Spec‑VLA, Mobile‑O, OccTENS, and latent‑CoT studies—while offering free access to the full collection.

3D occupancy modelingcausal reasoninglarge models
0 likes · 7 min read
How Large‑Model Research Is Shifting: Insights from 120 Top Papers
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Mar 12, 2026 · Artificial Intelligence

Nvidia’s Nemotron 3 Super Enters OpenClaw, Rivalling Opus 4.6

Nvidia unveiled the 120‑billion‑parameter Nemotron 3 Super, featuring a Mamba‑MoE hybrid architecture, LatentMoE routing, and Multi‑Token Prediction that together deliver up to 5× higher throughput and 3× faster inference, achieve 85.6% success on OpenClaw—matching Claude Opus 4.6 and GPT‑5.4—and set new records across Pinchbench, MMLU, SWE‑Bench, and other benchmarks, all while being fully open‑sourced with its training data and RL pipelines.

AI agentsLatentMoEMamba-MoE
0 likes · 14 min read
Nvidia’s Nemotron 3 Super Enters OpenClaw, Rivalling Opus 4.6
Old Zhang's AI Learning
Old Zhang's AI Learning
Mar 7, 2026 · Artificial Intelligence

vLLM 0.17.0 Release: Full Qwen 3.5 Support and Anthropic API Compatibility

The vLLM 0.17.0 release brings FlashAttention 4 integration, a mature Model Runner V2, complete Qwen 3.5 series support, a one‑click performance‑mode flag, Anthropic API compatibility, advanced weight‑offloading, broader hardware support beyond NVIDIA, ASR model integration, and detailed upgrade and installation guidance.

ASRAnthropic APIFlashAttention
0 likes · 12 min read
vLLM 0.17.0 Release: Full Qwen 3.5 Support and Anthropic API Compatibility
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Mar 5, 2026 · Artificial Intelligence

Mamba’s SSD Framework Shatters Serial Bottleneck, Outperforms vLLM and SGLang

The new Speculative Speculative Decoding (SSD) framework, built by the Mamba and FlashAttention authors, eliminates the serial draft‑verification bottleneck in LLM inference by running the draft model asynchronously, introducing a speculation cache and the Saguaro algorithm, which together deliver up to 5× speedup over autoregressive baselines and up to 2× over optimized engines on Llama‑3 and Qwen‑3, reshaping the latency‑throughput trade‑off.

Asynchronous ParallelismLLM inferenceSaguaro Algorithm
0 likes · 9 min read
Mamba’s SSD Framework Shatters Serial Bottleneck, Outperforms vLLM and SGLang
Old Zhang's AI Learning
Old Zhang's AI Learning
Feb 16, 2026 · Artificial Intelligence

A New Extreme Quantization Tool for Large Models: AngelSlim’s 2‑Bit Compression

AngelSlim introduces a full‑stack large‑model compression suite that uses quantization‑aware training to shrink a 1.8B LLM to 2‑bit precision, achieving less than 4% accuracy loss, supporting a wide range of models, speculative decoding, and providing end‑to‑end deployment instructions for MacBook M4 and server environments.

AngelSlimGGUFQAT
0 likes · 13 min read
A New Extreme Quantization Tool for Large Models: AngelSlim’s 2‑Bit Compression
AI2ML AI to Machine Learning
AI2ML AI to Machine Learning
Feb 4, 2026 · Artificial Intelligence

Google’s Second Sword: Accelerating LLM Inference with Speculative Decoding and Cascades

The article analyzes Google’s shift from scaling‑law to efficiency‑law, detailing how speculative decoding, language‑model cascades, distillation, CALM, accurate quantized training, and the Mixture‑of‑Recursions architecture together form a multi‑layered strategy to cut inference cost, boost throughput, and sustain the company’s AI moat.

Google TPULanguage Model Cascadesinference acceleration
0 likes · 8 min read
Google’s Second Sword: Accelerating LLM Inference with Speculative Decoding and Cascades
Code Archaeology
Code Archaeology
Jan 19, 2026 · Artificial Intelligence

LLM Inference Optimization: Metrics, Model & Service Techniques

This article systematically covers LLM inference optimization across performance metrics (TTFT, TPOT, throughput, goodput, MFU, MBU), model-level techniques (quantization, distillation, pruning, speculative decoding, attention optimizations), and service-level strategies (continuous batching, prefill/decode separation, prompt caching, parallelism) to improve latency, throughput, and cost-efficiency.

Inference OptimizationKV CacheLLM inference
0 likes · 14 min read
LLM Inference Optimization: Metrics, Model & Service Techniques
AI2ML AI to Machine Learning
AI2ML AI to Machine Learning
Dec 27, 2025 · Artificial Intelligence

Why Jeff Dean Champions Speculative Decoding: The Underlying Ideas

Jeff Dean highlighted speculative decoding as a lossless inference acceleration technique that can boost large language model throughput by 2–3×, and the article breaks down its core concepts—including parallel token verification, draft‑target model collaboration, rejection sampling theory, and practical optimizations such as continuous batching and tree‑based verification.

Draft-Target ModelKV CacheRejection Sampling
0 likes · 8 min read
Why Jeff Dean Champions Speculative Decoding: The Underlying Ideas
Xiaomi Tech
Xiaomi Tech
Dec 1, 2025 · Artificial Intelligence

Seven Xiaomi AI Papers Accepted at AAAI 2026: Multimodal, Embodied & Database Advances

AAAI 2026 accepted seven Xiaomi research papers—two oral presentations—covering multimodal sound editing, embodied 3D agent scheduling, scalable Text-to-SQL schema linking, parallel speculative decoding, long‑form speech QA, high‑level spatial navigation, and VLM‑driven autonomous‑driving adversaries, each with concrete datasets, methods, and benchmark gains.

AAAI 2026Autonomous DrivingEmbodied AI
0 likes · 13 min read
Seven Xiaomi AI Papers Accepted at AAAI 2026: Multimodal, Embodied & Database Advances
Tencent Tech
Tencent Tech
Oct 27, 2025 · Artificial Intelligence

How SpecExit Cuts Large Reasoning Model Inference Time by Up to 2.5×

SpecExit combines early‑exit and speculative decoding to let large reasoning models detect when they have almost finished thinking, trimming redundant chain‑of‑thought steps, reducing over‑thinking by 72% and achieving up to 2.5× faster end‑to‑end inference without noticeable accuracy loss.

AIearly exitinference acceleration
0 likes · 6 min read
How SpecExit Cuts Large Reasoning Model Inference Time by Up to 2.5×
Data Party THU
Data Party THU
Aug 22, 2025 · Artificial Intelligence

TwigVLM: How Tiny Branches Accelerate Large Vision‑Language Models

TwigVLM introduces a lightweight “twig” module that prunes visual tokens early and enables self‑speculative decoding, achieving up to 154% speedup on long‑text generation while preserving 96% of original LVLM accuracy, as demonstrated on LLaVA‑1.5‑7B and other benchmarks.

LVLMModel Accelerationmultimodal AI
0 likes · 14 min read
TwigVLM: How Tiny Branches Accelerate Large Vision‑Language Models
Wu Shixiong's Large Model Academy
Wu Shixiong's Large Model Academy
Aug 20, 2025 · Artificial Intelligence

Mastering Large‑Model Interview Questions: MHA, KV‑Cache, Scaled Dot‑Product, and Speculative Decoding

This guide walks through common large‑model interview challenges, including a hands‑on implementation of multi‑head attention with KV‑cache, the mathematical reason for scaling by sqrt(dₖ), a concise speculative decoding algorithm, and systematic debugging steps for NaN loss during training.

KV CacheLarge Model InterviewMulti‑Head Attention
0 likes · 14 min read
Mastering Large‑Model Interview Questions: MHA, KV‑Cache, Scaled Dot‑Product, and Speculative Decoding
Data Party THU
Data Party THU
Aug 10, 2025 · Artificial Intelligence

Can LLMs Predict Multiple Tokens at Once? A Deep Dive into Multi‑Token Generation

This article evaluates whether autoregressive large language models can generate several tokens in a single inference step, describing a mask‑based multi‑token prediction framework, gated LoRA adaptation, experimental results on Tulu‑3‑8B showing up to 5.2× speedup, and discusses implications for future research.

AI EfficiencyLLMMulti-token generation
0 likes · 13 min read
Can LLMs Predict Multiple Tokens at Once? A Deep Dive into Multi‑Token Generation
ELab Team
ELab Team
Jul 9, 2025 · Artificial Intelligence

How Fast‑Apply AI Models Revolutionize Code Editing with Speculative Decoding

This article explains the design of the edit_file tool, the fast‑apply model that rewrites whole files instead of diffs, its training and evaluation methodology, speculative decoding speed gains, and future research directions for large‑scale code‑editing AI systems.

AIModel Trainingcode editing
0 likes · 14 min read
How Fast‑Apply AI Models Revolutionize Code Editing with Speculative Decoding
AI Algorithm Path
AI Algorithm Path
May 1, 2025 · Artificial Intelligence

Uncovering the Secrets of LLM Inference Optimization

This article dissects the major bottlenecks of large‑language‑model serving—prefill vs. decode, sparsity, memory bandwidth, KV‑cache growth—and walks through concrete engineering tricks such as paged attention, radix‑tree KV caches, compressed attention, speculative decoding, FlexGen weight scheduling, FastServe queuing, plus a runnable vLLM code snippet.

FastServeFlexGenInference Optimization
0 likes · 18 min read
Uncovering the Secrets of LLM Inference Optimization
Architect
Architect
Mar 1, 2025 · Artificial Intelligence

How to Build a High‑Performance, Scalable LLM Inference Engine: From Paged Attention to Multi‑GPU Parallelism

This article analyzes the challenges of deploying large language models locally and presents a comprehensive set of engineering techniques—including CPU/GPU process separation, Paged Attention, Radix Attention, chunked prefill, output‑length reduction, multi‑GPU tensor parallelism, and speculative decoding—to dramatically boost inference throughput and cut response latency.

LLM inferencechunked prefillmulti‑GPU
0 likes · 23 min read
How to Build a High‑Performance, Scalable LLM Inference Engine: From Paged Attention to Multi‑GPU Parallelism
DeWu Technology
DeWu Technology
Feb 17, 2025 · Artificial Intelligence

Optimizing Large Model Inference: High‑Performance Frameworks and Techniques

The article reviews high‑performance inference strategies for large language models such as Deepseek‑R1, detailing CPU‑GPU process separation, Paged and Radix Attention, Chunked Prefill, output‑length reduction, tensor‑parallel multi‑GPU scaling, and speculative decoding, each shown to markedly boost throughput and cut latency in real deployments.

AIGPU accelerationLarge Model Inference
0 likes · 22 min read
Optimizing Large Model Inference: High‑Performance Frameworks and Techniques
NewBeeNLP
NewBeeNLP
Feb 8, 2024 · Artificial Intelligence

How Speculative Decoding Supercharges Large Language Model Inference

This survey examines speculative decoding—a draft‑then‑verify technique that parallelizes token generation to cut LLM inference latency, outlines its core components, compares independent and self‑drafting methods, discusses verification strategies, and highlights open research challenges.

LLM inferenceParallelismartificial intelligence
0 likes · 15 min read
How Speculative Decoding Supercharges Large Language Model Inference