Gemma 4 MTP Deep Dive: Speculative Decoding & KV‑Cache Sharing for 3× Faster Inference

Why LLM Inference Is Slow

Standard large language models generate text autoregressively, producing one token per forward pass. Each forward pass must load the entire model’s parameters from GPU memory, creating a memory‑bandwidth bottleneck that dominates latency for models with tens of billions of parameters.

Multi‑Token Prediction (MTP) Draft Model

MTP (also called speculative decoding, first described by Google researchers in 2022) introduces a lightweight draft model that predicts several future tokens while the main model is idle. The workflow consists of three steps:

Draft Generation: The draft model runs in parallel with the main model and silently predicts a short sequence of upcoming tokens.

Parallel Verification: After the main model finishes the current token, it validates the entire draft sequence in a single batch.

Accept + Bonus: If the main model accepts the draft, the whole sequence is emitted and the main model generates one additional “bonus” token, yielding draft length + 1 tokens in the time required for a single token.

The draft model acts like a stenographer that writes a rough draft; the main model serves as the editor that reviews and finalizes the text.

KV‑Cache Sharing

Traditional speculative decoding recomputes the KV‑Cache for the draft model, wasting the context already computed by the main model. Google’s innovation is to let the draft model directly share the main model’s KV‑Cache and activations. The official statement reads: “The draft models seamlessly utilize the target model’s activations and share its KV cache, meaning they don’t have to waste time recalculating context the larger model has already figured out.” Sharing the KV‑Cache eliminates the most time‑consuming part of inference, dramatically reducing draft‑model overhead.

Empirical Validation – Up to 3× Speedup with Zero Quality Loss

Benchmark data from Google and third‑party tests report the following acceleration results:

NVIDIA RTX PRO 6000 (26 B MoE): latency halved at equal output quality.

Apple Silicon M‑series (26 B MoE, batch = 4‑8): approximately 2.2× faster.

NVIDIA A100 (larger batch size): similar acceleration to the RTX PRO 6000 case.

All speedups retain the original output quality because the draft model’s suggestions are only emitted when the main model’s verification succeeds; otherwise the main model falls back to standard generation.

Local Deployment of Gemma 4 with MTP

Method 1: Ollama (simplest)

# Install Ollama if not present
brew install ollama
# Pull the Gemma 4 + MTP model
ollama pull gemma4:27b
# Run; Ollama automatically enables MTP
ollama run gemma4:27b

Method 2: Hugging Face Transformers (more flexible)

from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained(
    "google/gemma-4-27b-it",
    device_map="auto",
    attn_implementation="eager"
)
# GenerationConfig can enable speculative decoding in HF 0.30+

Method 3: vLLM (high‑throughput serving)

# Install vLLM
pip install vllm
# Start the service with MTP enabled
vllm serve google/gemma-4-27b-it \
    --enable-chunked-prefill \
    --gpu-memory-utilization 0.9

Verify the Acceleration

import time, ollama
start = time.time()
response = ollama.chat(
    model='gemma4:27b',
    messages=[{'role': 'user', 'content': 'Write a quick‑sort implementation in Python'}]
)
elapsed = time.time() - start
print(f"Generation time: {elapsed:.2f}s")
print(f"Token count: {response['eval_count']}")
print(f"Speed: {response['eval_count']/elapsed:.1f} tokens/s")

If the tokens‑per‑second metric is noticeably higher than a baseline run without MTP, the acceleration is active.

Why MTP Matters

MTP provides a “free lunch” for deep‑learning inference: up to three‑fold faster generation without any loss of quality. This makes 26‑billion‑parameter mixture‑of‑experts models feasible on consumer‑grade GPUs, enabling real‑time chat and voice interactions on edge devices and potentially reducing power consumption by shortening active compute time.

The combination of speculative decoding and KV‑Cache sharing forms a reusable architectural pattern that other model families can adopt, suggesting broader impact beyond the Gemma family.