Old Zhang's AI Learning

Oct 7, 2026 · Artificial Intelligence

AI-Generated Inference Engine Beats vLLM by 90% at 1,792 tokens/s

Baseten engineers used Claude Code to auto-generate a custom inference engine (VibeQwen) for Qwen-3.6-35B-A3B on B200, achieving 1,792 tokens/s decode (90% faster than vLLM), 12ms TTFT, and 10,307 tokens/s throughput at concurrency 32, via MetaInfer's LLM-as-Compiler loop with accuracy oracle and reusable optimizations, also speeding up SAM 3.1 by 50% in two days.