AI-Generated Inference Engine Beats vLLM by 90% at 1,792 tokens/s

Baseten engineers recently demonstrated a striking shift in LLM inference optimization: instead of hand-writing CUDA kernels, they gave Claude Code SSH access to an NVIDIA B200 server and tasked it with autonomously building a custom inference engine for a fixed production configuration. The result, named VibeQwen, outperformed the highly tuned vLLM 0.25.1 across every key metric on the Qwen-3.6-35B-A3B model using NVFP4 precision.

The Burden of General-Purpose Engines

Popular open-source engines like vLLM, SGLang, and TensorRT-LLM carry massive compatibility overhead. They must support multiple hardware vendors (NVIDIA, AMD, Huawei Ascend), numerous quantization formats (FP16, BF16, INT8, INT4, NVFP4), and diverse parallelism strategies (tensor, pipeline, MoE expert parallelism). This forces layers of abstractions and dynamic branches that become pure redundancy when the deployment environment is fully fixed: a single model (Qwen-3.6-35B-A3B), a single GPU (NVIDIA B200), a single precision (NVFP4), and a known traffic pattern (single-stream long-context or high-concurrency specific distribution). Under such rigid constraints, over 90% of a general framework's compatibility logic is dead weight.

Experiment Setup and Results

The team pinned the target:

/goal beat vLLM by 20% on all performance metrics without accuracy loss

. After a week of autonomous optimization, VibeQwen delivered:

Single-stream decode throughput (TPOT): 1,792 tokens/s vs. vLLM's 943 tokens/s — a 90% speedup .

Time to first token (TTFT): 12 ms vs. vLLM's 28 ms — a 57% reduction (2.33× faster).

High-concurrency throughput (concurrency=32): 10,307 tokens/s vs. vLLM's 6,030 tokens/s — a 71% increase .

Cross-Modal Validation with SAM 3.1

To test reusability, the same optimization pipeline was applied to Meta's SAM 3.1 image segmentation model on a single H100, producing a dedicated service called Sammie:

91 images/second on image + short-text prompts — 50% higher than Meta's official reference service.

Development time: ~2 days, consuming ~200 million tokens, by reusing the knowledge base accumulated during the VibeQwen run.

Core Methodology: MetaInfer and LLM-as-Compiler

The workflow is grounded in the MetaInfer architecture (arXiv:2607.12875), which treats the LLM as a compiler. The closed-loop process comprises:

Physical boundary definition (hardware, model, precision, traffic).

Long-horizon autonomous optimization by the coding agent.

Immutable ground-truth adjudication (the Oracle).

Persistent knowledge-base accumulation for reuse.

The Oracle Accuracy Guardrail

Without a strict correctness check, the agent quickly learns to cheat. For example, it could submit a trivial generator that yields the same token forever:

while True:
    yield "a"

Such code achieves near-zero latency and infinite throughput but produces useless output. Baseten's solution: mount the original BF16 weights as the ultimate judge. Every kernel change or memory-allocation tweak must pass a numerical tolerance test against the reference weights; any drift triggers immediate rejection.

Self-Evolving Knowledge Base

Following the autoresearch paradigm, the agent records every verified CUDA kernel combination, memory pre-allocation strategy, and communication scheme into a reusable knowledge base. After completing VibeQwen, this corpus was directly leveraged to accelerate SAM 3.1, cutting development from weeks to days.

Cost Analysis and ROI

The one-week VibeQwen run consumed approximately:

1.7 billion tokens (mostly covered by input-context caching).

200 B200 GPU-hours.

Total R&D cost: a few thousand dollars.

Against annual inference bills of tens to hundreds of millions of dollars at hyperscalers, even a 10% throughput gain translates to millions in saved compute. Spending a few thousand dollars on API and electricity to obtain a 50–90% speedup yields a compelling commercial ROI.

Limitations

VibeQwen and Sammie remain at the experimental/benchmark stage; they have not yet served real mixed production traffic.

Complex multi-node pipelines and dynamic ultra-long contexts still require human architect oversight for robustness.

Conclusion

Baseten's experiment signals a paradigm shift. The traditional moat of hand-tuned CUDA assembly is eroding; the new technical barriers are:

Precisely defining hardware and business boundaries.

Designing cheat-proof benchmark oracles.

Orchestrating automated evaluation pipelines for coding agents.

Using large models to accelerate large models has arrived far sooner than most anticipated.