Google Open-Sources TorchTPU: TPU Now Native in PyTorch, Challenging CUDA's Moat

TorchTPU: Native PyTorch on TPU

At PyTorch Conference 2026, Google announced the open-source release of TorchTPU , a complete software stack that makes TPU a first-class PyTorch device alongside CPU and GPU. In Hugging Face Transformers v5, deploying a model to TPU now requires only a single parameter: device_map="tpu".

Architecture: Not Another Adapter

Previous efforts like PyTorch/XLA (2020) and TorchAX relied on lazy tensors or translation to JAX, requiring invasive source changes, suffering high compilation latency, and lacking eager-mode support. TorchTPU, started mid-2025, was built on three principles:

No runtime dependency on JAX

Solid eager execution

Migration from GPU as simple as changing .to("cuda") to .to("tpu") Implementation uses PyTorch's PrivateUse1 interface to register a new C++ backend in the dispatcher. Developers get plain PyTorch tensors—no subclasses or wrappers.

Three-Tier Eager Execution

Strict (default) : dispatches ops one-by-one, async with CPU, matching typical GPU experience.

Debug : synchronous execution for shape/NaN debugging.

DeferAndFuse : new hybrid eager/compiled mode that buffers a sequence of ops and fuses them into large kernels when a value is needed. It runs >50% faster than Strict and reduces peak memory.

All three modes share a layered compilation cache; the same op sequence compiles once and reuses.

Compiled Path via torch.compile

For peak performance, torch.compile(backend="tpu") captures graphs via TorchDynamo, then lowers operators to StableHLO and hands off to XLA for TPU codegen—bypassing Inductor. Google cites three reasons: reuse of eager-mode infrastructure, XLA's proven stability, and Inductor's reliance on Triton, which is hard to port to TPU. This underscores how deep the PyTorch–GPU binding goes, down to the compiler layer.

Distributed Training

TorchTPU implements TPU collective communication at the Process Group level, enabling DDP, FSDP2, and DTensor out of the box. It also supports asymmetric patterns (e.g., rank 0 doing extra logging), which SPMD-only PyTorch/XLA could not. Per PyTorch/XLA docs, TorchTPU will replace PyTorch/XLA.

Report Card: Coverage, Performance, Scale

Model Coverage

Google continuously tests the ~5,000 most-downloaded Hugging Face models on TPU. As of the report: 4,917 models across 500+ architecture families ; 98% run without crashing ; on 657 representative architectures, 97% match GPU outputs numerically .

Out-of-the-Box Performance

Using only standard PyTorch modules (no custom kernels) for a 30B MuseGlimmer model:

Single TPU v7: 25,700 tokens/s prefill at 5120 seq len, MFU 59%.

4× TPU v7 training: MFU 38%.

Ported TorchTitan GPU recipe with minor changes: training MFU 36%.

Large-Scale Scaling

On Qwen3 235B MoE:

64 TPU v7: PyTorch throughput reaches ~94% of Google's JAX-based MaxText.

Scaling from 1,024 to 8,192 chips (full v7 Pod): per-TensorCore throughput drops only ~6%, MFU stable at 27–29%, loss curves nearly overlap across scales.

Qwen3 8B training MFU up to 52%.

Inference: TPU v7 + vLLM-TorchTPU achieves $2.01 per million tokens (8K in / 1K out), down ~40% from the September debut of 3.6¢.

Case Studies

FLUX training: eager-mode step 11.49s → fully compiled 2.78s.

Meta DLRM v2 on 32 TPU v7x matches highly optimized JAX baseline.

Already running inference and training in production for select Google Cloud customers.

Full-Stack Ecosystem Drop

Inference : vLLM backend (with Inferact), SGLang backend (with RadixArk) preserve upstream schedulers/APIs; llm-d 1.0 (with Red Hat, IBM) unifies GPU/TPU entrypoint on GKE.

Training : TorchTitan recipes for dense, MoE, diffusion; VeRL adds TPU backend for RL; Ray schedules workflow stages across CPU/GPU/TPU.

Edge : LiteRT (ex-TensorFlow Lite) joins PyTorch ecosystem.

Four core repos—TorchTPU, vLLM-TorchTPU, SGLang-TorchTPU, TorchTitan—open-sourced simultaneously.

Industry Signals

Meta's PyTorch engineering director Jana van Greunen stated Meta cannot be locked to a single hardware architecture; TorchTPU already powers key LLM, ads, and recommender production workloads—despite Meta being Nvidia's largest customer.

AWS presented TorchNeuron for Trainium, also moving off PyTorch/XLA to native PrivateUse1 integration. PyTorch is becoming a de-facto hardware abstraction layer; PrivateUse1 is the standard slot. Competition shifts from "ecosystem compatibility" to "throughput per dollar"—the metric Google emphasizes.

Remaining Gaps & Roadmap

Framework parity ≠ hardware parity: TPU TensorCores are single-threaded matrix units needing wide shapes; GPU favors many small kernels. Tuning still required.

Numerics: TorchTPU defaults to BF16 multiply + FP32 accumulate vs. full FP32 on CUDA.

Default collective communication compilation incurs 30–40% end-to-end loss; needs manual annotation.

One process per TPU; co-located training+inference in RL not yet supported out of the box.

Most benchmarks are Google-internal; community validation needed.

TPU availability/pricing for average developers remains a variable.

Google's 6–12 month focus: production hardening, dynamic shapes, FP8/FP4 quantization for next-gen TPU.

Conclusion

For a decade, choosing PyTorch effectively meant choosing CUDA. With TorchTPU, that equation no longer holds.