Google TPU v8 Deep Dive: Dual-Chip Strategy for Training vs. Inference

Google's 8th-Generation TPU: Two Chips, Two Workloads

Google has unveiled its eighth-generation Tensor Processing Units (TPUs), splitting the line into two distinct chips optimized for different stages of the ML lifecycle: TPU 8t for large-scale pre-training and TPU 8i for low-latency Agentic AI inference. Both share architectural compatibility, native FP4 support, PCIe Gen5, HBM3E, liquid cooling, and RAS features, forming an end-to-end ML pipeline solution.

TPU 8t Core Architecture

TPU 8t targets massive pre-training with high compute, large memory pools, and scalable interconnect.

1. Compute Cores

TensorCore: Contains TCS, VPU+Vmem, 2×XLU, 2×MXU — the primary matrix compute units.

SparseCore: 2 units dedicated to sparse computation; 13% area reduction yields an extra 10% TFLOPS.

MXU Optimization: 6% power reduction, 5.8% area reduction, delivering an additional 5% TFLOPS.

2. Memory Subsystem

HBM3E: 12-Hi HBM3E stacks × 6 per chip for high-bandwidth memory access.

Shared Memory: 9,600 chips per Superpod share 2 PB of HBM via Optical Circuit Switching (OCS), enabling memory pooling.

3. Interconnect & I/O

ICI: ICI Router (ICR), 6×Link Stack, 6×224G SerDes octals.

Host Interface: PCIe Gen5 x16 + PCIe Gen2 x1.

On-Chip Interconnect: Memory and DMA Interconnect linking TensorCore, SparseCore, HBM controllers, and ICI.

I/O Bandwidth: 2.4 TBps per chip, supporting glueless scale-up.

4. Network Topology

3D Torus: 8×8×16 structure, 9,600 chips per pod.

Virgo Network: Global non-blocking cluster network supporting 134,400 TPUs, 1.6 YottaFlops, 47 Petabits/s.

5. Reliability & Cooling

HBM link CRC protection, UECC retry, D/Q parity, on-chip interconnect control-path parity.

Voltage/temperature/aging sensors, idle-cycle self-test, advanced fleet health monitoring.

Water-cooled optics (ML first), liquid cooling reduces temperature-related degradation.

TPU 8t (Training Chip) Key Parameters

TPU 8i Core Architecture

TPU 8i targets inference with low latency, high HBM bandwidth, and efficient collective communication.

1. Compute Cores

TensorCore: Includes MXU and other units; 5.3% area reduction yields 5% more FLOPs under same thermal budget.

SparseCore: Helps meet aggressive timing targets.

CAE (Collective Acceleration Engine): Located on the ICI I/O die, executes collective operations in-network, reducing on-chip latency by 5×.

2. Memory Subsystem

High HBM Bandwidth: Inference demands more HBM bandwidth per compute unit; TPU 8i packages more HBM stacks and larger die area.

Memory Access Optimization: CAE avoids HBM accesses to cut latency.

3. Interconnect & I/O

Boardfly Topology: 4 TPUs per tray fully connected; 8 trays form a fully connected group; up to 36 groups fully connected.

Max Hops: 7 hops, far below 3D Torus's 16 hops.

Scale: 1,152 chips per pod (36 groups × 32 chips).

4. Network Characteristics

All-to-All Communication: In MoE inference, all-to-all is latency-bound; Boardfly dramatically reduces hop count.

Low Latency: CAE pushes collective communication into the ICI I/O die, shortening intra-package distance and avoiding HBM accesses.

5. Reliability & Cooling

Shares RAS features with TPU 8t: HBM CRC/retry, on-chip interconnect parity, advanced telemetry, liquid cooling.

Designed for near-100% goodput at 100K TPU system scale.

TPU 8i (Inference Chip) Key Parameters

TPU 8t vs. TPU 8i: Five Key Differences

The core divergence stems from different hardware requirements for training vs. inference :

1. Optimization Goals

TPU 8t: Large-scale pre-training → high compute, large memory pool, high scalability.

TPU 8i: Agentic AI inference → low latency, high HBM bandwidth, efficient collective communication.

2. Packaging & Memory Configuration

TPU 8t: Smaller die, fewer HBM stacks, but achieves memory pooling via 9,600 chips sharing 2 PB HBM.

TPU 8i: Larger die, more HBM stacks to satisfy inference's higher HBM-bandwidth-per-compute demand.

3. Interconnect Topology

TPU 8t: 3D Torus (8×8×16), max 16 hops, suited for massive parallel training.

TPU 8i: Boardfly fully-connected topology, max 7 hops, significantly cuts MoE all-to-all latency.

4. Key Acceleration Units

TPU 8t: Relies on SparseCore for sparse compute; MXU optimization boosts compute density.

TPU 8i: Relies on CAE to execute collectives in-network, reducing on-chip latency 5×.

5. Scale & Network

TPU 8t: 9,600 chips/pod; Virgo Network enables 134,400 TPU global clusters.

TPU 8i: 1,152 chips/pod; Boardfly 36-group full connection, focused on low-latency inference serving.

Summary: TPU 8t centers on 3D Torus + large memory pool + SparseCore for massive pre-training; TPU 8i centers on Boardfly + high HBM bandwidth + CAE for low-latency Agentic inference. Shared architectural compatibility, native FP4, PCIe Gen5, HBM3E, liquid cooling, and RAS create a complete ML pipeline from pre-training to inference service.