Wafer-Scale AI Chips: Co-Designing for 4x Token-per-Dollar Gains

Token Economics: Redefining AI System Evaluation

Professor Hu Yang (Tsinghua University) opened the 2026 Global AI Chip Summit (GACS 2026) with a talk on wafer-scale chips for token economics. Token/$ measures effective tokens produced per total dollar invested under given model quality and service requirements. This metric expands evaluation from single-chip peak FLOPS to full-system sustained token output, encompassing compute, memory, communication, utilization, plus network, rack, power, cooling, and ops costs.

Three Walls Limiting Token/$

Communication Wall: Data exchange and synchronization wait erode gains from added compute; larger clusters add switch layers, buffering, and forwarding overhead.

Power Wall: Power budget and cooling capacity cap sustained output; networking gear also consumes power.

Density Wall: Switches, cables, and ancillary equipment inflate deployment cost per token.

Hu cited Sam Altman's projection: supporting future token demand with current infrastructure would exceed China's GDP, making token economics imperative.

Wafer-Scale Integration: Systemic Solution

Moving integration inside the chip, wafer-scale designs convert cross-board communication to on-wafer links, use larger compute granules for scaling, and co-design power delivery and cooling. Quantitative claims:

Bandwidth density +2.2×

Power/cooling efficiency +2×

Switch count -90% (later refined to -60% in summary slide)

Two technical routes: chiplet integration (known-good die on large interconnect substrate, heterogeneous processes; e.g., Tesla Dojo, Tsinghua's Pizza) and monolithic wafer-scale (cross-reticle interconnects with redundancy/repair; e.g., Cerebras WSE). Same-area comparison shows interconnect bandwidth density from 0.7 to 5.2 GB/s/mm², communication energy from 10 to 0.15 pJ/bit, latency from ~400 ns to ~10 ns.

Full-System Co-Design Framework

Tighter integration couples design elements: compute must match internal dataflow, intra-/inter-wafer networks co-organized, power/thermal/I/O/packaging coordinated. The team jointly explores compute, interconnect, integration, fault tolerance, yield/cost, and thermal management to turn physical advantages into token-economic gains.

Key Research Projects

WSC-LLM (ISCA 2025)

Stage-specific resource configuration for Prefill/Decode. Jointly explores parallelism, stage resource ratio, traffic, and placement. Achieves 3.12× inference throughput over uniform configuration.

MoEntwine (HPCA 2026)

Co-optimizes token-to-expert mapping and expert load balancing for MoE. Reduces All-to-All contention and shortens transport distance. At 4 ms token interval, single-chiplet throughput reaches 1.84× baseline .

TEMP (HPCA 2026)

Bidirectional tensor flow orchestration over high-bandwidth die-to-die links. Topology-aware scheduling overlaps communication with computation, converting multi-hop tail waits into single-hop relays. Average training throughput 1.21× baseline .

Mesh-Switch & Tick-Tock (ISCA 2025)

Physical-aware topology co-exploration (area, wiring layers, distance). Mesh-Switch blends mesh and fat-tree, configuring compute/storage/communication resources together. Delivers 2.39× training throughput while matching realizable wafer resources.

Wafer-Scale Expansion Network

Uses wafer as scaling unit, collapsing traditional rail/spine hierarchies. Analytical model shows 68.2% lower expansion overhead, 90% fewer switches, 87% fewer cables, 46% lower power cost .

WATOS (HPCA 2026)

Fault-aware scheduling, adaptive rerouting, recomputation, and data migration to rebalance compute/memory after die/link failures. Turns fault tolerance into an architectural capability for sustained throughput.

TITAN (ISCA 2025)

Hierarchical model unifying compute, memory, interconnect, power/thermal, and rack organization under area, power, and thermo-mechanical warp constraints. Answers how to proportion resources within one wafer and one rack for best system return.

WSC-Cost (DAC 2026)

Incorporates substrate fabrication/assembly yield, wiring, die size, pin count, bonding process into cost model. Jointly evaluates performance gain, manufacturability, and manufacturing cost for design selection.

ReThermal (HPCA 2026)

Behavior-driven power model, fast DNN thermal prediction, and static/dynamic scheduling. Throttles hotspots to improve overall throughput. Peak temperature reduced 7.5–8.0°C , system throughput up 11.21%–39.23% .

Domestic Prototypes & Software Stack

Team fabricated wafer-scale test chips and a test system using domestic process and supply chain, establishing large-substrate design PDK, large-scale integration ADK, chiplet integration, vertical power delivery, and system validation capabilities. Built a dataflow-oriented software stack with operator library, compiler optimizations, and hardware backend support, closing the loop from research to engineering verification.

Token/$ Comparison & Conclusion

Five configurations evaluated (million tokens/$): DGX-H100 3.53, DGX-B200 14.29, NVL72-GB200 21.02, 4nm wafer-scale 87.43, domestic-process wafer-scale 25.35. Domestic wafer-scale achieves 1.21× NVL72-GB200 ; 4nm wafer-scale reaches 4.16× . This demonstrates the opportunity to improve token economics via full-system co-design under domestic process conditions, with further headroom from advanced nodes.

Hu concluded: improving token economics is both the motivation for wafer-scale research and the yardstick for system value. Through architecture research, domestic prototypes, and supporting software, the goal is to have data computed at the right place and time with controllable cost.