Why AWS’s Self‑Designed Chips Are Redefining AI Infrastructure

From Three Big Chips: AWS AI Hardware Overview

To explain the most advanced AI infrastructure announced at AWS re:Invent 2024, we start with the three major chips in Amazon’s data‑center portfolio: Graviton4 CPU, Nitro5 DPU, and Trainium2 "GPU" accelerator.

Graviton4 CPU

Nitro5 DPU

Trainium2 accelerator

Since the first fully custom Nitro chip in 2013, AWS has become one of the few cloud providers that designs its own CPU, DPU, and GPU, enabling deep integration between hardware and services.

The main reasons AWS continues to invest in custom silicon are:

Innovation : Hardware innovation underpins software and service breakthroughs.

Efficiency : Owning the hardware lets AWS meet its own performance and feature requirements precisely.

Low Cost : Massive scale and a dedicated supply chain allow AWS to drive down chip costs and pass savings to customers.

Graviton4

Graviton is an ARM‑based CPU; the fourth generation launched at re:Invent 2023 and is now widely deployed. By mid‑2024, 50% of new CPU capacity runs on Graviton, offering higher‑efficiency EC2 instances.

Graviton4 uses the latest ARM Neoverse V2 architecture, one of the first chips to support ARM v9.

96 CPU cores per socket.

L2 cache doubled to 2 MiB per core (total 192 MiB).

12‑channel DDR5‑5600 memory, 75% higher bandwidth (up to 537.6 GB/s).

96 PCIe 5.0 lanes for high‑speed I/O.

Up to 60% lower power at comparable performance.

With three times the vCPU cores of comparable x86 servers, Graviton4 enables larger, more flexible EC2 instances at lower cost.

Real‑world workloads such as web, database, and Java applications see 30‑45% performance gains compared with benchmark‑only designs.

Nitro5

Nitro5 continues AWS’s DPU innovation, providing full‑stack offloading for network, storage, hypervisor, and security functions while eliminating CPU overhead.

Nitro’s security capabilities include:

Nitro Security Chip : Hardware‑rooted protection that restricts privileged access and prevents tampering.

Nitro TPM : TPM 2.0 support for key generation, storage, and attestation, ensuring instance integrity.

Nitro Enclaves : Isolated CPU‑and‑memory environments for processing highly sensitive data.

These features address hardware‑level security, supply‑chain integrity, and multi‑tenant isolation, which are critical for public‑cloud trust.

Trainium2

Trainium2 is AWS’s AI accelerator designed for training models with over 100 billion parameters. It delivers 1.3 FP8 PetaFLOPS per chip, supports BF16/FP8 precision, and includes 96 GiB HBM3 with 46 TB/s bandwidth.

Key architectural blocks:

Large NeuronCore integrating tensor, vector, scalar, and GPSIMD engines.

Dedicated collective‑communication core for multi‑chip networking.

Voltage regulators placed near the package for higher energy efficiency.

Trainium2 Server integrates 16 Trainium2 chips, delivering 20.8 Pflops and 1.5 TiB HBM; Trainium2 UltraServer packs 64 cards into a single cabinet, reaching 83.2 Pflops, 6 TiB HBM, and 185 TB/s bandwidth, enabling training of trillion‑parameter models.

Scale‑Up (single‑node) is essential because the largest AI models no longer fit on a single accelerator; scaling out alone cannot address memory and bandwidth limits.

Project Rainier

AWS and Anthropic are building an AI super‑computing cluster called Project Rainier, comprising hundreds of thousands of Trainium2 chips and delivering roughly 130 FP8 ExaFLOPS—over five times the performance of previous generations.

10p10u Network Architecture

To connect thousands of Trainium2 UltraServers, AWS designed a 10p10u network offering up to 10 PB of capacity and sub‑10 µs latency, scalable from a few racks to multiple data‑center campuses.

SIDR Routing Protocol

SIDR (Scalable Intent‑Driven Routing) manages the massive AI network, combining centralized planning with distributed execution to achieve sub‑second fault recovery, ten times faster than traditional methods.

Amazon Bedrock: Next‑Gen Gen‑AI Interface

Amazon Bedrock, launched in 2023, provides a fully managed service for accessing leading foundation models, supporting custom model import, fine‑tuning, retrieval‑augmented generation, and managed agents.

At re:Invent 2024, AWS introduced low‑latency inference on Trainium2 servers, delivering faster response times for models such as Llama 405B and Llama 70B, giving customers a competitive edge.

Conclusion

By continuously investing in custom silicon such as Graviton, Nitro, and Trainium, AWS builds more powerful AI servers. This relentless focus on hardware detail drives elasticity, security, performance, cost efficiency, reliability, and sustainability—key pillars for maintaining leadership in the fast‑moving AI landscape.