Why Storage Is the New Engine Driving AI Server Growth in 2024

AI Server Memory Landscape

Traditional servers typically cost under $100 k, while a DGX H100 AI server equipped with eight H100 accelerator cards can exceed $400 k (≈¥3 million). The price gap highlights the growing importance of high‑performance memory and storage bandwidth in AI workloads.

Memory‑Wall and High‑Bandwidth Memory (HBM)

Processor frequencies continue to rise, but DRAM bandwidth lags, creating a “memory wall” that can reduce effective compute performance by 50‑90 % in AI and vision tasks. HBM mitigates this bottleneck by stacking DRAM dies vertically, delivering >1 TB/s bandwidth (HBM3E) with lower power per bit compared to conventional DDR5.

Market data for 2023 shows SK Hynix with 55 % of HBM revenue, Samsung 41 %, and Micron 3 %. Micron entered the market later but launched its first HBM2 product in 2020, quickly progressed to HBM3, and released a 24 GB 8‑High HBM3E in July 2023. Micron’s roadmap includes:

2025: 36 GB 12‑High HBM3E

2026: 36 GB 12‑High HBM4 with >1.5 TB/s bandwidth

2027: 48 GB 16‑High HBM4

2028: HBM4E with >2 TB/s bandwidth

Emerging Storage‑Interconnect Technologies

CXL (Compute Express Link) – A new interconnect standard that enables DRAM pooling across servers, reducing data‑center construction costs and improving memory utilization.

MCR/MDIMM (Multiplexer Combined Ranks) – Combines multiple DRAM modules into pseudo‑channels, theoretically doubling DDR5 bandwidth. The JEDEC MR‑DIMM specification is being promoted by AMD (MemCon 2023) and supported by Intel in collaboration with SK Hynix and Renesas.

PCIe 5.0 – Provides up to 32 GT/s per lane (≈4 GB/s per lane raw, ~3.94 GB/s after encoding), enabling faster serial communication for AI accelerators and supporting future PCIe 6.0 upgrades.

Industry Implementations

Lankei Technology completed development of MCR control chips (MRCD/MDB) in 2022 and now ships DDR5 memory‑interface chips, retimers, and solutions for upcoming PCIe 6.0, addressing the bandwidth demands of AI servers.

Supply‑Chain Implications

The surge in AI‑driven memory demand creates growth opportunities across the upstream chain, including memory‑chip fabrication, PCB and packaging, and specialized EEPROM (e.g., SPD EEPROM for DDR5 modules). Companies expanding into memory‑interface chips, PCIe retimers, and hybrid secure memory modules can capture a share of this expanding market.

Outlook

AI‑server storage bandwidth is projected to grow at an 8.2 % compound annual growth rate over the next five years. Continuous iteration of DDR5, HBM3E/4, CXL, and MCR technologies will drive both performance improvements and cost efficiencies, shaping the next generation of AI infrastructure.

References

https://mp.weixin.qq.com/s?__biz=MzUzMzY1NTkwOQ==∣=2247519717&idx=1&sn=5b7a624389fce615de76ebc5eee7b52e&scene=21#wechat_redirect

https://mp.weixin.qq.com/s?__biz=MzAxNzU3NjcxOA==∣=2650749656&idx=1&sn=37492dc1340261225317ad45b07d8a01&scene=21#wechat_redirect

http://mp.weixin.qq.com/s?__biz=MzUzMzY1NTkwOQ==∣=2247519158&idx=1&sn=1c5c7fa790b6ee1b79b4e99b8b6ac721&scene=21#wechat_redirect

http://mp.weixin.qq.com/s?__biz=MzUzMzY1NTkwOQ==∣=2247519391&idx=1&sn=1175453d07ff66c8ef6fec5bdf880228&scene=21#wechat_redirect