Evolution and Challenges of AI Infrastructure: Scaling Large Models on Cloud GPUs

Li Peng, senior technical expert at Alibaba Cloud, delivered a keynote at the 2024 China Generative AI Conference describing how generative AI (AIGC) is reshaping cloud infrastructure requirements. He highlighted three major architectural challenges—power wall, memory wall, and communication wall—driven by the rapid growth of large‑model training and inference workloads.

The talk detailed the exponential increase in compute demand, citing that training a GPT‑3‑scale model (175 B parameters) would require roughly 3 640 PFLOP·days and over a thousand A100 GPUs for a month, making both hardware cost and energy consumption critical concerns.

Alibaba Cloud’s response is the Elastic Compute Service (ECS) DeepGPU toolkit, which enhances GPU utilization for both training and inference. Reported performance gains include up to 80% improvement for LLM fine‑tuning and up to 60% for Stable Diffusion inference.

For training, the presentation covered the software‑hardware stack, emphasizing model architecture (Transformer), massive data, and gradient optimization, as well as hardware scaling from single GPUs to multi‑node clusters. It explained model loading and parallelism challenges, illustrating with a 175 B model that requires ~2.8 TB of GPU memory and multiple parallelism strategies (tensor, pipeline, and data parallelism) to achieve efficient scaling.

The session also examined communication bottlenecks in distributed training, such as frequent All‑Reduce operations in tensor parallelism, and described how NVLink, PCIe P2P, and affinity‑aware scheduling can mitigate these overheads.

Inference challenges were addressed by focusing on three factors: GPU memory capacity, memory bandwidth, and quantization. The speaker showed that inference is largely memory‑bandwidth bound, with examples like LLaMA‑7B on A100 versus A10, and highlighted the importance of multi‑GPU inference and quantization to reduce memory usage.

Case studies demonstrated DeepGPU’s impact: fine‑tuning Stable Diffusion achieved 15‑40% end‑to‑end speedup, LLM fine‑tuning saw up to 80% performance gains, and a customer’s question‑answering service realized nearly fivefold improvement in request processing latency.

Finally, the talk referenced emerging video generation models such as OpenAI’s Sora, which demand even higher compute resources (4000‑10 000 H100 GPUs for a month of training), underscoring the ongoing escalation of AI infrastructure needs.