From Black Iron to Silver: The Evolution of Large Model Infrastructure (2019‑2024)

Overview

This summary traces the technical evolution of large‑model (LLM) training and inference infrastructure from the early experimental stage to the current period of concentrated resources. It highlights the major research breakthroughs, open‑source frameworks, hardware milestones, and the shifting business model that have shaped how engineers build and operate LLM systems.

2019‑2021 – “Black‑Iron” Era

After the 2017 Attention Is All You Need paper, Transformer‑based models (GPT‑1, BERT, T5) quickly became the dominant NLP architecture. Researchers began to explore scaling laws and Mixture‑of‑Experts (MoE) designs, leading to prototypes such as Google’s GShard (600 B parameters on 2 K TPUs) and the TensorFlow‑based distributed training libraries MeshTensorFlow and GShard.

Key open‑source training projects emerged:

Megatron‑LM (NVIDIA, 2019) – PyTorch implementation of tensor parallelism and pipeline parallelism.

DeepSpeed (Microsoft, 2020) – introduced ZeRO optimizer, offloading, and flexible parallelism strategies.

During this period the community also revisited earlier parallelism concepts:

FlexFlow (automatic model parallelism, 2019)

GPipe (pipeline parallelism, 2019)

Parameter‑Server style sharding (2014)

Infrastructure challenges were severe: GPU resources were scarce, most companies lacked dedicated infra teams, and integrating multi‑GPU training into existing pipelines required substantial engineering effort.

2022‑2023 – “Golden” Era

Commercial hardware such as NVIDIA DGX SuperPOD (2021) made training 100‑billion‑parameter models practical. Open‑source model releases – Meta’s OPT‑175B (2022) and HuggingFace’s BLOOM‑176B (2022) – provided publicly available checkpoints for research.

Training frameworks matured:

Megatron‑LM added support for tensor, pipeline, and sequence parallelism.

DeepSpeed refined ZeRO‑3, added FlashAttention and Sequence Parallelism, dramatically reducing memory consumption and improving throughput.

Inference systems caught up. Two fundamental problems for decoder‑only models were solved:

Dynamic batching without excessive padding – addressed by the ORCA continuous‑batching technique (OSDI 2022).

Efficient KV‑cache memory allocation – solved by Paged Attention (SOSP 2023).

Open‑source inference stacks that combined scheduling, quantization, and kernel optimizations appeared:

HuggingFace Text‑Generation‑Inference (TGI)

LMDeploy (InternLM)

vLLM – introduced Paged Attention and achieved high‑throughput serving (2023).

NVIDIA TensorRT‑LLM – integrates TensorRT, Triton‑server, and FasterTransformer.

Quantization methods such as GPTQ and AWQ, as well as advanced scheduling strategies (DistServe, Splitwise) and speculative decoding (Medusa, FastGen) further increased inference efficiency.

2024‑Present – “Silver” Era

Model proliferation continues, but compute resources and pretrained weights are increasingly concentrated in a few large organizations. The market shifts toward Model‑as‑a‑Service (MaaS), making fine‑tuning of open‑source checkpoints expensive and technically demanding. Retrieval‑Augmented Generation (RAG) becomes the dominant downstream approach because it avoids costly full‑model fine‑tuning.

Key trends:

GPU clusters are allocated to a handful of internal pre‑training teams; external teams rely on MaaS APIs.

Inference competition focuses on ultra‑low latency scheduling, aggressive quantization, and memory‑efficient KV‑cache handling.

Algorithmic research (new attention kernels, multimodal tokenizers, agent architectures) remains a fertile area for innovation.

Practical Recommendations for Engineers

Master core infra concepts: tensor/sequence parallelism, ZeRO stages, offloading, and pipeline scheduling.

Contribute to active open‑source projects (Megatron‑LM, DeepSpeed, vLLM, TGI) to stay aligned with state‑of‑the‑art implementations.

Develop a broad skill set that includes cloud provisioning, distributed‑system debugging, and basic AI algorithm knowledge.

Target concrete engineering problems—e.g., implementing continuous batching, designing KV‑cache allocation policies, or integrating quantization pipelines—that directly impact production workloads.

References

[1]

Attention Is All You Need – https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html [2] ELMo – https://arxiv.org/abs/1802.05365 [3] MoE – https://arxiv.org/abs/1701.06538 [4] GShard – https://arxiv.org/abs/2006.16668 [5] MeshTensorFlow – https://proceedings.neurips.cc/paper/2018/hash/3a37abdeefe1dab1b30f7c5c7e581b93-Abstract.html [6] Gopher – https://arxiv.org/abs/2112.11446 [7] Chinchilla – https://arxiv.org/abs/2203.15556 [8] FlexFlow – https://proceedings.mlsys.org/paper_files/paper/2019/hash/b422680f3db0986ddd7f8f126baaf0fa-Abstract.htm [9] GPipe – https://proceedings.neurips.cc/paper_files/paper/2019/hash/093f65e080a295f8076b1c5722a46aa2-Abstract.htm [10] Parameter Server – https://proceedings.neurips.cc/paper/2014/hash/1ff1de774005f8da13f42943881c655f-Abstract.html [11] OneFlow – https://arxiv.org/abs/2110.15032 [12] M6 – https://arxiv.org/abs/2103.00823 [13] GLM – https://arxiv.org/abs/2210.02414 [14] Pangu‑Alpha – https://arxiv.org/abs/2104.12369 [15] PatrickStar – https://arxiv.org/abs/2108.05818 [16] FasterTransformer – https://github.com/NVIDIA/FasterTransformer [17] TurboTransformers – https://github.com/Tencent/TurboTransformers [18] ORCA – https://www.usenix.org/conference/osdi22/presentation/yu [19] Paged Attention – https://dl.acm.org/doi/abs/10.1145/3600006.3613165 [20] Text‑Generation‑Inference – https://github.com/huggingface/text-generation-inference [21] LMDeploy – https://github.com/InternLM/lmdeploy [22] vLLM – https://github.com/vllm-project/vllm