Estimating the GPU Count Needed to Train a Large Language Model

Core Idea

GPU planning reduces to two numbers: the total compute required for training (measured in FLOPs) and the per‑GPU throughput (FLOP/s). With these, the number of GPUs needed for a target training duration T is calculated as: GPUs = C / (GPU throughput × T) Here C is the total training compute, GPU throughput is the sustained FLOP/s of a single GPU, and T is the desired training time in seconds.

Step 1 – Per‑Token Compute Cost (the “6N rule”)

Training proceeds token by token. For a standard Transformer with N parameters, the forward pass costs roughly 2N FLOPs and the backward pass about 4N FLOPs, giving a total of approximately 6N FLOPs per token.

This approximation matches the scaling‑law studies of Kaplan et al. (2020) and Hoffmann et al. (2022).

Step 2 – Total Training Compute

If training runs on D tokens, the total compute is: C ≈ 6 × N × D where N is the model parameter count and D the number of training tokens. Larger models or larger datasets increase C linearly.

Step 3 – GPU Throughput

Peak FLOP numbers from vendor spec sheets are optimistic. Real‑world sustained throughput is typically 30 %–60 % of peak due to memory limits, inter‑GPU communication, and software overhead. A classic reference for realistic efficiency is Amodei & Hernandez (2018).

Example peak numbers (TFLOPS):

K80 → 2.8
K40 → 3.7
M40 → 6.0
P100 → 9.5

Modern GPUs such as A100 or H100 should be substituted with their sustained throughput values using the same method.

Step 4 – Example Calculation

Consider a 1.5 B‑parameter model ( N = 1.5×10⁹) trained on 100 B tokens ( D = 10¹¹) using an NVIDIA P100 (9.5 TFLOPS = 9.5×10¹² FLOP/s). C = 6 × (1.5×10⁹) × 10¹¹ = 9×10²⁰ FLOPs Time on a single P100:

T_single = C / GPU throughput ≈ 9×10²⁰ / 9.5×10¹² ≈ 9.5×10⁷ s ≈ 1,097 days

To finish in 30 days ( T = 30×24×3600 = 2.59×10⁶ s), the required GPU count is: GPUs = (9.5×10⁷) / (2.59×10⁶) ≈ 37 Accounting for realistic Model FLOPs Utilization (MFU) of about 40 % (the typical 30 %–60 % range), the adjusted GPU count becomes roughly 37 / 0.4 ≈ 93 GPUs.

Practical Considerations

The 6N rule is an approximation; it ignores attention softmax, LayerNorm, embeddings, and other overheads, which are usually small but non‑zero.

GPU throughput varies with precision (FP16/BF16), batch size, model architecture, and software optimizations; the TFLOPS figures are only rough guides.

MFU captures the fraction of peak performance actually achieved during distributed training; using a conservative MFU improves the reliability of the estimate.

Useful Tools

LLM Compute‑Optimal Estimator (Hugging Face) : online calculator implementing the 6ND rule.

Spheron AI & LLM Training Cost Calculator : similar inputs with a realistic MFU factor.

LLM Model VRAM Calculator (Hugging Face) : checks whether a model fits in GPU memory.

DeepSpeed (Microsoft) : provides memory‑usage estimates for distributed training.

Conclusion

Starting from the simple formula C ≈ 6 × N × D and a realistic GPU throughput, one can quickly estimate training cost, duration, and the number of GPUs needed. The method yields results within an order of magnitude, sufficient for early‑stage planning; more precise budgeting can later rely on the listed tools.

References:

Kaplan et al. (2020). Scaling Laws for Neural Language Models .

Hoffmann et al. (2022). Training Compute‑Optimal Large Language Models (Chinchilla) .

Amodei & Hernandez (2018). AI and Compute .