Why Training Transformers Faces an Impossible Triangle of Speed, Performance, and Cost

Impossible Triangle in Transformer Training

Training large Transformer models involves a trade‑off among three factors: training speed (efficiency), model performance (quality), and computational cost (money). Because self‑attention has O(n²) time and memory complexity, improving any two dimensions inevitably degrades the third.

Intuitive Analogy

Similar to the “fast, good, cheap” dilemma, a practitioner can only satisfy two of the three goals at once.

Increasing model size or data improves performance but slows training and raises GPU count and cost.

Reducing parameters or using lower precision speeds up training and cuts cost but typically lowers accuracy.

Adding more GPUs accelerates training and keeps performance but increases expense.

Concrete Examples

GPT‑3 / GPT‑4 : Hundreds of billions of parameters, terabytes of text, and months of training on tens of thousands of GPUs. Achieves state‑of‑the‑art performance but consumes electricity comparable to a small town and costs billions of dollars.

BERT‑base : ~110 M parameters, trained on 16 TPU‑v3 chips for 4 days. Offers a balance between cost and performance for many downstream tasks.

Mobile‑size models (e.g., DistilBERT, MobileBERT): less than 100 M parameters, can be fine‑tuned on a single GPU in hours, but their downstream accuracy lags behind large models.

Rapid fine‑tuning approaches (e.g., LoRA, adapters): keep training time and cost low by freezing most weights, yet the resulting models inherit the base model’s limitations.

Technical Breakdown

Model Performance (Quality)

Parameter count up to 10⁹–10¹¹.

Training corpus size in the terabyte range.

Training duration of weeks to months.

Training Speed (Efficiency)

Parameter reduction (pruning, quantization) – may degrade quality.

Data‑parallel or model‑parallel scaling across many GPUs – increases hardware cost.

Mixed‑precision (FP16/BF16) – reduces memory bandwidth but can affect numerical stability.

Computational Cost (Money)

GPU/TPU count directly impacts hourly cost.

Model compression techniques lower inference cost but often reduce accuracy.

Shortening training epochs saves money but may under‑fit the data.

Implications for Practitioners

When selecting a model or training strategy, engineers must decide which two vertices of the triangle to prioritize based on application constraints (latency, budget, required accuracy). For example, a research prototype may favor performance and speed, accepting high cost, whereas an on‑device application may prioritize cost and speed, sacrificing some accuracy.

Reference

https://arxiv.org/pdf/2204.06130