Common Debugging Signals for Large Language Models

Review of Basic Workflow

The training pipeline is broken into six stages: (1) data collection, (2) data preprocessing (cleaning, formatting), (3) repartitioning for distributed processing, (4) training‑data preparation (secondary preprocessing, tokenization with versioned tokenizers, sampling & sharding to produce sharded training data), (5) model training (large‑scale GPU clusters, pre‑training 1‑3 warm‑up/stable/decay rounds, fine‑tuning 1‑3 rounds for QA, preference, reinforcement), (6) model evaluation & storage, followed by (7) log analysis and post‑mortem.

Challenges

Beyond the standard distributed pipeline, developers must rely on frameworks that resolve parallelism issues; the article mentions DeepSpeed, Megatron, and references nanochat source analysis for data‑parallel and optimizer‑parallel techniques.

Simple Experience – Strategies and Common Issues

Debugging Strategies

Start with a small configuration and scale gradually.

Record everything using WandB or TensorBoard.

Monitor gradients and memory usage.

Use a performance profiler to locate bottlenecks.

Take frequent checkpoints for quick recovery.

Apply binary‑search style isolation to pinpoint faults.

Fix one issue at a time.

Debug data and gradients before tackling loss values.

Automate reproducibility with deterministic seeds, fixed configs, and saved states.

Prioritize sanity checks over blind logging.

Convergence Prevention

Run a tiny sanity check (1% of data, 1 GPU, 100 steps) and verify loss decreases.

Gradient clipping: torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0) Mixed‑precision training via torch.cuda.amp or DeepSpeed fp16/bf16.

Deterministic mode: torch.backends.cudnn.deterministic = True Seed fixing: seed_everything(42) (set random, numpy, torch seeds).

Drive runs with Hydra/YAML/JSON for reproducibility.

Common Problems

Loss does not drop after 100 steps (monitor with TensorBoard/WandB).

Gradient norm > 1e3 or < 1e‑6 (exploding/vanishing gradients).

Learning‑rate stalls or explodes (scheduler failure).

GPU memory > 95 % (OOM risk).

NaN/Inf crashes (use torch.autograd.detect_anomaly()).

Golden rule: Never debug a 1000‑step, 64‑GPU run before a 10‑step, 1‑GPU run.

Specific Issues

CUDA Out‑of‑Memory

If the model exceeds GPU memory, reduce batch size or enable mixed‑precision; DeepSpeed ZeRO can also be used.

Performance Bottlenecks

Typical sources include data‑loading lag (GPU idle), communication overhead in multi‑GPU training, and mis‑configured mixed‑precision that prevents expected speed‑ups.

Lack of Contingency Plans

Even a small configuration error can cause severe slow‑downs or total failure; examples listed are communication overhead, inefficient memory management, low compute efficiency, and poor fault tolerance.

Reducing Debug Complexity

Avoid heterogeneous hardware (different GPU architectures, memory configs) and complex framework interactions such as custom PyTorch‑DeepSpeed loops or version mismatches.

Framework Overviews

Torchtitan

Torchtitan offers a PyTorch‑native “one‑stop” solution with modular 3D/4D parallelism (data parallel DP, pipeline parallel PP, tensor parallel TP) via DTensor and DeviceMesh, supporting elastic scaling and hardware‑software co‑optimizations like Float8 training and SymmetricMemory. It provides distributed checkpointing, comprehensive logging, custom GPU‑memory monitoring, and debugging tools. Configuration is TOML‑based; it integrates FSDP2 (per‑parameter DTensor) to scale models 3‑6×, adds activation checkpointing, mixed‑precision support, custom context managers, peak‑memory statistics, NCCL barrier optimizations, and fixes low‑level bugs such as FlashAttention‑2 kernel issues. It also supports ROCm and disables cascade attention to avoid transpose bugs.

veScale + veOmni (ByteDance)

veScale is an open‑source PyTorch‑native LLM training framework that introduces a new RNG algorithm guaranteeing identical results across sharded operators, supports online auto‑resharding checkpoint recovery, and embraces the SPMD paradigm. It works in eager mode, hybrid eager‑compile mode, and provides real‑time monitoring for OOM and gradient issues. veOmni extends veScale to multimodal MoE models (text, image, audio, video) with a recipe zoo that enables high‑efficiency scaling with minimal code changes.

The author hopes newer platforms will lower debugging complexity and integrate more automatic detection mechanisms.

References

DeepSpeed debugging guide (BytePlus)

NVIDIA GPUDirect‑Storage troubleshooting guide

Cameron Wolfe’s LLM debugging post

Medium article on LLM hyper‑parameters

Scaling laws book (jax‑ml)

UVaDL‑C notebooks

Colossal‑AI training blog