Why a Decreasing Loss Isn't Enough: Critical Logs and Metrics for Large‑Model Training

Problem Background

Training large models consumes hundreds of GPUs for weeks and can cost hundreds of thousands of dollars. Most teams focus solely on the loss curve, assuming that a smooth decline means training is normal, but loss reduction is only a necessary, not sufficient, condition.

What to Monitor Besides Loss

Learning‑rate changes (warm‑up, decay)

Gradient norm (stability, explosion, vanishing)

Parameter update magnitude

Validation set metrics (Loss, Perplexity, BLEU, Accuracy)

Training throughput (tokens/s, samples/s, GPU‑hours/token)

GPU utilization and memory usage

Checkpoint status (path, timestamp, size, integrity)

Data loading speed (loader time, num_workers)

Core Knowledge Points

Loss Limitations

Loss is the optimization target, not a direct measure of model quality. Common pitfalls:

Low loss may indicate over‑fitting.

Smooth loss does not guarantee stable gradients; the model might be learning nothing.

Occasional loss spikes can be caused by hard samples, learning‑rate changes, or gradient‑accumulation steps.

Learning‑Rate Scheduling

Typical schedules include Warmup, Cosine Decay, Step Decay, and Linear Decay. Record the learning‑rate at every step and verify that it follows the expected schedule.

lrs = []
for step in range(10000):
    lrs.append(scheduler.get_last_lr()[0])
    scheduler.step()
import matplotlib.pyplot as plt
plt.plot(lrs)
plt.xlabel("Step")
plt.ylabel("Learning Rate")
plt.title("Learning Rate Schedule")
plt.savefig("lr_curve.png")

Gradient Norm Monitoring

Record the global gradient norm each step. Normal range is roughly 0.1‑10. Values < 0.01 indicate vanishing gradients; > 100 indicate explosion.

grad_norm = torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
logger.info(f"Grad norm: {grad_norm:.2f}")

Parameter Update Magnitude

Compute the change in parameter norm before and after an optimizer step. A very small update_norm suggests the learning‑rate is too low.

param_norm_before = torch.norm(torch.stack([torch.norm(p.detach()) for p in model.parameters()]))
optimizer.step()
param_norm_after = torch.norm(torch.stack([torch.norm(p.detach()) for p in model.parameters()]))
update_norm = param_norm_after - param_norm_before

Validation Set Evaluation

Periodically evaluate on a validation set. If training loss keeps dropping while validation loss rises, the model is over‑fitting.

if step % eval_interval == 0:
    model.eval()
    val_loss = 0.0
    val_steps = 0
    with torch.no_grad():
        for val_batch in val_dataloader:
            val_loss += compute_loss(model, val_batch)
            val_steps += 1
    val_loss /= val_steps
    val_perplexity = torch.exp(torch.tensor(val_loss))
    logger.info(f"Validation | Step {step} | Val Loss: {val_loss:.3f} | Val Perplexity: {val_perplexity:.2f}")
    model.train()

Training Throughput Monitoring

Measure tokens processed per second to assess hardware efficiency.

start_time = time.time()
for step, batch in enumerate(dataloader):
    loss = train_step(model, batch, optimizer)
    tokens_processed += batch['input_ids'].numel()
    if step % 10 == 0:
        elapsed = time.time() - start_time
        tokens_per_sec = tokens_processed / elapsed
        logger.info(f"Throughput: {tokens_per_sec:.0f} tokens/s")
        start_time = time.time()
        tokens_processed = 0

GPU Utilization and Memory

Log allocated and reserved memory and monitor utilization with nvidia‑smi. Low utilization may indicate data‑loading bottlenecks or small batch size.

gpu_mem_allocated = torch.cuda.memory_allocated() / 1024**3
gpu_mem_reserved = torch.cuda.memory_reserved() / 1024**3
logger.info(f"GPU Mem: Allocated={gpu_mem_allocated:.1f}GB, Reserved={gpu_mem_reserved:.1f}GB")

Checkpoint Management

Save checkpoints regularly and verify integrity.

if step % checkpoint_interval == 0:
    checkpoint_path = f"/checkpoints/step_{step}.pt"
    torch.save({
        'step': step,
        'model_state_dict': model.state_dict(),
        'optimizer_state_dict': optimizer.state_dict(),
        'scheduler_state_dict': scheduler.state_dict(),
        'loss': loss,
        'grad_norm': grad_norm,
    }, checkpoint_path)
    logger.info(f"Checkpoint saved to {checkpoint_path}")

Overall Troubleshooting Workflow

Continuously monitor loss, learning‑rate, and gradient norm.

Periodically evaluate validation metrics.

Watch throughput and GPU utilization for hardware bottlenecks.

Ensure checkpoints are saved and can be re‑loaded.

If anomalies appear (loss spikes, NaN/Inf, OOM), log detailed context (step, batch size, sample, GPU memory) and stop training to investigate.

Adjust hyper‑parameters (learning‑rate, gradient clipping, batch size) or data pipeline based on the diagnosis.

Resume from the latest valid checkpoint.

Common Issues and Solutions

Loss Not Decreasing

Increase learning‑rate.

Check gradient norms for vanishing.

Inspect data for quality problems.

Loss Becomes NaN/Inf

Reduce learning‑rate.

Increase gradient clipping (e.g., max_norm=0.5).

Enable mixed‑precision training with torch.cuda.amp.

Low GPU Utilization

Increase DataLoader.num_workers.

Raise batch size or use gradient accumulation.

Profile data‑loading time and communication overhead.

Out‑of‑Memory Errors

Decrease batch size.

Use gradient checkpointing or DeepSpeed ZeRO.

Check input sequence lengths.

Unstable Throughput

Fix sequence length (e.g., pad/truncate to a constant).

Pre‑load data into memory or use faster storage.

Switch to a more efficient communication backend (e.g., NCCL).

Monitoring and Alerting

Expose metrics with Prometheus gauges and set up Grafana dashboards for loss, learning‑rate, gradient norm, throughput, GPU utilization, and memory. Example alert rules detect NaN loss, gradient explosion, and low throughput.

from prometheus_client import Gauge, start_http_server
loss_gauge = Gauge('training_loss', 'Training loss')
grad_norm_gauge = Gauge('grad_norm', 'Gradient norm')
throughput_gauge = Gauge('throughput_tokens_per_sec', 'Training throughput')
# Inside training loop:
loss_gauge.set(loss)
grad_norm_gauge.set(grad_norm)
throughput_gauge.set(tokens_per_sec)

Conclusion

Large‑model training requires a holistic monitoring strategy. By tracking learning‑rate, gradient statistics, validation performance, hardware utilization, and checkpoint health, engineers can quickly spot anomalies, prevent wasted resources, and ensure the model converges to a useful solution.