Deep Learning Hyperparameter Tuning and Training Tips: Insights from Zhihu Experts

Effective hyperparameter tuning is crucial for deep learning, and this guide aggregates practical tips from several Zhihu experts.

General workflow : Start with a small training set to verify over‑fitting capability; if the model cannot over‑fit, reduce the learning rate or check data loading and model dimensions.

Monitor train and eval loss curves: train loss should decrease log‑like and stabilize, while eval loss may plateau or rise, indicating early stopping points. Abnormal curves often signal data issues such as mislabeled samples.

Use modest dataset sizes (e.g., ~20k training, ~2k validation) for initial tuning, and prefer proven open‑source codebases before building models from scratch.

Optimizers and schedulers : Adam (1e‑3 or 1e‑4) is the default, with Radam as an alternative; avoid plain SGD‑M for speed. Apply torch.optim.lr_scheduler.CosineAnnealingLR with T_max of 32 or 64 to reduce manual learning‑rate tuning.

Apply gradient clipping for RNNs using torch.nn.utils.clip_grad_norm , and initialize parameters with orthogonal (LSTM hidden state), He, or Xavier methods depending on activation functions.

Regularization: use ReLU (or leaky ReLU) activations, batch‑norm and dropout (especially before the final output layer and after embeddings), and consider layer‑norm for RNNs. Adjust batch size (often smaller improves performance) and embedding dimensions (64‑128) while scaling hidden sizes (256‑512).

Model architecture tips: start with 2‑3 LSTM layers, use weight decay (~1e‑4), add shortcuts in CNNs, and increase CNN depth only until performance saturates. GRU and LSTM usually perform similarly.

Data‑centric advice: analyze data first, use pandas describe() for statistics, and ensure padding lengths cover >90% of sequences. Compute per‑token or per‑pixel loss to gauge magnitude.

Learning‑rate warm‑up: begin from a very small value (e.g., 1e‑7) and exponentially increase (e.g., ×1.05) for a few hundred steps, selecting the region with the steepest loss decline.

Common pitfalls: writing models from scratch too early, forgetting gradient clipping (causing NaNs), using large learning rates with tied embeddings, training on full data without a small‑scale benchmark, overlooking hyper‑parameter details in papers, and misinterpreting batch‑norm statistics during early training.

Recommended reading for NLP includes papers on LSTM language models, neural machine translation architectures, transformer training tricks, and RoBERTa. For CV, suggested works cover large‑batch ImageNet training, CNN tricks, object detection freebies, and EfficientNet scaling.

Additional practical notes: Adam can both solve and introduce issues; subword tokenization is generally stable; run GPU errors on CPU for clearer messages; be patient as training may require hours or days; and understand that some metrics lag behind training progress.

Ensemble strategies : vary initialization, combine models from different cross‑validation folds, or linearly fuse heterogeneous models (e.g., RNN with traditional classifiers).