Turning LLM Fine‑Tuning into a Skill‑Building Journey: Practical Strategies

Data Work

Approach 1: Directly inherit training data from a lab or colleague and use it without checking quality.

Approach 2: Download an open‑source dataset and construct a "system + query + answer" collection.

Approach 3: Generate data with GPT‑4, craft diverse prompts, deliberately add noisy prompts for robustness, and meticulously review each data point while aligning labeling standards with annotators.

Approach 4: Leverage user interaction logs to drive data construction, extracting real user prompts and using rules or GPT‑4 to analyze feedback for high‑quality answers.

Approach 5: Apply ideas from CoT, RAG, function calls, agents, etc., to decompose tasks that the model cannot handle directly at the data level (e.g., "model writes novel outline → model expands outline into full novel").

Training Code

Approach 1: Inherit existing training scripts, modify data_path, and run bash train.sh.

Approach 2: Download a training codebase, study every launch parameter (e.g., why enable offload, meaning of sequence_parallel), examine the dataloader, loss calculation per epoch, and special token handling.

Approach 3: Beyond understanding parameters, question choices such as epoch count, dataset size, number of special tokens, learning rate for a 7B model, warm‑up steps, and seek answers from ChatGPT or relevant papers.

Approach 4: Critically evaluate and improve the training pipeline, e.g., compare DeepSpeed vs. Megatron, combine their strengths, and profile bottlenecks such as RoPE overhead versus attention.

Experiment Analysis

Approach 1: Run a prepared evaluation set; if results improve, the work is done. If not, treat it as a data‑quality issue, clean data, or generate more data, focusing on tasks with poor metrics.

Approach 2: Compare pre‑train and SFT base models, categorize bad cases (hallucination, pattern over‑fit, insufficient training, model capacity limits), and prioritize debugging accordingly.

Based on analysis, design experiments: up‑sample data for under‑fitted tasks, create prompt variations to test over‑fitting, or benchmark against other same‑size chat models (LLaMA, Qwen, Mistral, DeepSpeed‑based models).

Let the pre‑train model continue generation to check whether a capability is missing or was overwritten during fine‑tuning.

Observe token‑level probabilities and the token position where errors start.

When the model outputs malformed JSON, feed the expected json snippet back to see its continuation behavior.

Analyze “bad cases” such as answering "Beijing" for "Japan's capital" by inspecting token over‑fit and training corpus frequency.

Approach 3: Correlate model results with training logs, TensorBoard metrics, and loss curves; ask why initial SFT loss is high, whether too many special tokens or overly creative tasks cause it, and monitor loss thresholds (e.g., < 0.5 may indicate over‑fitting), channel loss trends, and epoch‑level differences.

Approach 4: Run benchmark suites to evaluate general abilities, identify any capability drop (e.g., math vs. creativity), and study the trade‑off between task‑specific fine‑tuning and catastrophic forgetting.

Overall, the article emphasizes that while many “quick‑fix” methods can achieve comparable model performance, the depth of investigation and iterative experimentation determines the real technical growth of the practitioner.