Enterprise LLM Fine-Tuning: End-to-End SFT Pipeline with LoRA/QLoRA

Problem Background and Applicability

General LLMs (Qwen2.5/3, Llama 3/4) often fail in enterprise verticals due to: (1) insufficient business intent alignment with domain terminology and rules; (2) uncontrolled structured output (JSON/YAML/DSL) breaking automation pipelines; (3) factual hallucinations and inability to refuse when evidence is lacking.

Prompt vs RAG vs SFT Decision Tree

The article presents a decision flowchart: if the task requires latest/massive external facts → RAG; if it requires changing model output style/behavior/format → SFT; if neither → zero/few-shot prompt. The recommended hybrid practice: SFT for format rules, CoT, and refusal boundaries; RAG for dynamic knowledge injection at inference.

SFT in LLM Lifecycle

Modern training loop: Pre-training → SFT (instruction tuning) → RLHF/DPO (preference alignment) → Inference/Serving. SFT teaches the model to follow human instructions via high-quality prompt-response pairs.

Core Mechanisms and Algorithm Selection

Full Fine-Tuning vs PEFT

Full fine-tuning updates all parameters, offering highest fitting capacity but requiring ~1 TB VRAM for 70B FP16 (AdamW optimizer states + gradients) and suffering catastrophic forgetting. PEFT (LoRA) freezes backbone and trains only 0.1–1% extra parameters, reducing VRAM 60–80% and producing small adapter weights (tens of MB).

LoRA Mathematical Principle

LoRA assumes weight update ΔW has low intrinsic rank. For pre-trained weight W₀ ∈ ℝ^{d×k}, ΔW = (α/r) B·A where A ∈ ℝ^{r×k} (Gaussian init), B ∈ ℝ^{d×r} (zero init), r ∈ {8,16,32,64}, α typically 2r. The scaling α/r stabilizes training dynamics when r changes.

Target Modules Strategy

Early LoRA only targeted q_proj, v_proj. Modern ablation experiments show applying LoRA to all linear layers (q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj) yields overwhelming downstream improvements for 7B–72B models.

QLoRA Innovations

QLoRA enables 14B on 24GB GPU or 70B on 8×80GB via: (1) 4-bit NormalFloat (NF4) optimized for normally distributed weights; (2) Double Quantization saving 0.37 bits/parameter; (3) Paged Optimizers offloading AdamW states to CPU RAM via unified memory during activation spikes.

Advanced PEFT Variants

DoRA decomposes weight into magnitude and direction, applying LoRA only to direction, matching full fine-tuning trajectory with ~15% extra compute. PiSSA uses truncated SVD to initialize A and B with principal singular vectors/values, accelerating early convergence by 30%+ but more sensitive to learning rate.

Training Environment Planning and VRAM Budgeting

VRAM Formula

Total VRAM = M_model + M_grad + M_optim + M_act. For 7B BF16: model weights 14 GB; gradients 14 GB (full) or 70 MB (LoRA); optimizer states 84 GB (full) or <1 GB (LoRA); activations scale with batch size, sequence length, and attention heads. Gradient checkpointing reduces activation memory >70%.

Hardware Recommendations

Table maps model size, strategy, precision, GPU topology, and parallelism. Example: 7B LoRA BF16 on 1×A100 80GB; 7B QLoRA 4-bit on 1×RTX 4090 24GB; 70B LoRA BF16 on 4×A100 80GB with ZeRO-3; 70B QLoRA on 2×A100 80GB with ZeRO-2.

DeepSpeed ZeRO Stages

ZeRO-1 shards optimizer states (75% optimizer VRAM saved). ZeRO-2 shards optimizer + gradients, recommended for enterprise LoRA multi-GPU. ZeRO-3 shards optimizer, gradients, and model weights, for 70B+ or full fine-tuning with 50% more communication.

Infrastructure and Dependency Hardening

Environment: Ubuntu 22.04/Rocky 9, NVIDIA driver ≥550.54.14, CUDA 12.4+, Python 3.11, PyTorch ≥2.4.0, Transformers ≥4.45.0, TRL ≥0.9.6, PEFT ≥0.12.0, DeepSpeed ≥0.15.0, LlamaFactory latest, FlashAttention-2 compiled. Provided bash script uses uv for isolated environment setup and verification.

Data Engineering: From Raw to High-Quality SFT Set

Format Standardization

ShareGPT/ChatML multi-turn format with system/user/assistant roles. Example shows a DevOps audit conversation with system prompt, user script, and assistant risk analysis + fixed script.

Special Tokens and Chat Template

Qwen uses 【role; Llama 3 uses <|begin_of_text|><|start_header_id|>role. Critical: never manually concatenate special tokens; rely on tokenizer.apply_chat_template at training load time.

Loss Masking (Train-on-Completion)

Must not compute loss on system prompt and user query tokens. Labels for prompt portion set to -100 (ignore_index). LlamaFactory automates this via train_on_prompt: false.

Data Scale and Quality Control

Format/style alignment: 500–2,000 high-quality human-reviewed samples. Complex reasoning: 5,000–20,000 expert-verified multi-turn dialogues. Cleaning pipeline: filter short responses (<20 tokens), remove HTML/Base64/garbage, semantic deduplication via MinHash LSH (Jaccard >85%).

Validation Script

Python script audits JSONL for syntax, required fields (messages with user+assistant), non-empty content, token length distributions, and exits with error on any malformed sample.

Enterprise Fine-Tuning with LlamaFactory

Dataset Registration

Add entry to data/dataset_info.json with file_name, formatting: sharegpt, columns mapping, and role/content tags.

Production Training Config (YAML)

Key parameters: model_name_or_path, stage: sft, finetuning_type: lora, lora_rank: 64, lora_alpha: 128, lora_dropout: 0.05, lora_target: all, flash_attn: fa2, gradient_checkpointing: true, deepspeed: ZeRO-2 config, per_device_train_batch_size: 2, gradient_accumulation_steps: 8 (effective batch 64 on 4 GPUs), learning_rate: 1.5e-4, cosine scheduler, warmup_ratio: 0.05, num_train_epochs: 3.0, eval_strategy: steps, eval_steps: 100, save_strategy: steps, load_best_model_at_end: true, metric_for_best_model: eval_loss.

DeepSpeed ZeRO-2 Config

JSON with stage 2, allgather_partitions, reduce_scatter, overlap_comm, contiguous_gradients, bf16 enabled.

Distributed Launch Script

Sets CUDA_VISIBLE_DEVICES, NCCL env vars, runs llamafactory-cli train with tee to log file.

Training Observability and Metric Interpretation

Loss Curve Diagnosis Matrix

Healthy: train loss drops from 2.0+ to 0.4–0.6, val loss tracks closely. Overfitting: train loss → 0.1, val loss rises → early stop, reduce epochs or increase LoRA dropout. Underfitting: train loss stalls >1.5 → check learning rate (should be 1e-4–2e-4), label masking, weight decay. Gradient explosion/NaN: enable max_grad_norm, use BF16, check for ultra-long sequences.

Enterprise Evaluation System (LLM-as-a-Judge)

Three-Layer Pyramid

Bottom (30%): objective benchmarks (regex, JSON Schema, syntax compile). Middle (60%): LLM-as-a-Judge batch scoring on instruction adherence, technical accuracy, risk perception, conciseness. Top (10%): human spot-check on real blind traffic.

Golden Benchmark Construction

100–200 cases physically isolated from training: 60% baseline typical, 20% long-tail ambiguous, 20% safety/adversarial.

Automated Judge Script

Uses GPT-4o/Claude 3.5 via OpenAI API with structured JSON output prompt. Evaluates each dimension 1–5, computes overall score. Includes position-swapping and human calibration to mitigate judge bias.

Weight Merging and Inference Deployment

Merge Principle

W_merged = W₀ + (α/r) B·A. Merged model becomes standard weights with zero inference overhead. Export config specifies base model, adapter path, export_dir, export_size: 4 (GB shards), export_device: cpu.

vLLM Verification

Launch vllm serve with merged model, test via curl to /v1/chat/completions.

Production Troubleshooting and Root-Cause Loops

OOM at Startup

Check zombie processes (fuser -v /dev/nvidia*), enable gradient_checkpointing, reduce per_device_train_batch_size to 1–2 with proportional gradient_accumulation_steps increase, lower cutoff_len.

Loss Flatline

Check labels all -100 (print non -100 count), verify LoRA attached (model.print_trainable_parameters() >0%), ensure learning rate ~1e-4.

Catastrophic Forgetting

Caused by overfitting on vertical data without replay. Fix: mix 15–20% general dialogue data (Alpaca-GPT4, COIG), reduce epochs to 1–2, lower LoRA rank to 16–32, add generic regression tests (GSM8K, MMLU) requiring <5% drop.

Security and Model Asset Versioning

PII redaction via regex/NER before training. Semantic versioning: {base}-{size}-{domain}-{method}-{gitCommit}-{version}. TRAIN_METADATA.json records base weight SHA256, dataset snapshot hash, eval scores, full training config.

Version Baseline: Lock Dependencies First

Compatibility traps: transformers 4.46+ chat template changes, PEFT 0.11+ target_modules matching, TRL 1.x SFTTrainer interface overhaul, DeepSpeed-torch version binding. Provide verified baseline (Python 3.11, PyTorch 2.6, Transformers 4.56, TRL 1.x, PEFT 0.17, DeepSpeed 0.16, LlamaFactory 0.9.5) and requirements.lock generation/installation scripts. Environment self-check script validates versions, CUDA, BF16 support, FlashAttention, bitsandbytes 4-bit, DeepSpeed ops.

Data Pipeline Deepening: Four Stages

Raw → Normalize (line endings, HTML tags, PDF line breaks, punctuation) → Filter (length, repetition, placeholders, refusal rate, language mix, token limit) → Deduplicate (exact SHA256 + MinHash LSH Jaccard 0.85) → Split (stratified by business tag, fixed seed, same conversation not across splits). Each stage outputs versioned artifacts with manifest (counts, filters, SHA256).

Loss Masking Implementation Details and Verification

Three paths: manual masking, LlamaFactory auto (train_on_prompt: false, mask_history: true/false), TRL auto (assistant_only_loss). Verification script prints token counts, confirms masked prefix ends at 【assistant , ensures training tokens = assistant tokens + EOS. Multi-turn masking must cover each assistant turn unless mask_history enabled. EOS token presence mandatory (Qwen: 【, Llama: <|eot_id|>).

VRAM Budget Calibration

Formula underestimates by 20–40% due to optimizer temp buffers, comm buffers, fragmentation, CUDA context (~300–600 MB). Plan with 1.3× formula + 10% margin. VRAM probe callback logs allocated/reserved/peak per GPU per step. Downgrade sequence: gradient checkpointing → batch size 1 + accumulation → cutoff_len → LoRA rank → QLoRA → ZeRO-3/offload. Real 7B LoRA on A100 80GB: no checkpoint 62 GB, with checkpoint 24 GB, BF16 22 GB, 4-bit 13 GB.

Training Config Interpretation

Learning rate: LoRA 1e-4–2e-4 (r=8–32), 5e-5–1e-4 (r=64–128), QLoRA 1e-4–3e-4, DoRA 5e-5–1e-4. Effective batch = per_device × accumulation × GPUs, target 32–128. Epochs 1–3, stop at val loss inflection. Warmup 0.03–0.1, cosine scheduler (linear if <500 steps). Pre-flight checklist includes template match, lora_target=all, train_on_prompt=false, cutoff_len coverage, LR magnitude, effective batch, gradient_checkpointing, BF16, eval/save strategy alignment, save_total_limit, report_to, persistent output_dir.

Observability and Intervention

Six curves: learning_rate (smooth decay), grad_norm (0.1–2, spikes indicate bad samples), loss (trend down), VRAM peak (stable), tokens/sec (stable), eval_loss (tracks train). Log analysis via grep/awk to find high grad_norm steps. Resume training requires optimizer.pt, scheduler.pt, trainer_state.json, rng_state.pth. Preemption protection: save_steps 50–100, save_total_limit ≥3.

Evaluation Implementation: From Scoring to Decision

Must answer: (1) Did fine-tuning help? (2) Any side effects? (3) Worth deploying? Objective layer: JSON Schema validation, bash -n, sqlparse, keyword presence, refusal detection. LLM-as-a-Judge layer: mitigate length/style bias via prompt, position swap, human calibration (Cohen's kappa), lock judge version, ensure dimension independence. Human blind test: 50–100 samples, shuffled, at key milestones. Regression check: run generic subset (common sense, arithmetic, reasoning) comparing base vs fine-tuned, fail if drop >5%. Evaluation report structure: version info, objective metrics, subjective scores, failure case analysis, go/no-go conclusion.

Deployment Integration: Merge, Quantize, Serve

Merge vs dynamic LoRA: merge for zero overhead and broad engine compatibility; dynamic for multi-tenant with single base. vLLM dynamic LoRA via --enable-lora --lora-modules. Merge verification: file manifest, weight index consistency, tokenizer load, meta device structure check. Quantization order: BF16 baseline → FP8 (if <2% diff) → 4-bit AWQ/GPTQ (if <5% diff, watch format adherence). Smoke test: format compliance, safety refusal, script dry-run habit.

Additional Failure Scenarios

Training loss OK but inference poor → template mismatch (train vs inference chat_template). Multi-GPU speed far below linear → check GPU utilization (nvidia-smi dmon), data loader CPU, NCCL_DEBUG=INFO for NET/Socket vs IB, ZeRO-3 overhead, cutoff_len variance. Repetition loops → clean training data repetition, ensure EOS in every sample, add repetition_penalty 1.05–1.1, include stop tokens.

Cost Accounting and Resource Scheduling

7B LoRA true cost: 32 GPU-hours, 15–40 person-days data prep, 3–5 person-days eval, 2–3× training for iteration, 2 person-days deploy. Data prep often largest hidden cost. Kubernetes PriorityClass for production vs experimental, activeDeadlineSeconds, LoRA on L40S/L20, full fine-tune on H100. When not to fine-tune: knowledge injection (use RAG), <300 samples, no eval set, no ground truth from business, mere politeness (preference alignment).

Summary

Lock versions first: dependencies, data, config, seed.

Data pipeline staged, each stage auditable.

Verify loss masking by inspecting rendered sequences.

Plan VRAM by measured peaks, downgrade stepwise.

Evaluation must answer utility, side effects, ROI.

Post-merge verification mandatory (chat_template loss is silent killer).

Fine-tuning is a continuous pipeline: data versions, model versions, eval versions.

Code example

[业务场景需求评估]
|
+--------------------+--------------------+
|                                         |
[需要引入最新/海量外部事实?]              [需要改变模型的输出风格/行为/格式?]
|                                         |
+--------+--------+                       +--------+--------+
|                 |                       |                 |
(是)               (否)                    (是)               (否)
v                 v                       v                 v
[ RAG 检索增强 ]   [ Prompt 提示词工程 ]    [ SFT 监督微调 ]    [ 零样本/少样本 Prompt ]
|                 |                       |
+--------+--------+                       |
|                                |
+--------------------------------+
|
v
[ 混合架构: RAG + SFT ]
(SFT 规范格式与引用规范，RAG 动态注入领域上下文事实)