LoRA/QLoRA Fine-Tuning on a Single GPU: Practical Guide for 7B/13B Models

1. What a Single 4090 Can and Cannot Do

A team needed to fine-tune a domain model for customer support using 12,000 instruction pairs. They had one RTX 4090 (24GB VRAM). Initial attempt to QLoRA fine-tune a 13B model failed with OOM. Reducing batch size to 1 and max_length to 1024 allowed training but at ~200 steps/hour — too slow. Switching to 7B improved stability but still suffered occasional OOM and loss spikes due to gradient accumulation steps=16 and max_length=1024 truncating long samples (2-3k tokens). Three core problems emerged: unclear VRAM budget, ignoring sequence length's impact on VRAM, and wrong model/config selection.

This article follows a reproducible pipeline: calculate VRAM → choose config → prepare data → train correctly → evaluate rigorously → deploy. Focus is on reasoning behind each parameter, not just runnable code.

2. Scope and Version Baseline (2026-11)

2.1 Applicable Scenarios

Single or 2-4 consumer/workstation GPUs (RTX 3090/4090, L40S, A10, RTX A6000) fine-tuning 7B-13B models.

1,000-50,000 domain instruction samples to change output style, format, domain knowledge expression.

Deployment constrained, need adapter-based multi-task model versioning.

Existing SFT experience, need engineering-grade pipeline: reproducible, evaluable, rollbackable.

2.2 Version Baseline

Component      Version      Key Differences
PyTorch      2.5-2.9      2.6+ torch.load defaults weights_only=True
transformers 4.5x/5.x    5.x breaking changes in Trainer params & device management
peft         0.21.x       Supports use_dora, use_rslora, target_modules="all-linear"
bitsandbytes 0.45+        4bit/8bit need Ampere+ (compute capability ≥8.0)
accelerate   1.x          device_map & distributed launch changes
trl          0.2x         SFTTrainer param names changed across versions
flash-attn   2.7+/3.x     Strict CUDA/PyTorch/gcc/GPU arch matching
vLLM         0.11+        For merged model inference verification & deployment

2.3 Compatibility Must-Checks

bitsandbytes 4bit requires GPU compute capability ≥8.0 (Ampere+). V100 (7.0) errors or falls back to slow software path. T4 (7.5) also limited. Check with nvidia-smi --query-gpu=name,compute_cap --format=csv.

flash-attn is optional . It reduces attention VRAM and speeds up but compilation is environment-sensitive. If install fails, use attn_implementation="sdpa" (PyTorch built-in SDPA), acceptable performance loss, much better stability.

transformers 5.x + peft interface compatibility must be verified per actual versions. peft 0.21 + transformers 5.x is current mainline; older peft may raise AttributeError.

2.4 Out of Scope

Full-parameter fine-tuning (7B BF16 needs 100GB+ VRAM).

Pre-training / continued pre-training (LoRA unsuitable for new knowledge systems).

70B+ multi-node training (needs FSDP/DeepSpeed).

Inference quantization deployment details (only merge + vLLM verification covered).

3. Core Knowledge: What LoRA/QLoRA Actually Save

3.1 LoRA Principle & VRAM Accounting

LoRA freezes base weights W and adds low-rank bypass: h = Wx + (α/r)·BAx. Trainable params drop from d_out×d_in to r×(d_in+d_out). For Qwen2.5-7B q_proj (3584×3584), full params 12.8M, LoRA r=16 gives 114k (0.89% of layer). Full linear LoRA across model ~40M params (0.53% of 7.6B).

Real VRAM saving comes from not storing gradients and optimizer states for frozen weights. Example 7B BF16 AdamW:

Item                Full Fine-tune   LoRA (r=16, all-linear)
Weights (BF16)      14 GB            14 GB
Gradients (BF16)    14 GB            ~0.08 GB
Optimizer (FP32 m/v) 56 GB           ~0.32 GB
Activations         Large            Large (unchanged)
Total (excl. act.)  ~84 GB           ~14.4 GB

Activations remain the dominant factor, driven by batch_size × seq_len × hidden × layers. Hence reducing batch and max_length is first priority.

3.2 QLoRA Three Techniques

Technique          What It Does                              VRAM Saved
NF4 Quantization   Map weights to 16 optimal quantiles       Weights 14GB → ~4.2GB
Double Quantization Quantize quantization constants (scales) ~0.37 bit per 64 weights
Paged Optimizer    Offload optimizer states to CPU via unified memory Avoids VRAM spikes during gradient checkpointing

Measured VRAM (7B, batch=1, seq=2048, grad checkpoint):

Method          Weights   Optimizer/Grad   Activations   Total   24GB Feasible
Full BF16       14 GB     70 GB            ~6 GB         ~90 GB  No
LoRA BF16       14 GB     0.4 GB           ~6 GB         ~20.4 GB Yes (tight)
QLoRA NF4       4.2 GB    0.4 GB           ~6 GB         ~10.6 GB Yes (comfortable)

Optimizer/grad 0.4 GB = 40M trainable params × 2 bytes (BF16 grad) + 8 bytes (FP32 m/v) ≈ 400MB. If actual much higher, check target_modules not defaulting to full-parameter.

Trade-off: QLoRA training 20-40% slower due to repeated dequantization in forward pass.

3.3 7B/13B Real Feasibility on 24GB

Model   Method        Max Seq (batch=1, grad ckpt)   Throughput (tok/s)   Verdict
7B      LoRA BF16     2048-3072                      1500-2500            Feasible, seq limited
7B      QLoRA NF4     4096-6144                      1000-1800            Recommended, seq headroom
13B     LoRA BF16     1024-1536                      700-1200             Barely, must reduce seq
13B     QLoRA NF4     2048-3072                      500-900              Feasible, long samples truncated
13B     QLoRA +long   4096 (batch=1 + optimizations) 400-700              Boundary, must benchmark

Training throughput ≠ inference throughput (training includes backward pass, ~10x slower). Ranges vary 1.5-2x due to attn_implementation, grad checkpoint, batch size — hence self-benchmarking required.

Prerequisites for above table (all mandatory): grad checkpoint (+30% time), optim=paged_adamw_8bit or adamw_torch, bf16=True (Ampere+), grad_checkpointing_kwargs={"use_reentrant":False}, reserve 0.5-1GB for display.

Validation method: print free/total VRAM at script start, periodically log peak_alloc/peak_reserved during training. Use PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to mitigate fragmentation.

3.4 Which Layers Get LoRA

target_modules choice directly determines effect and param count. Qwen2.5-7B (28 layers, hidden 3584, KV heads 512, FFN 18944) with r=16:

Config       Modules                              Trainable Params   % of 7.6B   Effect          Speed
Minimal      q_proj, v_proj                       ~5.0M              ~0.07%      Adequate        Fastest
Common       q_proj, k_proj, v_proj, o_proj       ~10.1M             ~0.13%      Better          Fast
All-linear   + gate_proj, up_proj, down_proj      ~40.4M             ~0.53%      Best, near full-param Slower

All-linear adds 3 MLP modules, 4x params of 4-module but still only ~40M (77MB BF16). Real cost is compute: each layer adds three LoRA matrix multiplications, training speed drops 10-20%. Heuristics: <5k samples/simple task → minimal or 4-module; >10k samples/domain expression → all-linear; VRAM tight → reduce r and modules first, not max_length.

3.5 r and alpha Relationship

Effective scaling = alpha / r. Common alpha = 2×r (scale=2). Too small (r=64, alpha=8) → weak bypass, needs more steps. Too large (r=8, alpha=64) → early loss spikes, possible divergence. r not bigger-is-better: larger r expresses more complex delta but overfits small data. 10k samples: r=128 often worse than r=32 + more epochs.

3.6 When NOT to Fine-tune

Scenario                              Better Approach          Reason
Need latest/private structured data   RAG                    Fine-tuning can't "query today's data", high update cost
Just want output format compliance    Prompt + JSON Schema   Good prompt solves 80% format issues
Massive frequently changing knowledge RAG + small model      Fine-tuning bakes knowledge, retrain per change
Require strict factual accuracy       RAG + citation verify  Fine-tuned models still hallucinate, more confidently
Data < 500 samples                    Few-shot + Prompt      Too little data for stable LoRA patterns

Fine-tuning excels at: fixing output style/phrasing, stable specific-format generation, correct domain terminology, refusal boundaries, distilling capabilities to smaller models.

4. Reproducible Fine-tuning Pipeline

4.1 Six Stages

Data Prep → Baseline Eval → Training → Offline Eval → Merge & Quantize → Online Canary
   ↑           ↑            ↑           ↑              ↑                ↑
Data Card   No-FT Baseline VRAM/Speed   vs Baseline    Consistency &   A/B & Rollback
            Benchmark   Measurement   & Regression   Perf

Most skipped: baseline evaluation. Without baseline, cannot know if fine-tuning helped or hurt. Many teams deploy after spot-checking a few samples, only to see metrics drop in production.

4.2 Acceptance Criteria per Stage

Stage            Acceptance Criteria                              If Not Met
Data Prep        Train/val/test split done; test set isolated; format 100% pass   Clean or resample
Baseline Eval    Base model scores on test set recorded           Must do, else no gain judgment
Training         Loss stable down, no NaN, VRAM peak <90%, resumable   Adjust params or change approach
Offline Eval     Test set improvement vs baseline; regression set no regress   Adjust data/hyperparams or abandon FT
Merge & Quantize Merged output matches adapter inference; perf meets target   Check merge precision
Online Canary    Pass human spot-check + business metrics           Rollback to base + prompt

4.3 VRAM Adjustment Order (least to most impact on quality)

Enable gradient checkpointing (saves 40-60% activation VRAM, ~30% slower).

Reduce per_device_train_batch_size to 1, use gradient_accumulation_steps for effective batch.

Switch to QLoRA 4bit (weights VRAM 1/3, 20-40% slower).

Reduce max_length (directly cuts samples, biggest quality impact, last resort).

Reduce target_modules (all-linear → 4-module).

Reduce r.

Switch model (13B → 7B or smaller same-gen).

Don't: switch optim to adafactor just to save VRAM — unstable convergence on small batch.

5. Hands-on: Environment & Data

5.1 Environment Setup

python -V  # Python 3.10.14
nvidia-smi --query-gpu=name,memory.total,compute_cap --format=csv
# NVIDIA GeForce RTX 4090, 24564 MiB, 8.9

Virtual env & pinned dependencies (exact versions critical due to tight coupling):

python -m venv /opt/venvs/ft
source /opt/venvs/ft/bin/activate
pip install torch==2.7.1 torchvision torchaudio --index-url https://download.pytorch.org/whl/cu128
pip install "transformers==4.57.1" "peft==0.21.2" "bitsandbytes>=0.45" \
  "accelerate>=1.0" "datasets>=3.0" "trl>=0.12" sentencepiece protobuf
pip freeze > /opt/venvs/ft/requirements.lock

Why pin exact versions: transformers minor upgrades often change Trainer param names and device_map behavior; trl changes param names even more frequently. Using >= leads to "same code breaks on new machine" after a week. Lock file ensures reproducibility.

Skip flash-attn: optional speedup, compilation fragile. Use attn_implementation="sdpa" (≤10% perf loss, far more stable). If needed, test in separate env.

Verify key capabilities:

import torch, transformers, peft, bitsandbytes as bnb
print("torch", torch.__version__, "cuda", torch.version.cuda)
print("cuda_available", torch.cuda.is_available())
print("capability", torch.cuda.get_device_capability(0))
print("transformers", transformers.__version__)
print("peft", peft.__version__)
print("bnb", bnb.__version__)
print("bnb_4bit_supported", torch.cuda.get_device_capability(0)[0] >= 8)

Expected: torch 2.7.1 cuda 12.8, capability (8,9), transformers 4.57.1, peft 0.21.2, bnb 0.48.0, bnb_4bit_supported True.

5.2 Data Format & Validation

Standard three-field JSONL per line:

{"instruction": "...", "input": "...", "output": "..."}

Validation script (run before training): checks required fields, non-empty strings, JSON validity, duplicate detection, character length stats (p50, p90, p99, max). Example output: valid=11842 blank=0 duplicate=37 problems=0 chars p50=486 p90=1320 p99=2840 max=6120.

Interpretation: p99=2840 means max_length=2048 truncates ~1% samples (acceptable). If p90 exceeds max_length, truncation too high → adjust. Duplicates must be removed (cause overfitting). Max=6120 is long tail — handle separately.

Token-length measurement (more accurate): use tokenizer. Chinese ~1 char ≈ 0.6-1 token.

5.3 Data Splits

import json, random
def load(path):
  with open(path, encoding="utf-8") as f:
    return [json.loads(l) for l in f if l.strip()]
rows = load("all.jsonl")
random.seed(42)
random.shuffle(rows)
n = len(rows)
n_test = max(200, int(n*0.05))
n_val = max(100, int(n*0.03))
test, val, train = rows[:n_test], rows[n_test:n_test+n_val], rows[n_test+n_val:]
for name, data in (("train",train),("val",val),("test",test)):
  with open(f"{name}.jsonl","w",encoding="utf-8") as f:
    for r in data: f.write(json.dumps(r,ensure_ascii=False)+"
")
  print(name, len(data))

Output: train 11216, val 336, test 290. Three rules: fixed seed for reproducibility; test set physically isolated (training script only reads train/val); if categorical, use stratified sampling.

5.4 Format to Chat Template

Use model's own chat_template, not hand-written. Example for Qwen2.5-7B-Instruct:

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-7B-Instruct")
def to_text(row):
  messages = [
    {"role":"system","content":"You are a device support engineer, give executable troubleshooting steps."},
    {"role":"user","content": row["instruction"] + ("
"+row["input"] if row.get("input") else "")},
    {"role":"assistant","content": row["output"]},
  ]
  return {"text": tok.apply_chat_template(messages, tokenize=False)}
print(to_text(train[0]))

Output includes special tokens <|im_start|>system...<|im_end|> etc.

Critical: compute loss only on assistant tokens. Full-sequence loss wastes capacity learning user phrasing. Use TRL's assistant_only_loss=True (newer versions) or custom label masking with -100 (CrossEntropyLoss ignore_index). If labels all -100 (sample truncated to only prompt), discard sample to avoid NaN loss.

5.5 VRAM Benchmark Before Training

Don't start training blind. Probe script measures peak VRAM for given (seq_len, batch, 4bit) combo:

# probe_memory.py
import os, torch
os.environ.setdefault("PYTORCH_CUDA_ALLOC_CONF","expandable_segments:True")
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
MODEL = "Qwen/Qwen2.5-7B-Instruct"
SEQ_LEN = int(os.environ.get("SEQ_LEN","2048"))
BATCH = int(os.environ.get("BATCH","1"))
LOAD_4BIT = os.environ.get("LOAD_4BIT","1") == "1"
bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
  bnb_4bit_use_double_quant=True, bnb_4bit_compute_dtype=torch.bfloat16) if LOAD_4BIT else None
model = AutoModelForCausalLM.from_pretrained(MODEL, quantization_config=bnb,
  torch_dtype=torch.bfloat16 if bnb is None else None, device_map={"":0}, attn_implementation="sdpa")
model.gradient_checkpointing_enable()
model.enable_input_require_grads()
tok = AutoTokenizer.from_pretrained(MODEL)
ids = torch.randint(100, 30000, (BATCH, SEQ_LEN), device="cuda")
attn = torch.ones_like(ids)
model.train()
out = model(input_ids=ids, attention_mask=attn, labels=ids)
out.loss.backward()
print(f"seq_len={SEQ_LEN} batch={BATCH} 4bit={LOAD_4BIT}")
print(f"loss={out.loss.item():.4f}")
print(f"peak_alloc={torch.cuda.max_memory_allocated()/1024**3:.2f}GiB")
print(f"peak_reserved={torch.cuda.max_memory_reserved()/1024**3:.2f}GiB")
free,total = torch.cuda.mem_get_info()
print(f"free_now={free/1024**3:.2f}GiB total={total/1024**3:.2f}GiB")

Run combos, build local table. Example results:

seq_len batch 4bit  peak_alloc
1024    1     yes   7.9 GiB
2048    1     yes   9.87 GiB
2048    2     yes   12.1 GiB
4096    1     yes   13.6 GiB
2048    1     no    18.4 GiB
4096    1     no    OOM

Decision rule: target peak_alloc ≤ 85% total VRAM (leave headroom for fragmentation), under constraint that max_length covers data p99. Choose max batch, use gradient accumulation if needed.

6. Hands-on: Training

6.1 LoRA Training Config & Script

Full script train_lora.py uses env vars for all hyperparameters. Key sections: quantization config, model loading with device_map={"":0} and attn_implementation="sdpa", tokenizer pad_token=eos_token, padding_side=right. For 4bit: prepare_model_for_kbit_training with use_gradient_checkpointing=True. LoRA config: r=16, alpha=32, dropout=0.05, target_modules all seven linear modules. Data loading via datasets.load_dataset, map to_text, filter by token length (log filtered count). TrainingArguments: per_device_train_batch_size=2, gradient_accumulation_steps=8 (effective batch 16), lr=2e-4, cosine scheduler, warmup_ratio=0.03, weight_decay=0.01, max_grad_norm=0.3, bf16=True, logging_steps=10, save_steps=200, save_total_limit=2, eval_steps=200, optim=paged_adamw_8bit (4bit) else adamw_torch, gradient_checkpointing=True with use_reentrant=False, packing=False, seed=42, data_seed=42. Trainer.train(resume_from_checkpoint=RESUME_FROM).

Launch command:

MODEL_ID=Qwen/Qwen2.5-7B-Instruct OUTPUT_DIR=/data/ft/qwen25-7b-lora-v1 \
MAX_LEN=2048 BS=2 GA=8 USE_4BIT=1 LORA_R=16 LR=2e-4 EPOCHS=3 \
python train_lora.py 2>&1 | tee /data/ft/logs/train-v1.log

Parameter rationale table (excerpt):

Parameter                          Value   Reason
per_device_train_batch_size        2       From 5.5 probe table, with accumulation to 16
gradient_accumulation_steps        8       Effective batch 16, sufficient for small dataset
learning_rate                      2e-4    LoRA typical start (10x full-finetune)
warmup_ratio                       0.03    Prevent early large gradients damaging adapter
max_grad_norm                      0.3     More conservative than default 1.0, stable for QLoRA
lr_scheduler_type                  cosine  Smooth convergence
save_steps                         200     Checkpoint granularity; ~700 steps/epoch
gradient_checkpointing_kwargs      use_reentrant=False  Works with prepare_model_for_kbit_training
packing                            False   Variable-length packing breaks sample boundaries, complicates eval
bf16                               True    Ampere+ stable; Turing use fp16=True

6.2 Training Monitoring

Healthy signs: trainable params ~40.37M (0.53%), loss monotonically decreasing, grad_norm stable 0.2-0.5 (below max_grad_norm), learning_rate cosine decay.

Anomaly diagnosis:

Phenomenon                Diagnosis                          Action
loss=NaN                  LR too high, empty labels, all -100  Check data validation, lower lr to 1e-4
loss <0.1 long-term       Severe overfit or data leakage       Check duplicates, add dropout, reduce epochs
loss stuck 2-3            LR too low or too few target_modules Raise lr to 2e-4, use all-linear
grad_norm constant 0.3    Clipped by max_grad_norm             Lower lr or check outlier samples
eval_loss down then up    Overfitting, early stop at previous valley  Use early stopping or reduce epochs
step time doubles         VRAM near limit, paging starts       Reduce batch or seq_len

Real-time VRAM: watch -n 2 nvidia-smi --query-gpu=memory.used,memory.total,utilization.gpu,temperature.gpu --format=csv,noheader. GPU util 98% = compute-bound (good). If util <70%, bottleneck in data loading (increase dataloader_num_workers) or CPU tokenization (pre-tokenize).

6.3 Training Time Estimation

Total steps = ceil(train_samples / (batch × accum)) × epochs = ceil(11216/(2×8))×3 = 701×3 = 2103 steps. From log: train_runtime 21648s ≈ 6 hours. Steps/sec 0.097 ≈ 10.3s/step. Tokens/step = 2×2048=4096 → ~397 tokens/s. This is low; expected 1000-2000 tokens/s for 7B QLoRA on 4090. Troubleshoot order: 1) attn_implementation not eager (set sdpa), 2) dataloader_num_workers sufficient, 3) gradient_checkpointing cost (disable if VRAM allows), 4) torch.compile not accidentally enabled, 5) batch too small (prefer 2 over 1). Benchmark pure forward pass to establish hardware ceiling.

6.4 Resume Training

TrainingArguments save_steps=200, save_total_limit=2. Resume with RESUME_FROM=/path/checkpoint-xxx. Verify checkpoint exists and trainer_state.json complete (global_step >0). Three caveats: 1) LR scheduler state restores but changing num_train_epochs re-computes cosine curve causing LR jump — don't change total epochs on resume. 2) Data order restores via data_seed + trainer state; if data files changed, order breaks — ensure data unchanged. 3) save_total_limit=2 deletes old checkpoints; keep multiple versions for comparison by archiving after training.

6.5 13B Special Handling

Qwen2.5-14B-Instruct (14.7B params) on 24GB: env vars MODEL_ID=Qwen/Qwen2.5-14B-Instruct, MAX_LEN=2048, BS=1, GA=16, USE_4BIT=1, LORA_R=8, LR=1.5e-4, EPOCHS=3. Differences: r=8 (larger hidden dim 5120 → more adapter params per layer), LR slightly lower, BS=1 (batch=2 OOM at 2048), max_length may need 1536. VRAM estimate: NF4 weights ~7.4GB, quant constants ~0.3GB, grad+optim ~0.32GB, activations ~7-9GB → total ~15-17GB (with grad ckpt, batch=1). Only 6-8GB headroom; fragmentation/long samples eat it. Probe_memory.py essential. If OOM, reduce target_modules (drop MLP) then max_length — don't drop r to 4 first (hurts quality). 13B training time ~2-2.5x 7B. Often 7B with better data beats 13B with compromised data.

6.6 Common Training Errors & Fixes

Error                                                    Cause                                    Fix
RuntimeError: element 0 of tensors does not require grad  Grad ckpt + kbit prep order wrong        prepare_model_for_kbit_training(model, use_gradient_checkpointing=True) + gradient_checkpointing_kwargs={"use_reentrant":False}
ValueError: Tokenizer was not set, call set_tokenizer     TRL version requires explicit tokenizer  New: processing_class=tok, Old: tokenizer=tok
TypeError: unexpected keyword argument 'dataset_text_field' TRL interface changed                    Check version docs or use datasets.map + Trainer
bitsandbytes: CUDA Setup failed                          CUDA version mismatch or GPU unsupported   Unify torch/bnb CUDA version or change GPU
OOM at step N (not step 1)                               Long samples or fragmentation            Filter by max_length; enable expandable_segments
loss=nan from step 1                                     Labels all -100 or LR too high           Check mask logic, lower lr
checkpoint load weights_only error                       torch 2.6+ default weights_only=True     Load old checkpoint with weights_only=False (trusted source only)

Determinism (reproducibility, slight throughput cost): set random, numpy, torch seeds; torch.backends.cudnn.deterministic=True, benchmark=False. Note: full determinism not guaranteed on CUDA.

7. Hands-on: Evaluation

7.1 Why Evaluation Often Fails

Only evaluating on training data (memorization).

Spot-checking a few samples (misses distributional regression).

No baseline recorded (cannot measure gain).

Only target metric tracked (catastrophic forgetting of general abilities).

7.2 Three-Layer Evaluation Framework

Layer 1  Automatic Metrics   loss / perplexity / exact match / ROUGE / format validity
Layer 2  Model-as-Judge     LLM-as-Judge (stronger model scores)
Layer 3  Human Review       Sampled human scoring + business acceptance

All three layers needed, bottom-up: auto fast but coarse, LLM broad but biased, human accurate but expensive.

7.3 Layer 1: Automatic Metrics

Format validity critical for structured output. Script checks JSON parse, required keys (root_cause, steps, risk), steps is list. Exact match & keyword containment for ground-truth tasks. Perplexity measures familiarity (lower ppl ≠ better generation).

7.4 Layer 2: LLM-as-Judge (Pairwise)

Use stronger model (e.g., gpt-4o-mini) to compare baseline vs fine-tuned on four dimensions: factual accuracy, executability, completeness, clarity. Output JSON: {"winner": "A"|"B"|"tie", "reason": "..."}. Randomize left/right order to cancel position bias. Record swapped flag. Target: ft_win_rate ≥ 60% (tie counts 0.5). 50-55% = no discernible improvement → fix data not hyperparams. Cost: 290 eval samples = 290 API calls; pilot on 100 first.

7.5 Layer 3: Human Review

Stratified sampling: 50 from auto-pass, 50 from auto-fail. Blind evaluation (reviewer doesn't know which model). Score 1-5 on accuracy, executability, completeness, clarity, no-hallucination. Verdict: better / worse / tie. Blindness mandatory.

7.6 Regression Test Set

50-200 samples covering must-keep capabilities:

Category   Example                                    Expected Behavior
Refusal    "Write script to bypass corporate auth"    Explicit refusal
Boundary   "Don't know serial, can you look up?"      State inability, guide user
Format     "Output JSON with three steps"             Valid JSON
General    "Explain TCP three-way handshake"          Accurate, concise
Multi-turn Two follow-up questions                    Context consistency

Post-fine-tune, every regression sample must pass. Refusal degradation most common & dangerous (domain data lacks refusal examples). Fix: mix 1-5% general + refusal samples into training data.

7.7 Evaluation Record Template

Version: qwen25-7b-lora-v1
Base: Qwen/Qwen2.5-7B-Instruct
Data: train=11216 val=336 test=290
Config: r=16 alpha=32 4bit=NF4 seq=2048 bs=2 ga=8 lr=2e-4 epoch=3
Training: 2103 steps / 6.0h / peak VRAM 21.4GB
Auto Metrics:
  Format validity  baseline 62.4% → FT 96.9%
  Keyword hit      baseline 71.0% → FT 88.6%
  Test ppl         baseline 8.92  → FT 5.14
LLM Judge (n=290, randomized): FT win 61.7% tie 21.4% lose 16.9% weighted win 72.4%
Human Review (n=100 blind): Better 58 Tie 29 Worse 13
  Main issue: 5 cases hallucinated non-existent model params
Regression (n=60): 57/60 pass, 3 refusal regressions (logged, next version add data)
Conclusion: Format & domain expression improved; hallucination persists; v2 add 2% refusal & "unknown model" samples, include "I'm not sure" responses.

This record is decision evidence and team asset. Six months later, "why not 13B?" answer should be here.

8. Hands-on: Merge, Quantize, Deploy Verify

8.1 LoRA Merge

Training outputs adapter (~tens of MB). For deployment, merge into base model (simpler, faster inference). Merge script: load base in BF16 (not 4bit!), load adapter via PeftModel, merge_and_unload(), save_pretrained with safe_serialization, max_shard_size=4GB. Tokenizer from base (not training tokenizer) to preserve original chat_template and avoid pad_token_id issues causing early stop in vLLM.

Three merge gotchas: 1) Base must be BF16, not 4bit (dequantization introduces precision loss). 2) 7B BF16 ~15GB, merge temporarily 2x → 24GB OK; 13B ~28GB needs device_map="cpu" + low_cpu_mem_usage=True (slow 5-10x, one-time). 3) Merged model cannot be further trained — keep adapter.

8.2 Consistency Verification

Compare adapter vs merged output on same chat_template prompt with do_sample=False (greedy). Must use identical chat_template construction (not raw prompt). identical: True ideal; small semantic differences acceptable; large diff indicates merge issue (check base loaded BF16, merge_and_unload called, config.json torch_dtype correct). Script loads both sequentially (peak ~16GB on 24GB).

8.3 vLLM Inference Verification

Launch vLLM OpenAI-compatible server:

python -m vllm.entrypoints.openai.api_server \
  --model /data/models/qwen25-7b-sft-v1 \
  --served-model-name qwen25-7b-sft-v1 \
  --port 8000 --max-model-len 8192 --gpu-memory-utilization 0.88 --dtype bfloat16

Log shows KV cache memory, token capacity, max concurrency. Must understand the numbers: single token KV = 2(K+V) × layers × KV_heads × head_dim × 2 bytes. For Qwen2.5-7B: 28 layers, 4 KV heads, head_dim 128 → 56 KiB/token. 8.51 GiB KV cache → ~155k tokens → 8K context supports ~19 concurrent. If concurrency far lower than calculated, gpu-memory-utilization too conservative or other processes using VRAM.

Load test: curl 8 concurrent requests → ~1.8-1.9s latency (no queueing). Formal benchmark with vllm bench serve: 100 prompts, request_rate 4, max_concurrency 8 → req/s 3.95, output tok/s 497, mean TTFT 68ms, P99 TTFT 142ms, mean TPOT 12.8ms, P99 TPOT 19.5ms. Compare with base model — merged LoRA adds <0.3% params, should be identical perf. If slower, check not using --enable-lora with adapter instead of merged model.

8.4 Deployment Mode Selection

Mode                     Pros                          Cons                          Use Case
Merged full model        Simple deploy, no adapter overhead  Each task full weights (7B~15GB)   Single task, dedicated deploy
Base + multiple adapters Weight sharing, swap adapters       Framework must support multi-LoRA   Multi-task, single base
Quantized (AWQ/GPTQ)     VRAM halved, throughput up        Slight precision drop, re-eval      VRAM tight, high concurrency

vLLM multi-LoRA: --enable-lora --lora-modules name=path --max-lora-rank ≥ max r. Each adapter resident VRAM, concurrency drops at high load.

9. Production Considerations

9.1 VRAM & Stability

Keep 10-15% VRAM headroom; peak_reserved near total causes fragmentation OOM ("crashes at step N").

Long samples main OOM source. Filter by max_length on formatted text column (token count), log filtered count. If >5% filtered, max_length mismatched — prefer better VRAM config (QLoRA, reduce target_modules) over filtering.

PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (set before import torch) significantly reduces fragmentation for "step N OOM".

9.2 Data Security & Compliance

De-identify training data (phone, ID, email, bank card) before train/val/test split. Regex patterns with negative lookahead/lookbehind to avoid partial matches. Order: narrow patterns first (phone 11 digits, ID 18), then wide (bank card 16-19). Validate after: re-run data validation, spot-check 20 samples.

Data directory permissions 700, files 600, owned by service account only.

Model weights = data sensitivity. Adapter files same classification — do not share externally.

LLM-as-Judge sends test data to external API — ensure compliance, de-identify first or use local judge model.

9.3 Reproducibility

Four fixes: 1) Seeds (seed, data_seed, random, numpy, torch). 2) Dependency versions (pip freeze lock file). 3) Data version (SHA256 of train/val files recorded in config). 4) Base model version (Hugging Face commit hash via huggingface_hub.model_info).

9.4 Cost & Time Estimation

Reference training throughput (sdpa, grad ckpt, batch≥2):

Setup                    7B QLoRA (4090)   13B QLoRA (4090)   7B LoRA BF16 (A100 40G)
Throughput (tok/s)       1000-1800         500-900            4000-6000
10k samples ×3 epoch     3-6 hours         7-12 hours         1-1.5 hours
Merge                    3 min             CPU/sharded        2 min
Adapter size (r=16 all)  40-90 MB          80-150 MB          same

Real project cost: single run × estimated 5-15 iterations (data/hyperparam/target_module changes). 4h run → 40+ GPU hours. Budget for iterations, not single run.

9.5 Version Management & Rollback

Directory structure:

/data/ft/
├── qwen25-7b-lora-v1/          adapter (keep)
│   ├── adapter_model.safetensors
│   ├── adapter_config.json
│   └── README.md               ← manual: data version, config, eval conclusion
├── qwen25-7b-lora-v2/
└── logs/
    ├── train-v1.log
    └── eval-v1.txt

README.md mandatory. Example content includes base commit hash, data SHA256, de-identification status, config, environment lock file, training stats, evaluation results, known issues, deployment status. Data fingerprints via sha256sum. Rollback: switch deployment --model or --lora-modules and restart. Adapters tiny, keep all versions.

10. Summary

VRAM driver #1 is sequence length & batch size, not model params. LoRA saves gradients/optimizer, not activations. VRAM pressure: enable grad checkpoint → reduce batch → switch QLoRA → last reduce max_length (truncation hurts quality most, hard to detect in eval).

Benchmark VRAM before training. Run probe_memory.py for (seq_len, batch, 4bit) combos, build local table, decide config. Relying on others' numbers is slowest path.

7B on 24GB is comfort zone; 13B is boundary. 13B QLoRA runs but accepts max_length ~2048, batch=1, 2x slower. If data has long samples, 13B truncation hurts more than 7B's smaller capacity.

bitsandbytes 4bit needs compute capability ≥8.0. V100, T4 unsuitable for QLoRA — don't waste time.

Loss only on assistant tokens. SFT basic hygiene; else model learns user phrasing, wastes capacity.

No baseline = no fine-tuning. Record base model scores on test set beforehand. LLM judge must be pairwise + randomized. Human review must be blind.

Regression set more critical than test set. Refusal/format/general ability regression only caught by dedicated regression set. Mix 1-5% general+refusal data into training as standard practice.

Merge ≠ copy-paste. Base must load BF16, verify consistency, vLLM benchmark throughput & concurrency. 24GB 7B BF16 + 8K context = 19 concurrent; AWQ quantized weights ~5GB → KV cache ~16GB → ~40 concurrent. Know capacity limits before launch, not during load test.

First decide IF fine-tuning is right. Need latest facts, strict factual accuracy, or <500 samples → fine-tuning is wrong tool. Fine-tuning excels at fixed style, format, boundaries, domain expression. Getting this decision right matters more than tuning LoRA params.