Why Full Fine‑Tuning Beats LoRA: When and How to Update Every Model Parameter

Full Fine‑tuning Definition

Full fine‑tuning updates every parameter of a pretrained model so that it fully adapts to a target task, analogous to a general‑practice doctor retraining at a top‑tier hospital to become a specialist.

Typical Workflow

Load a pretrained model → train with all parameters unfrozen → obtain a task‑specific model.

Comparison with LoRA/Adapter and Prompt Tuning

Parameters Updated: Full fine‑tuning updates all parameters; LoRA/Adapter updates only a small set of adapter weights; Prompt tuning modifies only prompt embeddings.

Training Cost: Full fine‑tuning is high; LoRA is low; Prompt tuning is negligible.

GPU Memory: Full fine‑tuning requires the entire model in memory; LoRA needs a small footprint; Prompt tuning needs almost none.

Final Performance: Full fine‑tuning yields the best results; LoRA reaches ~90‑95 % of that performance; Prompt tuning is generally weaker.

Training Time: Full fine‑tuning is long; LoRA is short; Prompt tuning is extremely short.

Suitable Scenarios: Full fine‑tuning for maximum accuracy with abundant data (>100 k examples) and powerful hardware (A100/H100); LoRA for resource‑constrained environments; Prompt tuning for rapid prototyping.

When to Use Full Fine‑tuning

Need the highest possible accuracy.

Dataset larger than 100 000 labeled examples.

Access to high‑end GPUs (A100/H100) or a large GPU cluster.

Not recommended when data is scarce (<10 000 examples), compute is limited, or many tasks must be fine‑tuned simultaneously because each task would require a separate full‑parameter model.

Hands‑On Example (Hugging Face Transformers)

from transformers import AutoTokenizer, AutoModelForSequenceClassification, Trainer, TrainingArguments

# 1. Load model (all parameters trainable)
model = AutoModelForSequenceClassification.from_pretrained("gpt2", num_labels=2)
tokenizer = AutoTokenizer.from_pretrained("gpt2")
tokenizer.pad_token = tokenizer.eos_token

# 2. Prepare datasets (train_dataset, val_dataset)

# 3. Configure training arguments
training_args = TrainingArguments(
    output_dir="./gpt2-finetuned",
    num_train_epochs=3,
    per_device_train_batch_size=8,
    learning_rate=5e-5,
    fp16=True,               # mixed precision saves ~50 % VRAM
    evaluation_strategy="epoch",
    save_strategy="epoch",
    load_best_model_at_end=True,
)

# 4. Create Trainer (no parameter freezing)
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    eval_dataset=val_dataset,
)

# 5. Start training (updates all ~124 M weights)
trainer.train()

Key Configuration Tricks

fp16=True

: enables mixed‑precision training, cutting VRAM roughly in half. gradient_accumulation_steps=4: simulates larger batch sizes on limited GPU memory. max_grad_norm=1.0: gradient clipping to prevent loss explosions. warmup_ratio=0.1: warm‑up phase stabilises early training. model.gradient_checkpointing_enable(): trades compute for memory when VRAM is insufficient.

Memory‑Saving with DeepSpeed ZeRO‑3

from pytorch_lightning.strategies import DeepSpeedStrategy

trainer = Trainer(
    strategy=DeepSpeedStrategy(
        stage=3,
        offload_optimizer=True,   # optimizer states to CPU
        offload_parameters=True  # model parameters to CPU
    ),
)

Using ZeRO‑3 can shrink a 7 B model’s memory from >40 GB to ~24 GB, verified in practice.

Common Pitfalls and Remedies

Catastrophic Forgetting: model loses general abilities after fine‑tuning. Solution: mix generic data, lower learning rate, or switch to LoRA.

Overfitting: training loss ↓ while validation loss ↑. Solution: early stopping, regularisation, data augmentation.

Training Instability (loss spikes): Solution: lower learning rate, extend warm‑up, enable gradient clipping.

Out‑of‑Memory (OOM): Solution: gradient checkpointing, smaller batch size, DeepSpeed ZeRO‑3.

Cold Knowledge Nuggets

LoRA reaches ~90‑95 % of full‑fine‑tuning performance at ~1/10 the compute cost.

Data preparation typically consumes ~70 % of total project time; clean data is critical for reliable results.

Learning rate is the “soul” of full fine‑tuning; a typical setting is 1/10 of the pre‑training rate (e.g., 1e‑5 – 5e‑5). Too high a rate leads to divergence.

BF16 precision on A100/H100 is more stable than FP16, with minimal accuracy loss.

References

Hugging Face Fine‑tuning Guide – https://huggingface.co/docs/transformers/training

DeepSpeed Official Tutorial – https://www.deepspeed.ai/tutorials/

LLaMA Fine‑tuning Blog – https://huggingface.co/blog/llama2