How to Fine‑Tune Large Language Models: From PEFT to Knowledge Injection

1. Introduction

Pre‑trained large language models (LLMs) such as Gemini 3 excel on general NLP tasks, but their generic weights limit performance on highly specialized domains. Fine‑tuning—training the model on a smaller, task‑specific dataset—updates the internal weights and yields substantial gains over prompt engineering alone.

2. Fundamentals of LLM Fine‑Tuning

Supervised fine‑tuning (SFT) uses labeled prompt‑response pairs. The model’s parameters are updated via back‑propagation with standard optimizers (e.g., AdamW). Because the model already contains broad linguistic knowledge, fine‑tuning reduces both compute cost and data requirements compared with training from scratch. High‑quality labeled data is therefore critical.

3. Parameter‑Efficient Fine‑Tuning (PEFT)

3.1 Low‑Rank Adaptation (LoRA) and Quantized LoRA (QLoRA)

LoRA freezes the original weights and injects small trainable low‑rank matrices (adapters) that approximate weight updates. The rank hyper‑parameter controls the trade‑off between efficiency and expressiveness. QLoRA extends LoRA by quantizing the base model to 4‑bit precision, further lowering memory usage. Both are available in Hugging Face PEFT and bitsandbytes .

3.2 Adapter Fine‑Tuning

Adapter layers are inserted into each transformer block while keeping the backbone frozen. A typical adapter consists of a down‑projection, a non‑linear activation, an up‑projection, and a residual connection. This reduces trainable parameters, speeds up training, and enables adding new tasks without degrading previously learned capabilities. Near‑identity initialization stabilises training.

3.3 Prefix (Prompt) Tuning

Prefix tuning learns continuous task‑specific vectors (soft prompts) that are prepended to the model input. Only the prefix vectors are updated; the base model remains frozen. This method requires fewer parameters than full fine‑tuning and works well for generation tasks such as summarisation.

4. Data‑Preparation Pipeline

4.1 Data Collection & Management

Define the target task, then identify relevant data sources. Prioritise relevance, diversity, and ethical compliance (privacy, bias). When real data are scarce, synthetic data can be generated.

4.2 Cleaning & Pre‑Processing

Remove unwanted characters, handle missing values, denoise, and normalise text (lower‑casing, punctuation removal, stop‑word filtering, HTML tag stripping). Tokenise the cleaned text using the tokenizer that matches the chosen model.

4.3 Annotation Strategies

Choose manual, semi‑automatic, or fully automatic labeling pipelines. Domain experts should verify annotations, especially for specialised fields.

4.4 Dataset Splits

Split the data into training, validation, and test sets (random or stratified sampling) to obtain unbiased performance estimates and to guard against over‑fitting.

4.5 Instruction‑Based Formatting

For instruction fine‑tuning, format each example as a JSON object with keys instruction, optional input, and output. The “Alpaca” format is a common convention.

5. Advanced Knowledge‑Injection Techniques

5.1 Knowledge Injection

Beyond textual fine‑tuning, external knowledge (e.g., knowledge graphs) can be embedded into model weights via joint learning or plug‑and‑play paradigms. Mapping‑based fine‑tuning aligns knowledge embeddings with model inputs, improving factual accuracy and reasoning.

5.2 Model Weight Fusion

Weight merging, ensemble, or reinforcement‑learning‑based fusion (e.g., FuseLLM) combines multiple pre‑trained or fine‑tuned checkpoints to create a more versatile model.

6. Key Challenges and Mitigations

6.1 Catastrophic Forgetting

When fine‑tuning on new data, models may lose previously learned knowledge. Mitigation strategies include experience replay, regularisation (elastic weight consolidation, synaptic intelligence), knowledge distillation, progressive learning, and PEFT methods such as LoRA.

6.2 Over‑fitting

Symptoms arise from small or imbalanced datasets and excessive epochs. Countermeasures: data diversification, early stopping, dropout, weight decay, data augmentation, cross‑validation, and careful hyper‑parameter tuning.

6.3 Compute Constraints

PEFT methods (LoRA, adapters) dramatically reduce GPU memory and compute. Additional compression techniques include quantisation and pruning. Cloud resources or multi‑GPU setups can be leveraged for very large models.

7. Evaluation Framework

7.1 Quantitative Metrics

Generation tasks: perplexity, BLEU, ROUGE. Classification tasks: accuracy, precision, recall, F1, exact match, semantic similarity. Choose metrics aligned with the fine‑tuned task.

7.2 Qualitative Assessment

Human evaluation assesses relevance, coherence, creativity, and appropriateness. Techniques include manual review, RLHF feedback loops, and using larger LLMs as judges.

8. Best Practices

Clearly define the task and success criteria.

Select an appropriate pre‑trained architecture (e.g., Llama 2, Mistral, Phi‑3).

Prioritise high‑quality, domain‑relevant data.

Set sensible hyper‑parameters: learning rate (e.g., 1e‑4 to 5e‑5), batch size (adjusted for GPU memory), number of epochs (monitor validation loss).

Use early stopping based on validation performance.

Continuously evaluate on a held‑out test set and iterate.

Monitor deployed models for drift and update data periodically.

9. Efficient Fine‑Tuning with Hugging Face Transformers

Typical workflow:

from transformers import AutoModelForCausalLM, AutoTokenizer, Trainer, TrainingArguments
from datasets import load_dataset

model_name = "meta-llama/Meta-Llama-3-8B"
model = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto")
tokenizer = AutoTokenizer.from_pretrained(model_name)

ds = load_dataset("json", data_files={"train": "train.json", "validation": "val.json"})

def tokenize_fn(example):
    return tokenizer(example["prompt"] + example.get("input", "") + example["output"], truncation=True, max_length=1024)

ds = ds.map(tokenize_fn, batched=True)

training_args = TrainingArguments(
    output_dir="./fine_tuned",
    per_device_train_batch_size=4,
    per_device_eval_batch_size=4,
    learning_rate=2e-5,
    num_train_epochs=3,
    evaluation_strategy="epoch",
    save_strategy="epoch",
    load_best_model_at_end=True,
)

def compute_metrics(eval_pred):
    # placeholder for metric computation (e.g., ROUGE)
    return {}

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=ds["train"],
    eval_dataset=ds["validation"],
    tokenizer=tokenizer,
    compute_metrics=compute_metrics,
)

trainer.train()

PEFT can be integrated by loading a LoRA configuration with peft.get_peft_model before creating the Trainer.

10. Conclusion

Fine‑tuning LLMs—especially with parameter‑efficient methods (LoRA, adapters, prefix tuning) and knowledge‑injection strategies—enables high performance on domain‑specific tasks while keeping resource usage manageable. Success depends on careful data preparation, appropriate model selection, disciplined hyper‑parameter tuning, and thorough quantitative and qualitative evaluation.