Simplify Large‑Model Fine‑Tuning with LLaMA‑Factory

Introduction

Large language models (LLMs) demonstrate strong reasoning, question answering, and translation capabilities. Fine‑tuning many models on limited resources is difficult, especially when handling hundreds of models. LLaMA‑Factory aggregates efficient training methods into a single extensible platform that supports over 100 LLMs and reduces model and dataset dependencies.

Prerequisites

Python 3.10 environment with the following packages:

python==3.10

torch==2.6.0

transformers==4.50.0

datasets==3.2.0

accelerate==1.2.1

peft==0.15.1

trl==0.9.6

Installation steps:

git clone --depth1 https://github.com/hiyouga/LLaMA-Factory.git
conda create -n llama_factory python=3.10
conda activate llama_factory
cd LLaMA-Factory
pip install -e '.[torch,metrics]'

Data Preparation

All available datasets are listed in ./data/dataset_info.json. To use a custom dataset, add its description to this file and set dataset:<name> in the configuration. LLaMA‑Factory currently supports Alpaca and ShareGPT formats.

Example Alpaca‑style JSON entry:

[
  {
    "instruction": "User instruction (required)",
    "input": "User input (optional)",
    "output": "Model response (required)",
    "system": "System prompt (optional)",
    "history": [
      ["First turn instruction", "First turn response"],
      ["Second turn instruction", "Second turn response"]
    ]
  }
]

Corresponding entry in dataset_info.json:

"dataset_name": {
  "file_name": "data.json",
  "columns": {
    "prompt": "instruction",
    "query": "input",
    "response": "output",
    "system": "system",
    "history": "history"
  }
}

Place the data file under the data directory.

Model Fine‑Tuning

LLaMA‑Factory supports LoRA, QLoRA, and full‑parameter fine‑tuning. The following steps illustrate LoRA fine‑tuning.

Put the pre‑downloaded base model in the LLaMA‑Factory directory.

Edit examples/train_lora/llama3_lora_sft.yaml to set desired hyper‑parameters.

Run the training command:

llamafactory-cli train examples/train_lora/llama3_lora_sft.yaml

A custom script can invoke the same CLI.

Inference

After fine‑tuning, merge the LoRA adapter with the base model for inference. The inference configuration is at examples/inference/llama3_lora_sft.yaml. Example command:

CUDA_VISIBLE_DEVICES=0 llamafactory-cli chat \
  --model_name_or_path /Meta-Llama-3-8B-Instruct \
  --adapter_name_or_path ./saves/LLaMA3-8B/lora/sft \
  --template llama3 \
  --finetuning_type lora

model_name_or_path

points to the original model; adapter_name_or_path points to the LoRA adapter produced in the fine‑tuning step.

Model Export

To export the merged model, use the YAML at examples/merge_lora/llama3_lora_sft.yaml. Example command:

CUDA_VISIBLE_DEVICES=0 llamafactory-cli export \
  --model_name_or_path /Meta-Llama-3-8B-Instruct \
  --adapter_name_or_path ./saves/LLaMA3-8B/lora/sft \
  --template llama3 \
  --finetuning_type lora \
  --export_dir merged-model-path \
  --export_size 2 \
  --export_device cpu \
  --export_legacy_format False

This completes the pipeline from dataset construction, fine‑tuning, inference, to model export.

Reference

GitHub repository: https://github.com/hiyouga/LLaMA-Factory