Fine‑Tuning LLaMA‑7B with Alpaca‑LoRA to Build a Chinese ChatGPT

Recent breakthroughs such as GPT‑4 have sparked renewed interest in large language models (LLMs). Open‑source projects like Stanford's Alpaca (built on LLaMA) and its LoRA‑based variant Alpaca‑LoRA enable fine‑tuning a 7B model at a cost of less than $600, even on a single consumer‑grade GPU.

Why train your own ChatGPT? The author lists several motivations: personal curiosity, enabling the model to speak Chinese, generating code comments and tests, answering product‑related questions, and more.

Step 1 – Prepare the dataset – Two common fine‑tuning goals are (1) creating input/output prompt pairs for specific tasks (e.g., Alpaca style) and (2) language modeling with text completion. For a Chinese ChatGPT the author uses a Chinese‑translated Alpaca dataset provided by Luotuo.

Step 2 – Train and apply LoRA – The dataset is used to fine‑tune the base LLaMA‑7B model with LoRA adapters. The following commands illustrate the workflow.

git clone [email protected]:tloen/alpaca-lora.git

wget https://github.com/LC1332/Chinese-alpaca-lora/blob/main/data/trans_chinese_alpaca_data.json

conda create -n alpaca python=3.9
conda activate alpaca
cd alpaca-lora
pip install -r requirements.txt

Single‑GPU fine‑tuning:

python finetune.py \
    --base_model 'decapoda-research/llama-7b-hf' \
    --data_path '/path/to/trans_chinese_alpaca_data.json' \
    --output_dir './lora-alpaca-zh'

Multi‑GPU (2 × RTX 3090 Ti) fine‑tuning (requires adjusting --micro_batch_size to avoid OOM):

WORLD_SIZE=2 CUDA_VISIBLE_DEVICES=0,1 torchrun \
    --nproc_per_node=2 \
    --master_port=1234 \
    finetune.py \
    --base_model 'decapoda-research/llama-7b-hf' \
    --data_path '/path/to/trans_chinese_alpaca_data.json' \
    --output_dir './lora-alpaca-zh'
    --micro_batch_size 2
    --num_epochs 2

During training the author monitors GPU memory with nvitop and observes convergence after roughly two epochs.

Step 3 – Merge the model (optional) – Merging LoRA weights into the base model can improve inference speed and simplify later quantization.

Step 4 – Quantization (optional) – Quantization reduces memory footprint and speeds up inference; the author points to the Sparsebit quantization README for ready‑to‑use scripts.

Inference – For single‑GPU inference:

python generate.py --base_model "decapoda-research/llama-7b-hf" \
    --lora_weights './lora-alpaca-zh' \
    --load_8bit

For multi‑GPU inference the author modifies generate.py to expose a server_name="0.0.0.0" argument, allowing remote access.

Results – The fine‑tuned model can converse in Chinese and perform simple coding tasks, though occasional quality gaps remain because the original LLaMA training data is predominantly English and the translated Alpaca data is imperfect. The author emphasizes the importance of high‑quality data for LLM performance.

Conclusion – Fine‑tuning LLMs is enjoyable and educational for engineers interested in distributed systems. The author plans to explore purpose‑specific LLMs (e.g., cooking assistants) and encourages readers to follow future developments.