Local Deployment, Fine‑tuning, and Inference of the Open‑source Alpaca‑LoRA Model on GPU Servers

The Alpaca model is an open‑source LLM developed by Stanford, fine‑tuned from Meta's LLaMA‑7B and containing 7 billion parameters. LoRA (Low‑Rank Adaptation) reduces the computational cost of fine‑tuning by freezing the pretrained weights and inserting trainable low‑rank matrices into each Transformer block.

LoRA works by adding a bypass that first reduces the dimensionality (matrix A) and then expands it back (matrix B). During training only A and B are updated; at inference the product BA is added to the original weight matrix W, so no extra runtime overhead is introduced.

Figure 1: LoRA architecture.

The target deployment uses a GPU server with four NVIDIA Tesla P40 cards. After installing the appropriate NVIDIA driver and CUDA toolkit (downloaded from https://www.nvidia.com/Download/index.aspx), the driver file (e.g., NVIDIA‑Linux‑x86_64‑515.105.01.run) is executed as root, ensuring no running NVIDIA processes.

To isolate dependencies, an Anaconda environment is created:

conda create -n alpaca python=3.9
conda activate alpaca

Additional packages are installed manually (setuptools, pip) and the required Python libraries are installed via: pip install -r requirements.txt After confirming the GPU and CUDA status with nvitop, the Alpaca‑LoRA repository is cloned and the Chinese instruction‑response dataset is downloaded.

Fine‑tuning is performed with the following command (single‑GPU):

python finetune.py \
    --base_model 'decapoda-research/llama-7b-hf' \
    --data_path 'trans_chinese_alpaca_data.json' \
    --output_dir './lora-alpaca-zh'

For multi‑GPU training:

WORLD_SIZE=2 CUDA_VISIBLE_DEVICES=0,1 torchrun \
    --nproc_per_node=2 \
    --master_port=1234 \
    finetune.py \
    --base_model 'decapoda-research/llama-7b-hf' \
    --data_path 'trans_chinese_alpaca_data.json' \
    --output_dir './lora-alpaca-zh'

Training took about 31.7 hours on two GPUs, after which the model converged. Inference is launched by modifying generate.py and running:

python generate.py --base_model "decapoda-research/llama-7b-hf" \
    --lora_weights './lora-alpaca-zh' \
    --load_8bit

The service prints the IP and port, which can be accessed via a browser. The fine‑tuned model can answer simple Chinese questions, though more complex queries sometimes produce English responses or garbled text due to limited dataset size and encoding issues.

Key observations include: limited Chinese corpus leads to modest comprehension; inference latency is high (30 s–1 min per request) on three GPUs; occasional Chinese character garbling; and the need to monitor GPU utilization during inference.

Future work should expand the Chinese dataset, explore more efficient model variants, and consider larger GPU clusters for real‑time chat performance.