Multi‑LoRA Deployment for Large Language Models: Concepts, Fine‑tuning, and Cost‑Effective Strategies

Deploying multiple fine‑tuned large language models (LLMs) can be costly because each scenario often requires a separate model instance, leading to GPU waste. This article proposes a multi‑LoRA approach that merges deployments, allowing several scenario‑specific adapters to share a single base model.

LoRA (Low‑Rank Adaptation) reduces the number of trainable parameters to less than 2% of the original model. For example, fine‑tuning GPT‑3 (175 B parameters) with LoRA cuts the trainable parameters by about 10,000× and reduces GPU memory requirements threefold.

Fine‑tuning with LoRA is straightforward using the LLaMA‑Factory framework. The required command is: llamafactory-cli train examples/train_lora/llama3_lora_sft.yaml . After training, a small LoRA adapter file is produced.

The adapter can be merged with the base model via: llamafactory-cli export examples/merge_lora/llama3_lora_sft.yaml , yielding a new fine‑tuned model ready for inference.

For serving the merged model, the open‑source VLLM engine is recommended. Install it with: pip install vllm and start the service using: vllm serve {model_path} . VLLM’s PageAttention design provides high throughput.

The traditional pipeline—training a LoRA per scenario, merging, and deploying each model—requires multiple GPUs when many low‑traffic scenarios exist. Multi‑LoRA solves this by loading the base model once and keeping several LoRA adapters in GPU memory, selecting the appropriate adapter at inference time.

Multi‑LoRA is suitable for diverse business scenarios with low request volume per scenario. It dramatically reduces hardware costs while maintaining performance.

VLLM currently supports multi‑LoRA. Enable it with: vllm serve {model_path} --enable-lora --lora-modules {lora1_path} {lora2_path} ... .

Performance tests on Llama‑3‑8B with 20 GPU cards show that multi‑LoRA adds negligible latency and throughput impact. The main constraints are: all adapters must share the same base model, and the LoRA rank R should not exceed 64 when using VLLM.

In summary, multi‑LoRA provides a cost‑effective, scalable solution for deploying many scenario‑specific LLMs by sharing a single base model and leveraging modern inference engines.