How to Efficiently Deploy and Manage 100 LoRA‑Enhanced LLMs with vLLM

A colleague needed a way to efficiently deploy and manage a hundred LoRA‑fine‑tuned models for role‑playing characters, and asked for a practical solution.

Using vLLM, which supports loading multiple LoRA adapters, a script can create separate LoRA requests, generate responses for each character, and optionally run the base model without any LoRA.

from vllm import LLM, SamplingParams
from vllm.lora.request import LoRARequest
from transformers import AutoTokenizer

# Sample prompt
prompts = "你是谁？"

# Generation parameters
sampling_params = SamplingParams(temperature=0.7, top_p=0.8, top_k=50, max_tokens=2048)

# Define two LoRA adapters
lora_request1 = LoRARequest("self_adapter_v1", 1, lora_local_path="output_dir_qwen2.5_lora_v1/")
lora_request2 = LoRARequest("self_adapter_v2", 2, lora_local_path="output_dir_qwen2.5_lora_v2/")

# Load the base model with LoRA support
llm = LLM(model="Qwen2.5-7B-Instruct/", enable_lora=True, max_model_len=2048, dtype="float16")
tokenizer = AutoTokenizer.from_pretrained("Qwen2.5-7B-Instruct/")

# Tokenize prompts using the chat template
temp_prompts = [tokenizer.apply_chat_template([
    {"role": "user", "content": prompt}
], tokenize=False, add_generation_wohaisprompt=True) for prompt in prompts]
prompt_token_ids = tokenizer(temp_prompts).input_ids

# Generate with first LoRA
print("加载角色孙悟空Lora1进行模型推理：")
outputs = llm.generate(sampling_params=sampling_params, prompt_token_ids=prompt_token_ids, lora_request=lora_request1)
for i, (prompt, output) in enumerate(zip(prompts, outputs)):
    print(f"prompt: {prompt}, output: {output.outputs[0].text}")

# Generate with second LoRA
print("加载角色猪八戒Lora2进行模型推理：")
outputs = llm.generate(sampling_params=sampling_params, prompt_token_ids=prompt_token_ids, lora_request=lora_request2)
for i, (prompt, output) in enumerate(zip(prompts, outputs)):
    print(f"prompt: {prompt}, output: {output.outputs[0].text}")

# Generate without any LoRA
print("不加载任何角色的Qwen底座Lora进行模型推理：")
outputs = llm.generate(sampling_params=sampling_params, prompt_token_ids=prompt_token_ids)
for i, (prompt, output) in enumerate(zip(prompts, outputs)):
    print(f"prompt: {prompt}, output: {output.outputs[0].text}")

The script demonstrates loading two different character LoRA weights (Sun Wukong and Zhu Bajie) and producing distinct outputs, while also showing how to run the base model without any LoRA.

In practice, the workload is far more complex. The request distribution follows a long‑tail pattern: the top five models handle about 90 % of traffic, while the remaining 95 models still serve loyal users who expect consistent service. Deploying each model on a dedicated GPU leads to massive GPU memory requirements (14‑16 GB per 7‑8 B model) and, for 100 models, roughly 100 × 24 GB GPUs, costing tens of thousands of yuan per month for a small startup.

Beyond hardware, operational overhead is significant. A seasoned LLM engineer can script automated Docker deployments that pull the correct LoRA weights based on IP hash, but less experienced staff would need to manually install vLLM and load each LoRA on 100 machines, incurring substantial human‑resource costs.

Additional hidden costs include repeated loading of the same base model (each load takes ~2 minutes and adds >1300 GB of cumulative VRAM usage) and idle GPU capacity when low‑traffic models occupy whole cards. For example, a 4090 card priced at ¥10,000 per month could waste nearly ¥40,000 if under‑utilized.

To address these issues, the author evaluated a multi‑LoRA serving platform (Infini‑AI GenStudio). The platform lets a single base model serve many LoRA adapters, charging only for token usage rather than GPU time, making it ideal for long‑tail scenarios.

Typical workflow on the platform:

Create a model service and upload the base model.

Create a deployment task, selecting the uploaded model as the source.

Test the service via the GenStudio UI or a curl request.

curl "https://cloud.infini-ai.com/maas/deployment/mif-damenkp32lcout5v/chat/completions" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $API_KEY" \
  -d '{
        "messages": [
            {"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": "你是谁"}
        ]
    }'

Most platforms require uploading the entire model file, and even if they support LoRA, they often lock you into their proprietary base model, preventing custom LoRA uploads. Infini‑AI’s design avoids this limitation and bills per token, eliminating upfront GPU costs.

By deploying the few high‑traffic models on‑premises and offloading the remaining 90 models to the token‑based service, the author saved close to several hundred thousand yuan per month for the startup.

Overall, combining local vLLM testing with a hosted multi‑LoRA serving solution provides a scalable, cost‑effective strategy for managing large numbers of fine‑tuned LLMs.