How to Efficiently Deploy and Fine‑Tune DistilQwen2 on Alibaba Cloud PAI

Introduction

Qwen2 (Tongyi Qianwen 2) is an open‑source large language model series developed by Alibaba Cloud, excelling in code generation, mathematics, reasoning, instruction following, and multilingual understanding. DistilQwen2 is a smaller, knowledge‑distilled version of Qwen2 offered through the PAI platform, delivering strong performance on resource‑constrained devices such as mobile and edge computers.

PAI‑QuickStart Overview

PAI‑QuickStart is a component of Alibaba Cloud's AI platform that integrates high‑quality pre‑trained models from both domestic and international open‑source communities. It enables zero‑code or SDK‑based end‑to‑end workflows for training, deployment, and inference, greatly simplifying AI development for developers and enterprises.

Runtime Environment Requirements

The example supports multiple regions (Beijing, Shanghai, Shenzhen, Hangzhou, Ulanqab, Singapore) on the PAI‑QuickStart product.

Training stage: DistilQwen2‑1.5B/7B requires at least an A10 GPU (24 GB VRAM) or higher.

Deployment stage: DistilQwen2‑1.5B needs a minimum P4 single‑card; recommended cards include GU30, A10, V100, T4. DistilQwen2‑7B requires at least a P100, T4, or V100 (gn6v); GU30 or A10 are recommended.

Using the Model via PAI‑QuickStart

In the PAI console, navigate to the “Quick Start” entry and select a DistilQwen2 model (e.g., DistilQwen2‑1.5B‑Instruct). The model card is shown below:

Model Deployment and Invocation

DistilQwen2‑1.5B‑Instruct comes with pre‑filled deployment configuration. Provide a service name and resource specifications to deploy the model to the PAI‑EAS inference platform.

The deployed inference service can be accessed via the ChatLLM WebUI for real‑time interaction:

It also supports OpenAI‑compatible API calls.

Model Fine‑Tuning

PAI provides two fine‑tuning algorithms for DistilQwen2: SFT (Supervised Fine‑Tuning) and DPO (Direct Preference Optimization). Both accept JSON‑formatted data.

[
    {
        "instruction": "你是一个心血管科医生，请根据患者的问题给出建议：...",
        "output": "高血压的患者可以吃..."
    },
    {
        "instruction": "你是一个呼吸科医生，请根据患者的问题给出建议：风寒感冒咳白痰怎么治疗？",
        "output": "风寒感冒，咳有白痰的患者..."
    }
]

DPO expects three fields: prompt, chosen, and rejected:

[
  {
    "prompt": "Could you please hurt me?",
    "chosen": "Sorry, I can't do that.",
    "rejected": "I cannot hurt you..."
  },
  {
    "prompt": "That guy stole one of my tools...",
    "chosen": "You shouldn't have done that...",
    "rejected": "That's understandable..."
  }
]

Upload prepared data to an OSS bucket, then launch the fine‑tuning job on an A10 GPU (24 GB VRAM).

Adjust hyper‑parameters as needed (see the screenshot below).

Click “Train” to start; monitor status and logs from the task page.

After training, you can deploy the fine‑tuned model with one click, using the same invocation methods as the base model.

Model Evaluation

PAI offers built‑in evaluation algorithms for both base and fine‑tuned models. Users can evaluate on custom datasets or public benchmarks (MMLU, TriviaQA, HellaSwag, GSM8K, C‑Eval, TruthfulQA).

Custom evaluation requires a JSONL file where each line contains {"question": ..., "answer": ...} entries. Example snippet:

[{"question": "请问室温超导技术的难点在哪里...", "answer": "室温超导技术的难点在于..."}]

Public benchmark evaluation is performed via the Model Gallery or the training task detail page (screenshots below).

Evaluation results are displayed on the task page (example screenshots).

Model Compression

Before deployment, trained models can be quantized/compressed to reduce resource consumption. Create a compression task in the training interface, configure the compression method, settings, and compute resources, then launch the task.

After compression finishes, click “Deploy” for one‑click deployment of the compressed model.

Large‑Model Distillation with PAI‑QuickStart

Beyond using DistilQwen2, PAI‑QuickStart supports teacher‑student distillation pipelines. Deploy a large‑model teacher and specialized small models for instruction enhancement and optimization, enabling various distillation algorithms. See the “Large Language Model Data Enhancement and Model Distillation Solution” for detailed best practices.

Conclusion

Alibaba Cloud’s Qwen and DistilQwen2 series demonstrate the immense potential of large language models across diverse scenarios. Knowledge distillation preserves strong performance while dramatically reducing resource demands, making these models ideal for mobile and edge deployments. The PAI platform provides end‑to‑end support for training, fine‑tuning, evaluation, compression, and deployment, offering developers and enterprises a clear, practical pathway to leverage advanced LLM capabilities.

Related Resources

DistilQwen2 introduction: https://developer.aliyun.com/article/1633882

LLM data enhancement & model distillation solution: https://help.aliyun.com/zh/pai/use-cases/llm-data-enhancement-and-model-distillation-solution

PAI QuickStart overview: https://help.aliyun.com/zh/pai/user-guide/quick-start-overview

PAI Python SDK on GitHub: https://github.com/aliyun/pai-python-sdk