Fine‑Tune Qwen2‑VL with LLaMA Factory on Alibaba Cloud to Build a Tourism QA Bot

Introduction

The rapid development of AI drives demand for fine‑tuning large models. This tutorial shows how to combine Alibaba Cloud AI platform PAI with the open‑source low‑code framework LLaMA Factory to fine‑tune the multimodal Qwen2‑VL model and quickly build a tourism‑domain knowledge‑question answering robot.

1. Environment and Resource Preparation

Enable the interactive modeling service PAI‑DSW and create an instance. New users can claim a free‑trial resource package; existing users may create pay‑as‑you‑go instances (≈ 6‑30 CNY per hour).

Configure the PAI console:

Region: choose Beijing (or Hangzhou, Shanghai, Shenzhen as needed).

Do not select additional products such as MaxCompute or DataWorks.

Authorize the service role.

Open the PAI NotebookGallery, locate the tutorial “LLaMA Factory Multimodal Fine‑Tuning: Fine‑Tune Qwen2‑VL to Build a Tourism Large Model”, and click “Open in DSW”.

Create a DSW instance (e.g., DSW_LlamaFactory ) with a GPU of at least 24 GB VRAM (A10 or higher). Use the official image modelscope:1.14.0-pytorch2.1.2-gpu-py310-cu121-ubuntu22.04 . Keep other parameters at default, confirm, and wait about three minutes for the instance to run.

Open the Notebook once the instance is ready.

Install LLaMA Factory by executing the provided commands in the Notebook cells.

Download the provided multi‑turn dialogue dataset (train.json) or use the built‑in datasets located in the data directory.

2. Model Fine‑Tuning

Launch the Web UI from the Notebook and open the generated URL.

In the Web UI (switch language to Chinese if desired), select the Qwen2VL‑2B‑Chat model and set the fine‑tuning method to full . Use the downloaded train.json as the dataset and preview it to confirm.

Configure training parameters: learning rate 1e‑4, epochs 10, compute type pure_bf16 , gradient accumulation 2, and save interval 1000 to reduce disk usage.

Set the output directory to train_qwen2vl and start the fine‑tuning job.

The training takes about 14 minutes; progress and loss curves appear in the UI, and a “training completed” message indicates success.

3. Model Dialogue

In the Web UI’s Chat tab, set the checkpoint path to train_qwen2vl and load the fine‑tuned model.

Upload one of the provided test images, set the system prompt to “You are a tour guide, answer visitors’ questions vividly,” and submit queries.

The model generates responses that reflect the knowledge in the dataset, e.g., describing the Shanxi Museum in a lively guide tone.

You can unload the fine‑tuned model and reload the original model to compare outputs; the original model fails to recognize the museum correctly.

4. Resource Cleanup and Follow‑Up

After completing the experiment, stop or delete the DSW instance in the console to avoid ongoing charges.

During the trial period you may continue using the DSW instance for further model training, inference, or exploring additional AI image‑editing scenarios.