Boosting Vision‑Language Model Performance: Prompt‑First vs. Fine‑Tuning Strategies

When a task can be solved with prompts, prefer prompt engineering because fine‑tuning often reduces a model's general capabilities and incurs higher training and deployment costs, including time.

If prompts fail—e.g., unstable or incorrect output formats, incomplete task coverage—switch to Supervised Fine‑Tuning (SFT).

Reinforcement learning is rarely needed in business scenarios; it mainly refines outputs that are already close to the target. For fine‑grained output control, Direct Preference Optimization (DPO) is preferred over PPO because DPO requires only paired data and is easier to construct.

Key data principles : high‑quality business data and sufficient quantity are essential; harder tasks need more data. The data should match the desired output in both content and format. Manual annotation combined with GPT‑based rewriting (including Vision‑LLM versions) can generate high‑quality data.

Data volume recommendations :

If the model already understands the task but output format is unstable, a few dozen to a few hundred business examples suffice.

Typical business tasks need around 1,000 examples, supplemented with a small amount of generic data (ratio 10:1).

Very difficult tasks (e.g., novel image‑to‑text relations) may require 10,000–20,000 business examples with a 5:1 generic‑to‑business ratio.

Training epochs : Small 4B models may need up to 10 epochs; 7B models usually converge after 5 epochs, while 70B models can overfit after 2 epochs.

Typical training workflow:

Collect, clean, rewrite, and augment business data. Convert each query into multiple formats (e.g., multiple‑choice) to simplify evaluation.

Start with a small VLM (2B‑7B). Use a relatively high learning rate (1e‑5) and many epochs (≈10) on pure business data. Test on the training set first to verify data quality and rule out framework issues.

After confirming training‑set performance, use validation curves to finalize hyper‑parameters such as epochs, learning rate, batch size, sequence length, MoE expert count, and parallelism settings (TP, PP, DP).

Introduce a modest amount of generic data (caption and instruction types) with a 10:1 or 5:1 ratio to preserve general capabilities. Caption data can be sourced from ShareGPT‑4o; instruction data from the ALLaVA‑4V dataset.

ShareGPT-4o: https://huggingface.co/datasets/LYM2024/share_gpt4o_zh?row=0

https://huggingface.co/datasets/FreedomIntelligence/ALLaVA-4V

If business images are generated by diffusion models, their generic ability may degrade. Mitigate this by:

Running an existing VLM to caption the diffusion images, generating large‑scale generic caption data.

Using LCM’s SSD or SDXL for img2img with zero guidance (guidance_scale=0, strength≈0.025) to produce diffusion‑style images without prompts.

Training can be staged: begin with a larger, lower‑quality batch, then fine‑tune on a smaller, high‑quality batch. Slightly different question phrasing between stages helps the model distinguish data sources.

Observations:

When data volume is large, the performance gap between a 7B and a 72B model on domain‑specific tasks narrows; business impact matters more than overall generality.

Higher‑quality data allows longer training, while lower‑quality data benefits from shorter training to retain generalization.

Training for at least 5 epochs is advisable; beyond that, monitor validation loss versus generic loss to avoid over‑fitting on niche cases.