How Alibaba’s AIGC Model Revolutionizes Virtual Fashion Try‑On

1. Overview

In today’s digital wave, fashion e‑commerce has shifted from simple shelf‑style displays to a "experience‑first" era. Model‑centric content is the supply foundation, and the "everything‑wearable" AIGC technology promises to transform the fashion supply chain.

For users : moving from "imagination" to "foresight" improves shopping certainty. Users can virtually dress digital avatars with clothing and accessories, instantly seeing fit and styling, achieving a personalized experience.

For merchants : shifting from "high cost" to "high efficiency" yields dual benefits. AI‑driven rapid generation of high‑quality assets reduces traditional photo‑shoot effort, while virtual try‑on lowers return rates by increasing purchase certainty.

We launched the Taobao Star·Fashion Raw Image model, capable of one‑click generation of model‑display assets across multiple categories (clothing, bags, shoes), diverse model demographics (male, female, children), and varied input forms (flat, non‑flat, mask‑free). Compared with prior solutions, it advances three core dimensions:

Better consistency : maintains condition consistency across multiple controls, making generated assets more "realistic".

More beautiful content : enhances human skin tone and pose naturalness, making the model subject appear more "true".

Broader business support : supports diverse categories and presentation forms, enriching assets for a wider range of scenarios.

2. Solution

2.1 Data Infrastructure

High‑quality training data is the engine for model improvement. We built an automated data‑screening pipeline covering single‑image filtering, multi‑image grouping, and more. Quality is defined by:

Condition consistency : precise alignment between input conditions and generated results, avoiding SKU mismatches.

Category diversity : balanced labeling across product types, model poses, backgrounds, and shooting styles.

Content aesthetics : filtering low‑quality images to retain aesthetically pleasing visuals for better supervision signals.

These steps yielded a million‑scale multimodal fashion dataset with high consistency, diversity, and quality, accelerating model capability and supporting varied business scenarios.

We also introduced a differentiated caption paradigm that adapts description granularity based on control signals, using concise summaries for image‑derived elements and detailed, structured descriptions for text‑controlled elements.

To evaluate captions, we co‑developed a dual‑track assessment system with the Future Life Lab, defining quantifiable dimensions such as pose accuracy, accessory integration, and background rendering. Scores correlate positively with downstream model performance, forming a measurable bridge between data quality and model effect.

For scalable application, we built the FashionCaptioner model in two stages: first, leveraging a state‑of‑the‑art multimodal large model with expert‑crafted instructions to create a small, high‑density "golden" image‑text pair dataset; second, fine‑tuning our ReCaption model on this dataset to master the unique captioning style.

2.2 Model Introduction

Generating multi‑condition consistent assets is challenging due to scarce paired data. We designed a framework that trains on single‑reference images to produce multi‑reference outputs, enabling easy category expansion and strong generalization.

Stage 1 – Consistency Learning : massive single‑condition mixed data teaches the model consistent generation.

Stage 2 – Aesthetic Fine‑Tuning : high‑quality aesthetic data refines visual quality and texture.

Stage 3 – Reinforcement Training : face‑consistency rewards and a distortion‑detection model reduce malformed bodies, enhancing stability under increased control conditions.

3. Effect Demonstration

Model‑Based (user‑provided model image)

Input

Output

Model‑Free (no user model image)

Input

Output

Pose‑Transfer

Input (same background, different pose)

Output

Input (different background, different pose)

Output