Vitron: How a Pixel‑Level Multimodal LLM Bridges Vision and Language

Vitron is a pixel‑level visual multimodal large language model that integrates Vicuna‑7B (v1.5) as its language core with dedicated front‑end encoders (image, video, and region‑sketch) and a back‑end that orchestrates state‑of‑the‑art vision modules via a text‑centered calling strategy.

Supported Tasks

Low‑level visual semantics : panoptic, instance, semantic, and referring segmentation.

High‑level visual semantics : pixel‑level understanding for both images and videos.

Vision segmentation & grounding : panoptic, instance, semantic, referring, phrase grounding, video grounding, video object tracking.

Pixel‑level vision understanding : image/video captioning, referring captioning, image/video QA, language‑image/video retrieval, video temporal grounding.

Vision synthesis & generation : text‑to‑image, text‑to‑video, image‑to‑video generation.

Vision editing & inpainting : object addition/removal, object replacement/movement, style or color modification.

Additional capabilities : pixel‑aware user interaction, modular extensibility, image‑video interconversion, multi‑turn conversation.

Architecture

The system consists of three logical blocks:

Front‑end Modules

Image encoder : converts raw images into dense feature vectors.

Video encoder : extracts spatio‑temporal features from video clips and outputs a unified feature representation.

Region‑sketch encoder : encodes user‑drawn sketches or masks into the same feature space, enabling pixel‑level interaction.

Core LLM

Vicuna‑7B (v1.5) processes natural‑language instructions, generates prompts for vision modules, and synthesizes final textual or visual outputs.

Back‑end Modules

State‑of‑the‑art vision models for segmentation, grounding, captioning, retrieval, generation, and editing (e.g., SoTA segmentation networks, diffusion generators, video‑to‑text models).

All modules are invoked through a text‑centered calling strategy : the LLM formulates a textual request, the dispatcher routes it to the appropriate vision module, and the result is fed back to the LLM for further reasoning.

Experimental Evaluation

Vitron was benchmarked on 22 public datasets covering four major visual task categories—segmentation, understanding, generation, and editing. Across all datasets it achieved superior performance compared with existing visual LLMs, demonstrating both high accuracy (e.g., state‑of‑the‑art scores on panoptic segmentation and video QA) and flexible multi‑turn interaction.

Installation & Usage

Requirements: Python ≥3.8, PyTorch 2.1.0, CUDA ≥11.8.

git clone https://github.com/SkyworkAI/Vitron
cd Vitron
conda create -n vitron python=3.10 -y
conda activate vitron
pip install --upgrade pip
pip install -e .
pip install -e ".[train]"
pip install flash-attn --no-build-isolation
pip install decord opencv-python
pip install git+https://github.com/facebookresearch/pytorchvideo.git@28fe037d212663c6a24f373b94cc5d478c8c1a1d

Typical troubleshooting steps include reinstalling ffmpeg, rebuilding detectron2, matching the required gradio version, and fixing deepspeed library links.

Demo

After downloading model checkpoints, launch the Gradio demo with: python app.py The demo supports continuous multi‑turn interaction, allowing users to issue commands such as segmentation, captioning, video generation, or pixel‑level editing.

Resources

Paper (PDF): http://haofei.vip/downloads/papers/Skywork_Vitron_2024.pdf

Code repository: https://github.com/SkyworkAI/Vitron