How LLaMA Invades Computer Vision: Introducing VisionLLaMA from Meituan and Zhejiang University

Overview

VisionLLaMA investigates whether a LLaMA‑style Transformer can process 2‑D images and proposes a unified modeling framework for most vision tasks.

https://arxiv.org/abs/2403.00522<br/>https://github.com/Meituan-AutoML/VisionLLaMA

Model Design

The naive VisionLLaMA follows the ViT processing pipeline but replaces the positional‑encoding self‑attention with Rotary Position Encoding (RoPE) and adopts the SwiGLU activation function while keeping ViT's LayerNorm (instead of RMSNorm). Because 1‑D RoPE does not scale well to different image resolutions, the authors extend it to a 2‑D form called AS2DRoPE (automatic scaling 2‑D RoPE). AS2DRoPE builds a diagonal matrix for the 2‑D encoding and interpolates it to arbitrary resolutions.

Pyramid VisionLLaMA

Inspired by Swin‑Transformer, a pyramid version is built on the Twins backbone. The design removes the conditional positional encoding (AS2DRoPE already provides positional information), drops the class token, and inserts a global average pooling layer before the classification head.

Training and Inference Beyond Fixed Sequence Length

Visual tasks often require variable input resolutions. Convolutional networks handle this with sliding windows; many visual transformers use local windows or interpolation (e.g., DeiT uses bicubic interpolation, CPVT adopts convolution‑based positional encoding). VisionLLaMA adapts RoPE from 1‑D to 2‑D, constructing a diagonal matrix for the 2‑D encoding and scaling it to any resolution, enabling training and inference on images of arbitrary size.

Experiments

Image Generation

Extensive pre‑training on image‑generation benchmarks shows VisionLLaMA surpasses existing SOTA ViT solutions.

Image Classification

Results on standard classification datasets demonstrate higher accuracy and faster convergence compared with strong ViT baselines.

Semantic Segmentation

VisionLLaMA achieves superior mean‑IoU scores on segmentation benchmarks, confirming the effectiveness of the 2‑D RoPE and SwiGLU components.

COCO Object Detection

On the COCO detection task, VisionLLaMA attains higher AP while converging in fewer epochs than comparable visual transformers.

Across all tasks, VisionLLaMA converges faster and delivers better performance without additional tricks, establishing a strong new baseline for both visual generation and understanding.

Code example

▲点击上方卡片，关注AIWalker公众号