How Large Language Models are Transforming Computer Vision: From Image Understanding to Video Generation

Background Introduction

In recent years, large language models (LLMs) have made remarkable progress in text understanding and generation, but their application to natural signals like images and video is still in early exploration. ByteDance’s Doubao visual foundation team, led by researcher Feng Jiashi, presented their work on LLMs for computer vision at the AICon conference.

Fundamental Problems in Computer Vision

Computer vision, a long‑standing AI subfield, can be abstracted into three core abilities: recognition (identifying what is in an image or video), detection (locating objects), and segmentation (pixel‑level understanding of each object). With the rise of AIGC, generative tasks such as text‑to‑image and text‑to‑video have also attracted significant interest.

LLM Unified Model

Traditional vision pipelines use separate specialized models for each task, but the success of unified language models (e.g., GPT) suggests a single model that can handle diverse tasks via prompting. LLMs excel at processing massive textual data, yet they lack direct visual grounding, prompting research into multimodal extensions.

LLM‑Based Image Understanding

Current multimodal LLMs (e.g., GPT‑4V, GPT‑4o) provide coarse image descriptions but struggle with fine‑grained, pixel‑level details. They also suffer from hallucinations because visual features are compressed into textual tokens, losing critical information. To address these issues, researchers aim to give LLMs explicit localization abilities, enabling them to point to image regions or 3D environments.

Our Solution: PixelLM

PixelLM is a pixel‑level LLM that adds a lightweight decoder and a set of segmentation tokens to a standard language model, allowing real‑time multi‑object localization and segmentation while reducing hallucinations. It uses a strong image encoder (CLIP) for multi‑scale feature extraction, a specially designed segmentation vocabulary, and an auto‑regressive decoder that iteratively decodes each object's mask.

Training incorporates segmentation‑specific loss functions to preserve the original language capabilities while enhancing visual grounding.

LLM‑Based Image and Video Generation

Beyond understanding, LLMs are being used for image and video generation. Existing video models face consistency, user‑friendliness, and expressiveness challenges. The proposed StoryDiffusion model tackles these by defining characters with LLMs, generating scripts, and using a consistency‑aware attention mechanism to maintain character appearance across frames, while a semantic‑space interpolation improves motion richness.

Conclusion and Outlook

The presented research demonstrates early but promising steps toward unified multimodal models that combine visual understanding and generation. Future work will focus on tighter integration of perception and action, more efficient data construction (e.g., the MUSE dataset), and improving the model’s ability to learn from the physical world similarly to humans.