SCALAR++: Scale‑Aware Visual Autoregressive Learning for Efficient Controllable Image Generation

Journal Introduction

The International Journal of Computer Vision (IJCV) is a top‑tier, Springer‑published journal recommended by the Chinese Computer Federation (CCF) as an A‑class venue, focusing on computer vision, image processing, pattern recognition, and generative AI.

Reading

Controllable image generation aims to synthesize visual content that precisely follows user‑provided guidance signals. Recent visual autoregressive (VAR) models, with their "next‑scale prediction" paradigm, have shown superior generation quality and inference efficiency compared with traditional diffusion models, yet equipping VAR with fine‑grained spatial control remains a critical open problem.

Project Introduction

To address this gap, the paper proposes SCALAR++, an efficient controllable generation framework based on scale‑aware conditional decoding. By sharing scale‑wise conditioning layers and employing lightweight LoRA modulation, SCALAR++ dramatically reduces parameter overhead while matching or surpassing diffusion baselines in both generation quality and control accuracy.

Core Contributions

SCALAR (Scale‑wise Conditional Decoding) : Projects frozen visual encoder (DINOv2) semantic features onto VAR backbone layers specific to each generation scale, providing continuous, precise conditioning throughout the multi‑scale generation process.

SCALAR++ (Layer‑Shared Scale‑wise Conditioning) : Introduces a Layer‑Shared Scale‑wise Conditioning strategy for high‑resolution text‑to‑image tasks, sharing a single projection module across scales and adding LoRA‑based layer‑specific modulation. This yields 43% fewer parameters and 39% lower memory consumption during inference.

SCALAR‑Uni (Unified Control Alignment) : Maps features from diverse control modalities into a unified latent space, enabling a single model to handle multiple condition types.

Research Background

Industry Demand : Controllable image generation has broad applications in creative design, content production, and assisted design. The rise of large language models (LLMs) makes autoregressive generation a natural fit for unified multimodal systems.

Core Challenges : Existing VAR controllable methods suffer from two major design flaws: (a) complex, disruptive injection mechanisms (e.g., parallel branches or joint modeling) that increase computational cost and degrade pretrained generation ability; (b) lightweight convolutional or VQ‑VAE control encoders that fail to capture rich spatial semantics.

Limitations of Prior Work : Approaches such as ControlVAR and CAR improve controllability but still lag behind raster‑scan AR methods like ControlAR in generation quality and control consistency, indicating untapped potential for VAR in controllable synthesis.

Highlights

Scale‑Aware Conditional Decoding : Aligns conditioning injection with the intrinsic "next‑scale prediction" of VAR, injecting control signals into every transformer layer at each scale for fine‑grained guidance.

Efficient Layer‑Shared Design : Combines shared projection modules with LoRA modulation, preserving control precision while drastically reducing resource consumption.

Unified Multi‑Condition Control : Through a unified control alignment loss, maps multiple control modalities to a shared image feature space, enabling SCALAR‑Uni and SCALAR++‑Uni to handle diverse conditions.

Zero‑Shot Generalization : Without additional training, SCALAR++ supports controllable inpainting, outpainting, and mixed‑control synthesis, demonstrating strong downstream generalization.

Cross‑Backbone Generalization : The design is backbone‑agnostic; it successfully adapts to VAR architectures with STAR and Infinity tokenization strategies, confirming broad applicability.

Experimental Results

Category‑to‑Image (C2I) Generation : On ImageNet 256×256, SCALAR achieves the best performance across five conditions. Using VAR‑d12, it reaches FID 3.12 (Canny) and 3.83 (Depth), substantially better than ControlAR’s 7.69 and 4.19.

Text‑to‑Image (T2I) Generation : On the MultiGen‑20M 512×512 benchmark, SCALAR++ outperforms strong baselines such as ControlAR and ControlNet++ under all conditions.

Visualization

Conclusion

SCALAR++ solves the structural mismatch between visual autoregressive models and controllable generation by introducing scale‑aware conditional decoding and an efficient layer‑shared parameterization, providing a clear, extensible technical path for controllable synthesis under the autoregressive paradigm.

Future Work

The authors plan to explore adaptive multi‑condition weight learning and extend the framework to ultra‑high resolutions (1024+), further pushing the boundaries of controllable visual synthesis.