From Parameter Tuning to Control: CFG‑Ctrl Boosts Stability and Precision in Text‑to‑Image Generation

Problem Motivation

Users of text‑to‑image tools often encounter contradictory outcomes: clear prompts describing spatial relationships or readable text lead to images with misplaced objects, distorted text, or unnatural colors. Repeated parameter tweaking can improve semantic alignment but degrades visual quality, requiring many generations to obtain a usable result. As generative AI moves into design, e‑commerce, and content creation, stability and structural correctness become critical.

Proposed Method: CFG‑Ctrl

The Tsinghua team reinterprets classifier‑free guidance (CFG) not as a simple parameter but as a control problem. They model the diffusion process as a dynamic system, treat semantic deviation as an error signal, and apply control theory—specifically sliding‑mode control—to redesign the guidance mechanism. This transforms the generation from trial‑and‑error to a stable convergence toward semantically constrained results.

Key Contributions

Recasting CFG as a controllable dynamic system.

Introducing sliding‑mode control (SMC‑CFG) to achieve stable, fast, and precise convergence.

Demonstrating cross‑model applicability on three diffusion models: SD3.5 (medium), Flux (large), and Qwen‑Image (ultra‑large).

Experimental Evaluation

Evaluation spans three metric layers:

Distribution quality measured by FID.

Semantic alignment measured by CLIP similarity.

Human‑preference metrics such as ImageReward, HPS, and PickScore.

Across all metrics, SMC‑CFG consistently outperforms standard CFG. For example, FID improves modestly, CLIP alignment remains stable, and human‑preference scores reach the highest levels among compared methods. The advantage grows with model size: larger models (Flux, Qwen‑Image) show clearer gaps.

Compared with prior improvements (CFG‑Zero*, Rectified‑CFG++), SMC‑CFG delivers holistic gains rather than isolated metric boosts, indicating a mechanism‑level advancement.

High Guidance Scale Stability

Standard CFG suffers from quality collapse as guidance scale increases—semantic alignment improves but visual quality deteriorates. SMC‑CFG maintains image quality while strengthening semantic information even at high scales, breaking the classic trade‑off.

Ablation Studies

The authors analyze two critical parameters: λ (direction of convergence) and k (correction strength). Too small or too large λ destabilizes the system; insufficient k slows convergence and weakens semantics, while excessive k causes oscillations and unnatural images. The best performance arises from moderate λ combined with balanced k, reflecting classic control‑system trade‑offs between stability and responsiveness.

Broader Implications

By turning CFG into a controllable system, the work shifts the field from empirical tuning to systematic control design, enabling analysis of stability, convergence, and robustness. This explains why high guidance scales previously caused color shifts, structural distortion, or text corruption: linear error amplification in a nonlinear diffusion process leads to oscillation and divergence, which sliding‑mode control mitigates.

Practically, the improved guidance reduces trial‑and‑error for designers, creators, and e‑commerce operators, lowering cost and increasing reliability of generated images for real‑world applications.

Research Team and Publication

The work is presented in the paper “CFG‑Ctrl: Control‑Based Classifier‑Free Diffusion Guidance” (arXiv:2603.03281) by Wang Hanyang (first author) and Prof. Duan Yueqi’s group at Tsinghua University. The team’s prior publications appear in CVPR, ICCV, NeurIPS, ECCV, TIP, and TPAMI.