RA-CFGCache: Risk-Aligned Caching for CFG-Guided Diffusion Models

Diffusion models generate high-quality visual content but suffer from high computational cost due to iterative denoising, especially under Classifier-Free Guidance (CFG) which requires computing both conditional and unconditional branches. Training-free caching methods (TeaCache, MagCache, DiCache) reuse historical features or predictions to skip computation, but they make caching decisions based on branch-level error estimates, ignoring two critical mismatches:

Branch–Guided Mismatch : Branch-level cache errors do not directly reflect the CFG-combined guided error because errors can cancel or amplify depending on their alignment (cosine similarity) and the CFG scale s > 1.

Local–Final Mismatch : Local guided errors at different timesteps have unequal impact on the final generated output due to varying error propagation gains through subsequent denoising steps.

Method: RA-CFGCache

RA-CFGCache (Risk-Aligned Caching under CFG) addresses both mismatches with two core modules and a lightweight online controller, without modifying the original model, sampling schedule, or CFG rule.

CFG-Aware Guided-Risk Composition

Instead of judging each branch independently, the method maps branch-level proxy risk estimates (from TeaCache, MagCache, or DiCache) into the CFG-combined guided-risk space. Let ρ_u and ρ_c be the proxy risks for unconditional and conditional branches when reusing cache from anchor step t_a to current step t. The guided risk is estimated as:

R_guided = ρ_u² + s²ρ_c² + 2s(1-s)ρ_uρ_c cosθ

where s is the CFG scale and cosθ is an offline-calibrated cross-branch alignment statistic approximating the cosine similarity between branch error directions. When s > 1, the cross-term coefficient 2s(1-s) is negative, so aligned errors partially cancel in CFG combination. This module is compatible with any base cache estimator that provides per-branch proxy risks.

Propagation-Aware Rescaling

To account for timestep-dependent downstream impact, the authors conduct single-step perturbation experiments: cache reuse at only one timestep, full computation elsewhere, and measure final generation deviation. They define propagation gain G(t) = Δ_final / Δ_local and fit a power-law prior: G(k) = λ (k/T)^α + β where k is the increasing sampling-step index, T total steps, and λ, α, β are offline-calibrated parameters. The propagation-aware risk becomes R_prop = G(k) · R_guided. Thus, two timesteps with similar local guided risk may receive different caching decisions if their propagation gains differ.

Online Cache Control

A cumulative risk threshold controller operates on joint branch reuse (both branches refreshed together). At each cache interval, an anchor step is set and cumulative risk R = 0. For current step t, incremental risk ΔR is added. If R + ΔR > τ (threshold), both branches are recomputed (Refresh), anchor updated, and R reset; otherwise caches are reused (Reuse) and R updated. The threshold τ trades off quality vs. speed.

Experiments

Setup

Models : FLUX.1-dev (text-to-image, 200 DrawBench prompts, 1024×1024), Wan2.1-T2V-1.3B (text-to-video, 100 VBench prompts, 832×480, 81 frames), CogVideoX-2B (text-to-video, 100 VBench prompts, 720×480, 49 frames).

Baselines : TeaCache, DiCache, MagCache, TaylorSeer, HiCache, FasterCache.

Metrics : Speedup, Latency, LPIPS (lower better), SSIM (higher better), PSNR (higher better). All compared against full-reasoning reference with same prompts/seeds.

Hardware : NVIDIA H200, 50 denoising steps.

Quantitative Results

RA-CFGCache offers Slow (fidelity-oriented) and Fast (speed-oriented) configurations. On FLUX.1-dev, Slow achieves 3.19× speedup with LPIPS 0.1137 vs. MagCache 3.18× / 0.1636 (30.5% lower perceptual error). Fast reaches 3.95× speedup. Similar trends hold on Wan2.1 and CogVideoX, confirming generality across image and video generation.

Efficiency–Fidelity Trade-off

Latency vs. LPIPS curves show RA-CFGCache consistently dominates MagCache and DiCache across three models: at comparable latency, it yields lower perceptual error, indicating robust improvement across acceleration levels, not just at a single threshold.

Qualitative Comparison

Visualizations on FLUX.1-dev show TeaCache, MagCache, and DiCache introduce local structural changes or detail deviations at similar speedups, while RA-CFGCache better preserves textures, object structures, and semantic details.

Ablation Study (FLUX.1-dev)

CFG Risk Composition | Propagation Rescaling | Speedup ↑ | LPIPS ↓
------------------- | ------------------- | --------- | -------
✗                   | ✗                   | 3.42×     | 0.1732
✓                   | ✗                   | 3.41×     | 0.1391
✗                   | ✓                   | 3.41×     | 0.1423
✓                   | ✓                   | 3.43×     | 0.1262

Both modules are effective and complementary: CFG risk composition alone reduces LPIPS from 0.1732 to 0.1391; propagation rescaling alone reduces to 0.1423; together they reach 0.1262 with unchanged speedup.

Additional Analyses

CFG Scale Robustness : Tested at scales 1.5, 3.5, 5.5, 7.5 — RA-CFGCache maintains strong trade-offs across all, showing adaptation to varying guidance strengths.

Guided Risk Estimation Validity : On 37,400 cache-reuse events from 50 prompts, full guided-risk estimation (with cross-term) achieves Pearson correlation 0.7575 with true guided error vs. 0.6082 without cross-term; high-risk detection AUPRC improves from 0.5671 to 0.6799.

Propagation Weighting Validity : Spearman correlation between risk statistic and final LPIPS rises from 0.6571 to 0.9429 after adding propagation weights, confirming better reflection of downstream fidelity impact.

Compatibility & Calibration Robustness : RA-CFGCache improves fidelity when combined with TeaCache, DiCache, or MagCache. Performance remains stable across different calibration sample sizes and subsets, indicating robustness to offline calibration choices.

Conclusion

RA-CFGCache provides a unified, lightweight caching control framework that aligns cache decisions with CFG error combination mechanics and timestep-dependent error propagation. It achieves state-of-the-art efficiency-fidelity trade-offs on image and video diffusion models, is compatible with existing cache estimators, and requires no model retraining or schedule changes. Paper: https://arxiv.org/abs/2609.36433 (NeurIPS 2026).