How Counterfactual Policy Optimization Boosts Visual Fidelity in Multimodal Reasoning (ICML 2026)

Research Background: Why Multimodal Reasoning Still "Doesn't See the Image"

Large vision‑language models have made great strides on visual QA, multimodal math reasoning, and complex instruction following, yet after RL fine‑tuning they often produce answers that appear to follow a reasoning chain but actually ignore visual evidence.

The authors attribute this to a flaw in current RL objectives: methods such as PPO, GRPO, and DAPO reward only final answer correctness, unable to distinguish whether the answer stems from genuine cross‑modal understanding or from language priors, data bias, and spurious correlations. Consequently, models learn a speculative reasoning style that defaults to language knowledge when visual cues are insufficient or ambiguous.

Three typical cross‑modal causal inconsistency problems are identified:

Insufficient saliency : the model overlooks key visual clues and guesses based on language priors.

Saliency mismatch : attention is drawn to salient but irrelevant visual regions.

Saliency inertia : even when the visual anchor is correctly attended, the model fails to intervene logically for hypothetical questions, persisting with static visual facts.

CFPO’s core insight is that multimodal RL must reward not only the final answer but also the necessity of specific visual evidence for that answer.

Related Work: Why Existing Counterfactual Methods Fall Short

Prior approaches such as VCD, M3ID, and ICD construct visual noise or misleading instructions as counterfactual baselines and calibrate decoding via output distribution differences. These methods incur extra forward passes, raising inference cost, and cannot alter the underlying causal bias in model parameters.

Training‑side methods like DeFacto and CF‑VLM rely on handcrafted or AI‑generated counterfactual data, which is expensive to supervise. PAPO adds perceptual constraints by randomly masking inputs, but pixel‑level interventions are too coarse to pinpoint the fine‑grained visual semantics that high‑level reasoning truly depends on.

CFPO places counterfactual interventions inside the RL loop and targets the language decoder’s multi‑head attention outputs rather than raw pixels, enabling a direct test of whether suppressing critical cross‑modal attention reduces answer confidence.

Method Overview: Counterfactual Policy Optimization

CFPO (Counterfactual Policy Optimization) explicitly adds a cross‑modal causal consistency regularizer to multimodal RL. The model processes an image and a textual query into image tokens and query tokens, respectively, which are fed into a language decoder. The decoder produces an attention matrix and a downstream attention‑output representation Z, treated as a causal mediator linking visual evidence to language generation.

Language Decoder Perspective

In the factual path, the policy distribution depends on the original Z. In the counterfactual path, a do‑intervention replaces the high‑saliency components of Z with a counterfactual representation Z_{cf}. If the answer truly relies on those visual cues, the probability of the original answer should drop under the counterfactual path.

Counterfactual Causal Graph

The simplified causal graph consists of: multimodal input → attention matrix → value vectors → Z → policy. The factual path uses the untouched Z; the counterfactual path applies a do‑intervention that suppresses key visual contributions, yielding Z_{cf}.

The “dynamic causal necessity test” checks whether suppressing a visual cue leads to a significant probability drop, indicating genuine visual dependence.

Cross‑Modal Counterfactual Intervention

Cross‑Modal Saliency Mask

CFPO extracts the cross‑modal attention sub‑matrix (text queries attending to image tokens) from the full attention matrix. Statistical outlier detection (mean + 2 × std) flags tokens whose attention scores exceed the threshold, defining the high‑saliency visual regions.

A saliency mask M_{sal} is built to suppress these regions in the counterfactual path.

Value‑Layer Intervention

Instead of masking raw pixels, CFPO intervenes at the attention‑output layer. The factual path retains the original high‑saliency contributions; the counterfactual path replaces or averages the corresponding value vectors. Three strategies are evaluated:

Image‑token average (best performance)

Text‑token average

Noise addition

Replacing with the image‑token average removes specific semantics while preserving overall feature distribution, whereas noise introduces out‑of‑distribution perturbations and text‑token averaging harms cross‑modal alignment.

Counterfactual Policy Optimization Objective

The objective combines the standard RL reward with a KL‑regularizer to a reference model, an entropy term (optional), and a new counterfactual KL term that penalizes small divergence between factual and counterfactual answer distributions. Two variants are presented:

CFPOG : integrated with GRPO

CFPOD : integrated with DAPO

Both variants require no extra reward model or supervised counterfactual data.

Experimental Setup

Training data: ViRL39K (38,870 verified multimodal QA pairs). Base model: Qwen2.5‑VL‑3B, trained from scratch with RL (no SFT). Hardware: 2 × NVIDIA A800 80G GPUs, rollout batch size 384, 5 responses per prompt.

Evaluation covers two task families:

Real‑world centered reasoning: C‑VQA‑Real, MARS‑Bench, POPE, TextVQA, MMMU‑Pro visual subset.

Math‑centered reasoning: Geo3k, We‑Math, MMk12, MathVerse, LogicVista.

Baselines: GRPO, DAPO, and perception‑enhanced PAPO. Metric: average accuracy over 8 rollouts.

Main Results: CFPO Consistently Improves Faithfulness

Across both task families, CFPO yields stable gains. Relative to standard RL baselines, CFPO improves overall accuracy by 3.17 %–6.25 %; compared to PAPO, gains of 1.32 %–2.13 % remain.

In the GRPO branch, CFPOG raises overall average accuracy from 53.88 % to 55.30 % (real‑world from 60.02 % to 60.98 %; math from 47.73 % to 49.61 %). In the DAPO branch, CFPOD lifts overall average from 55.60 % to 58.49 % (math average from 49.45 % to 53.72 %). The larger boost on math tasks suggests that counterfactual visual constraints are especially beneficial for multi‑step geometric and logical reasoning.

Qualitative Cases: How Counterfactual Constraints Change Reasoning Chains

Four failure modes of baseline models are illustrated:

Geometry case : baseline ignores parallel‑line relationships and resorts to a language template; CFPO correctly incorporates visual structure to infer the angle.

Zebra case : baseline is misled by scene priors and guesses a white rhino; CFPO retains body‑shape evidence and correctly identifies a horse.

Athlete case : baseline spots prominent “HW7” on a jersey but misses the required “below hwt” text; CFPO locates the lower‑position word and answers with the correct initial.

Flower case : baseline sticks to a static count and fails to adjust when a hypothetical extra yellow flower is added; CFPO updates the quantity under the counterfactual condition.

These examples show that CFPO does not merely improve answer formatting; it forces the model to bind answers to visual evidence, changing the internal reasoning pathway.

Ablation Studies: Which Design Choices Matter Most

Key findings:

Among value‑layer interventions, image‑token averaging yields the highest overall accuracy (55.30 %); text‑token averaging provides negligible benefit, and noise addition is less stable.

Saliency threshold “mean + 2 × std” offers the best trade‑off; lower thresholds over‑mask background, higher thresholds miss critical cues.

Regularization: CFPOG performs best with γ = 0.02 and no entropy loss; CFPOD prefers γ = 0.01 with entropy loss, reflecting DAPO’s reduced KL penalty.

Cost and Limitations

Training overhead: CFPOG’s per‑step time rises from 650.20 s to 1,001.58 s and peak memory from 31.03 GB to 40.66 GB; CFPOD’s time grows from 735.15 s to 1,098.71 s and memory from 36.21 GB to 46.20 GB.

The authors attribute the extra memory and slower throughput to the fine‑grained latent‑level interventions that break the continuous flow of Flash‑Attention kernels, forcing fallback to less efficient operators. Custom CUDA kernels could mitigate this.

Experiments are limited to Qwen2.5‑VL‑3B; future work should test scalability to larger models, additional modalities, and real‑world robotic or interactive scenarios.

Conclusion

CFPO introduces a counterfactual policy optimization framework that adds a cross‑modal causal consistency constraint to multimodal RL. By requiring answer probabilities to drop when high‑saliency visual evidence is suppressed, the method ensures that correct answers are genuinely grounded in visual cues. Empirical results demonstrate seamless integration with GRPO and DAPO, yielding consistent accuracy improvements, faster convergence, and more reliable reasoning chains across both real‑world and math‑centric benchmarks.