NeurIPS First IML Oral: PAL Boosts Cross-Domain F1 by 10.3%

Problem: Semantic Shortcuts in Image Manipulation Localization

Image Manipulation Localization (IML) aims to predict a binary mask indicating tampered regions. The key is to detect artifacts left by editing, such as edge discontinuities and abnormal noise patterns. However, these artifacts are difficult to annotate directly, so existing methods rely on indirect supervision from tampered region masks. This allows models to learn semantic shortcuts —relying on object categories or scene content rather than manipulation traces—hurting cross-dataset generalization. Label noise further exacerbates the issue.

Existing Approaches and Limitations

End-to-end learning : Uses tampered masks to supervise the model directly. Models easily learn semantic shortcuts and suffer from label noise.

Handcrafted features : Methods introduce priors like SRM filters or Bayar convolutions to focus on low-level signals. These are designed for specific signals and have limited adaptability across manipulation types.

Proposed Method: Pairwise Artifacts Learning (PAL)

The paper reformulates IML by treating artifacts as a latent variable z. Instead of learning p(y|x) directly, it decomposes into two steps: first learn artifact-sensitive representation p(z|x) via pairwise edit relations, then learn localization mapping p(y|z).

EditGroup-45K Dataset

To support pairwise training, the authors construct EditGroup-45K , containing ~45.5K edit groups, 52.1K source images, and 232.5K tampered images (total ~280K). Each group has at least one source anchor and its edited versions. ~60% are human manipulations (copy-move, splicing, removal), ~40% are AIGC manipulations (local inpainting, content modification). This dataset provides explicit edit relations for supervision.

Two-Stage Paradigm

Stage 1: PAL (Pairwise Artifacts Learning) – Two images pass through a shared-weight backbone; feature differences are aggregated into a relation vector to predict whether the pair has a direct edit relation (binary classification with BCE loss). Positive pairs are source and its direct edit; negative pairs include six carefully designed types satisfying marginal matching to prevent trivial solutions. Negative samples also contain tampered images, forcing the model to recognize edit relations rather than just detect fakes.

Stage 2: Standard Localization (SL) – Initialize the localization backbone with PAL pretrained weights, then train with pixel-level masks following each method's standard protocol. No architectural changes; pairwise input is only used in Stage 1. Inference remains single-image.

Experimental Results

Cross-Domain Generalization Improvement

Experiments follow Protocol-CAT, training on CASIA v2, IMD2020, FantasticReality, and Tampered COCO. Evaluation on six test sets: CASIA v1 (in-domain), Coverage, NIST16, Columbia, AutoSplice, CocoGlide (cross-domain). Metric: pixel-level F1.

With ConvNeXt-Small backbone, cross-domain average F1 (Cross Avg) improves from 0.5320 to 0.5868 (+10.3%), and overall average F1 (All-Avg) from 0.5661 to 0.6266 .

PAL boosts performance across 6 backbones (CNNs and Transformers) and 5 mainstream IML methods, with relative improvements of 5.9%–14.0% on All-Avg.

Qualitative results show PAL-initialized predictions align better with ground truth and produce fewer false positives on copy-move, splicing, and inpainting samples.

Ablation Studies

Supervision form : Compared Baseline (ImageNet init), ABS (single-image binary classification on EditGroup-45K), and PAL (pairwise edit relation). ABS cross-domain F1 (0.5292) slightly below Baseline (0.5320), while PAL reaches 0.5868, proving pairwise supervision is key, not just more data.

Data scaling : PAL benefits continuously as data ratio increases from 10% to 100%; ABS saturates at ~50% and slightly drops at 100%, showing supervision style determines data efficiency.

Negative sample design : Removing In-group Edited-Edited negatives hurts cross-domain most, indicating comparing different edits of same content is crucial. Removing Self-Augment improves in-domain but degrades cross-domain, suggesting reliance on source-specific cues.

Artifact Disentanglement Verification

Boundary activation strength of PAL backbone is 5–7 times higher than ImageNet-pretrained backbone across six test sets.

Using a fixed ImageNet encoder to estimate semantic baseline, the paper subtracts this from PAL's pairwise feature differences. Visualizations show responses concentrate on manipulation boundaries for local edits, and distribute widely for AIGC global edits, confirming PAL learns manipulation-relevant artifacts beyond semantic changes.

Robustness to Post-Processing

On CASIA v1 with Gaussian blur and JPEG compression, PAL maintains higher F1 at all severity levels and degrades more gracefully, better reflecting real-world scenarios like social media compression.

Significance

This work is accepted as an Oral presentation at NeurIPS 2026 , marking the first Oral paper in IML history at NeurIPS . The PAL paradigm and EditGroup-45K dataset provide a generalizable, plug-and-play pretraining strategy that explicitly decouples artifacts from semantics, advancing cross-domain generalization in image manipulation localization.