How the New AMD Framework Beats AI‑Generated Deepfakes with the MDSM Dataset

Background and the “Coherence Trap”

AI‑generated multimodal fake news can combine manipulated images with large‑language‑model‑crafted narratives that are semantically consistent, making detection based on low‑level mismatches ineffective. This phenomenon is termed the “coherence trap”.

MDSM Dataset

The Multimodal Deep‑Fake Synthetic Media (MDSM) dataset provides over 440,000 high‑quality image‑text pairs sourced from major media outlets such as The Guardian and The New York Times. It covers five core forgery categories—face swapping, attribute manipulation, textual tampering, and two others—and guarantees perfect semantic alignment between visual and textual modalities. The benchmark supports three tasks: binary forgery detection, forgery‑type classification, and manipulation‑region localization, making it the largest and most challenging dataset for multimodal deep‑fake detection.

Artificial Manipulation Detector (AMD) Framework

AMD tackles the coherence trap with three innovations:

Pre‑emptive artifact encoding : a “fake‑radar” module injects artifact‑sensitive tokens into the multimodal encoder, allowing the model to capture subtle manipulation traces while preserving extensive world knowledge.

Dual‑branch reasoning : separate visual and textual streams process the image and text, followed by a cross‑modal interaction layer that highlights inconsistencies, enabling detection of both the existence of a forgery and the precise manipulated region.

Lightweight architecture : the model contains only 0.27 B parameters and runs at 13.38 image‑text pairs per second on an RTX 4090, achieving performance comparable to billion‑parameter baselines.

Experimental Evaluation

On the MDSM cross‑domain test set AMD achieves:

Accuracy 88.18 %, mean Average Precision 60.25, mean IoU 61.02, surpassing ViLT, HAMMER++, and FKA‑Owl.

Zero‑shot evaluation of general‑purpose multimodal models (GPT‑4o, Gemini 2.0, Qwen‑3‑VL) yields near‑zero detection performance, confirming the necessity of specialized detectors.

Cross‑dataset testing on DGM4 retains 74.47 % accuracy, demonstrating strong generalisation.

Inference speed of 13.38 pairs/s on RTX 4090 with only 0.27 B parameters.

Resources

All code, pretrained models, and the MDSM dataset are publicly released at https://github.com/YcZhangSing/AMD.