How a T‑Shaped Outfit Evades Both Visible‑Light and Thermal Detectors – Tsinghua’s New Multimodal Adversarial Method

Background

RGB‑T (visible‑light and thermal) joint detection systems improve robustness in low‑light, night, and adverse weather conditions, making them attractive for autonomous driving, security, and robotics. Existing physical adversarial attacks target only a single modality; patterns that fool visible‑light detectors do not affect thermal images and vice‑versa, leaving multimodal detectors untested.

Method

The authors propose a non‑overlapping RGB‑T pattern (NORP). Each garment location is assigned either a printable RGB adversarial pattern (on ordinary fabric) or a thermal‑disrupting pattern (using common aluminum film). The two modalities never overlap spatially, avoiding the brightness loss of stacked patches.

Optimization follows a spatial discrete‑continuous scheme: regions designated for thermal material are treated as discrete choices, while the remaining regions are continuously optimized for RGB colors. During each iteration a random subset of pixels is discretized for the thermal material, and the complementary subset is updated with gradient‑based RGB optimization, satisfying manufacturability constraints while jointly minimizing detection scores for both modalities.

A full 3D human‑and‑clothing model is built to render synchronized RGB and thermal images from 0° to 360° viewpoints and from distances of 2.5 m to 20 m. The generated patterns are then fabricated into shirts and pants using printable fabric and aluminum foil.

To improve transferability across unseen detectors, a fusion‑stage integration strategy incorporates early‑fusion (Prob‑E), middle‑fusion (Prob‑M), late‑fusion (Prob‑L) and independent dual‑modal (YOLO‑11 RGB and thermal) detectors into the loss function, enabling a single garment to attack diverse RGB‑T architectures.

Experiments

Digital evaluation uses the FLIR test set (500 images) with random viewpoints, distances, backgrounds, and lighting. Detectors tested: Prob‑E, Prob‑M, Prob‑L, YOLO‑11 RGB, and YOLO‑11 thermal. NORP achieves an average adversarial success rate (ASR) > 90% across all detectors, substantially higher than solid‑color clothing, random RGB‑T patterns, and prior multimodal attacks.

Physical evaluation captures synchronized RGB (iPhone 13 Pro) and thermal (FLIR T560) streams indoors and outdoors at various times of day. The fabricated clothing yields an average physical ASR of 60%, markedly outperforming baseline garments.

Black‑box transfer tests on unseen detectors (RPN‑E, AR‑CNN, RPN‑L, Deformable DETR) show notable adversarial effects, indicating persistent security risks.

Angle and distance analysis covers full 0‑360° view and 2.5‑20 m range; NORP maintains stable high ASR across all angles and distances, unlike prior 2‑D patch methods limited to narrow view ranges.

Conclusion

The study demonstrates that a manufacturable, full‑viewpoint adversarial garment can reliably compromise fused RGB‑T detectors in both digital and physical settings, highlighting the need for more robust multimodal perception defenses.

Paper: https://arxiv.org/abs/2605.04675

Code: https://github.com/zxp555/RGBT-Clothing

Code example

来源：新智元
本文
约2500字
，建议阅读
5
分钟
清华大学提出一种新型物理对抗方法，利用特殊服装同时干扰可见光和
热成像
检测。这种服装通过非重叠设计和三维建模优化，可有效躲避RGB-T检测器，促进系统安全性研究。