针对可见光红外对齐,提出按重要性采样提升预训练效果
Not All Patches are Equal: Sampling Matters for Visible-Infrared Pre-Training

- 根据红外结构线索动态调整补丁权重,强化可靠区域学习
- 在多个任务上显著优于现有基线,尤其在红外语义分割上提升明显
- 无需修改模型架构,适配多种主流多模态预训练方法
可见光-红外(VIS-IR)对齐是实现鲁棒多传感器感知的关键预训练任务。现有方法普遍采用均匀补丁对比学习,但成像物理差异导致部分空间对应区域本就不易对齐,等强度对齐反而阻碍表征学习与下游迁移。本文从采样角度重新审视该问题,提出重要性感知采样(IAS):(i) 基于红外结构线索计算补丁权重,并用于重加权对比目标;(ii) 设计轻量级采样器学习软重要性掩码,可选地由手工先验初始化;(iii) 采用补丁课程学习策略,从高可靠性区域逐步扩展至更难区域。IAS为即插即用设计,兼容补丁级/相关性级对齐(如UNIV风格)与图像级对比基线(如ImageBind风格)。在多个VIS-IR基准测试中,其在红外语义分割、红外目标检测、可见光语义分割及跨模态检索任务上均持续优于强基线。代码将发布于https://github.com/KlayMa527/IAS。
原文摘要 · Abstract (English)
Visible-infrared (VIS-IR) alignment is a key pre-training task for robust multi-sensor perception. Most existing methods use uniform patch-wise contrastive learning, but this can be unreliable in VIS-IR data because imaging-physics differences make some spatially paired regions inherently less comparable, and aligning them with equal strength hinders representation learning and downstream transfer. In this paper, we revisit VIS-IR pre-training from a sampling perspective and propose Importance-Aware Sampling (IAS), which adjusts training emphasis based on patch reliability. Specifically, IAS (i) derives patch weights from infrared structural cues and uses them to reweight the contrastive objective; (ii) learns a soft importance mask with a lightweight sampler, optionally warm-started from the hand-crafted prior; and (iii) employs a patch curriculum learning strategy that gradually expands from high-reliability regions to harder patches. It is worth noting that IAS is plug-and-play and works with both patch-/correlation-level alignment (e.g., UNIV-style) and image-level contrastive baselines (e.g., ImageBind-style). Extensive experiments on multiple VIS-IR benchmarks demonstrate consistent improvements over strong baselines, including for IR semantic segmentation, IR object detection and VIS semantic segmentation and cross-modal retrieval task. Code will be released on https://github.com/KlayMa527/IAS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。