用一个模型统一适配近红外到远红外,实现跨波段红外感知
SpectraDINO: Modality-Conditioned Adaptation of RGB Vision Foundation Models Across Infrared Bands

- 通过轻量输入头和模态嵌入,让同一ViT模型适配不同红外波段
- 在7个基准上超越专用模型,最高提升2.6点mIoU,SWIR检测增2.7 mAP
- 适合需要多波段红外视觉的机器人与自动驾驶场景
视觉基础模型(VFMs)在大规模可见光数据上预训练,具备强大的通用表征能力,但红外感知(对低光及恶劣天气下的机器人与驾驶至关重要)仍依赖于为特定传感器构建的主干网络。现有迁移方法仅能处理单一波段或单个可见光-红外配对,缺乏可覆盖全红外光谱的统一主干。我们提出SpectraDINO,一种模态条件化的RGB视觉基础模型跨波段适应方法,支持NIR、SWIR和LWIR,仅使用一个共享的ViT。所有模态特异性行为通过轻量级输入茎、学习的模态嵌入和每模态瓶颈适配器实现,而变压器权重保持共享。我们设计分阶段协议,通过余弦蒸馏、对称对比对齐、像素级对应关系以及基于队列的邻域保持损失,将冻结的DINOv2教师模型蒸馏至谱学生模型,随后选择性微调深层块以适应新模态而不遗忘可见光先验。在涵盖三个红外波段的七个检测与分割基准上,该单一条件化主干模型表现优于或匹配专用方法,在FMB上提升mIoU 2.6点,在SemanticRT上提升1.4点,并在SWIR检测上比同等微调的基础模型高出2.7 mAP。
原文摘要 · Abstract (English)
Vision foundation models (VFMs) pretrained on large-scale RGB data provide strong general-purpose representations, yet infrared perception, which is essential for robotics and driving in low light and adverse weather, still relies on backbones built per sensor. Existing transfer methods bridge the visible-infrared gap for only a single band or a single RGB-IR pair, leaving no unified backbone that spans the infrared spectrum. We present SpectraDINO, a modality-conditioned adaptation of an RGB vision foundation model that supports NIR, SWIR, and LWIR with a single shared ViT. All modality-specific behavior is conditioned on the active sensor via lightweight input stems, learned modality embeddings, and per-modality bottleneck adapters, while the transformer weights remain shared. We propose a staged protocol to distill the frozen DINOv2 teacher into a spectral student via cosine distillation, symmetric contrastive alignment, patch-level correspondence, and a queue-based neighborhood-preservation loss, and then selectively fine-tune the deeper blocks to adapt without forgetting RGB priors. Across seven detection and segmentation benchmarks spanning all three infrared bands, this single conditioned backbone matches or surpasses modality-specific methods, improving mIoU over the best published results by 2.6 points on FMB and 1.4 on SemanticRT, and exceeding identically fine-tuned foundation-model baselines by 2.7 mAP on SWIR detection.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。