arXiv:2603.08018cs.CV2026-03中稿 · CVPR被引 2

用共享字典和系数域推理,解决红外图像缺失时的融合难题。

Missing No More: Dictionary-Guided Cross-Modal Image Fusion under Missing Infrared

  • 基于共享字典学习统一表示,实现跨模态可解释性融合。
  • 在红外缺失下,通过可见光引导生成伪红外系数,提升检测性能。
  • 适合需要鲁棒多模态感知的安防与自动驾驶场景。

红外-可见光(IR-VIS)图像融合对感知与安全至关重要,但现有方法依赖训练与推理时双模态同时可用。当红外模态缺失时,像素空间生成替代物难以控制且缺乏可解释性。本文提出一种基于共享卷积字典的系数域框架,包含三个核心组件:(1) 联合共享字典表征学习(JSRL)构建跨模态共享的可解释原子空间;(2) 可见光引导红外推断(VGII)在系数域将可见光系数转换为伪红外系数,并利用冻结大语言模型作为弱语义先验进行一步闭环优化;(3) 基于表示推断的自适应融合(AFRI)通过窗口注意力与卷积混合,在原子层融合可见光结构与推断出的红外线索,再通过共享字典重建。该编码-转移-融合-重构流程避免了不受控的像素空间生成,同时保持先验一致性。在红外缺失设置下,实验表明该方法在感知质量与下游检测性能上均持续提升。据我们所知,这是首个联合学习共享字典并执行系数域推理-融合的框架。源代码已公开于 https://github.com/harukiv/DCMIF。

原文摘要 · Abstract (English)

Infrared-visible (IR-VIS) image fusion is vital for perception and security, yet most methods rely on the availability of both modalities during training and inference. When the infrared modality is absent, pixel-space generative substitutes become hard to control and inherently lack interpretability. We address missing-IR fusion by proposing a dictionary-guided, coefficient-domain framework built upon a shared convolutional dictionary. The pipeline comprises three key components: (1) Joint Shared-dictionary Representation Learning (JSRL) learns a unified and interpretable atom space shared by both IR and VIS modalities; (2) VIS-Guided IR Inference (VGII) transfers VIS coefficients to pseudo-IR coefficients in the coefficient domain and performs a one-step closed-loop refinement guided by a frozen large language model as a weak semantic prior; and (3) Adaptive Fusion via Representation Inference (AFRI) merges VIS structures and inferred IR cues at the atom level through window attention and convolutional mixing, followed by reconstruction with the shared dictionary. This encode-transfer-fuse-reconstruct pipeline avoids uncontrolled pixel-space generation while ensuring prior preservation within interpretable dictionary-coefficient representation. Experiments under missing-IR settings demonstrate consistent improvements in perceptual quality and downstream detection performance. To our knowledge, this represents the first framework that jointly learns a shared dictionary and performs coefficient-domain inference-fusion to tackle missing-IR fusion. The source code is publicly available at https://github.com/harukiv/DCMIF.

图像融合多模态缺失模态可解释性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。