无需真实标签,自动对齐红外与可见光图像实现高质量融合
Bi-directional Self-Registration for Misaligned Infrared-Visible Image Fusion
- 用双向自监督机制,通过生成器和逆生成器同步校准全局与局部差异
- 在多个错位图像数据集上,融合结果在SSIM、PSNR等指标上优于现有方法
- 适合需要高精度跨模态图像对齐的军事、安防场景应用
获取精确对齐的多模态图像对是实现高质量多模态图像融合的基础。针对当前多模态配准与融合方法缺乏真实标签的问题,本文提出一种新型自监督双向自配准框架(B-SR)。B-SR利用代理数据生成器(PDG)与逆代理数据生成器(IPDG),实现自监督的全局-局部配准。通过配准模块对空间错位的可见光-红外图像对进行全局差异对齐;同时,使用PDG(如裁剪、翻转、拼接等)处理相同图像对,生成局部差异,并由IPDG将局部差异转化为伪全局差异,与真实全局差异进行一致性约束。此外,为消除模态差异对配准模块的影响,设计邻域动态对齐损失,实现跨模态图像边缘对齐。在多个错位多模态图像数据集上的实验表明,该方法在多模态图像对齐与融合性能上显著优于现有方法。代码将公开。
原文摘要 · Abstract (English)
Acquiring accurately aligned multi-modal image pairs is fundamental for achieving high-quality multi-modal image fusion. To address the lack of ground truth in current multi-modal image registration and fusion methods, we propose a novel self-supervised \textbf{B}i-directional \textbf{S}elf-\textbf{R}egistration framework (\textbf{B-SR}). Specifically, B-SR utilizes a proxy data generator (PDG) and an inverse proxy data generator (IPDG) to achieve self-supervised global-local registration. Visible-infrared image pairs with spatially misaligned differences are aligned to obtain global differences through the registration module. The same image pairs are processed by PDG, such as cropping, flipping, stitching, etc., and then aligned to obtain local differences. IPDG converts the obtained local differences into pseudo-global differences, which are used to perform global-local difference consistency with the global differences. Furthermore, aiming at eliminating the effect of modal gaps on the registration module, we design a neighborhood dynamic alignment loss to achieve cross-modal image edge alignment. Extensive experiments on misaligned multi-modal images demonstrate the effectiveness of the proposed method in multi-modal image alignment and fusion against the competing methods. Our code will be publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。