arXiv:2411.10036cs.CVcs.AI2024-11被引 2

改进归一化与卷积核设计,显著提升多模态图像融合效果

Rethinking Normalization Strategies and Convolutional Kernels for Multimodal Image Fusion

  • 采用实例与组归一化混合策略,保留稀疏特征
  • 在多个数据集上实现最优性能,MSRS分割mIoU提升8.1%
  • 适合需要高精度细节保留的医学或遥感图像融合任务

多模态图像融合(MMIF)通过整合不同模态信息生成综合图像,辅助下游任务。现有研究聚焦互补信息融合与训练策略,忽视了归一化和卷积核等底层结构组件的关键作用。本文重新评估端到端的UNet架构,发现广泛使用的批归一化会因平滑关键稀疏特征而限制性能。为此,提出实例与组归一化混合策略,维持样本独立性并强化内在特征相关性,从而生成更丰富的特征图,使大核卷积充分释放感受野优势,增强细节保留。此外,提出的多路径自适应融合模块动态校准不同尺度与感受野的特征,确保有效信息传递。该方法在MSRS、M$^3$FD、TNO和哈佛数据集上达到当前最优(SOTA)客观性能,生成视觉更清晰的显著目标与病灶区域。尤其在MSRS数据集上,分割mIoU较红外图像提升8.1%。这一性能源于归一化与卷积核的协同设计,有效保留关键稀疏特征。代码已开源:https://github.com/HeDan-11/LKC-FUNet。

原文摘要 · Abstract (English)

Multimodal image fusion (MMIF) integrates information from different modalities to obtain a comprehensive image, aiding downstream tasks. However, existing research focuses on complementary information fusion and training strategies, overlooking the critical role of underlying architectural components like normalization and convolution kernels. We reevaluate the UNet architecture for end-to-end MMIF, identifying that widely used batch normalization limits performance by smoothing crucial sparse features. To address this, we propose a hybrid of instance and group normalization to maintain sample independence and reinforce intrinsic feature correlations. Crucially, this strategy facilitates richer feature maps, enabling large kernel convolution to fully leverage its receptive field, enhancing detail preservation. Furthermore, the proposed multi-path adaptive fusion module dynamically calibrates features from varying scales and receptive fields, ensuring effective information transfer. Our method achieves SOTA objective performance on MSRS, M$^3$FD, TNO, and Harvard datasets, producing visually clearer salient objects and lesion areas. Notably, it improves MSRS segmentation mIoU by 8.1\% over the infrared image. This performance stems from a synergistic design of normalization and convolution kernels, which preserves critical sparse features. The code is available at https://github.com/HeDan-11/LKC-FUNet.

图像融合归一化卷积核多模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。