提出自适应跨域融合框架,提升多模态图像融合的通用性与细节保留能力。
Task-Generalized Adaptive Cross-Domain Learning for Multimodal Image Fusion
- 通过自适应小波变换分离高频低频成分,实现模态间频率特征精准对齐。
- 引入空间-频率双域动态融合模块,有效减少细节丢失并提升融合质量。
- 在四类任务中表现优异,适合遥感、医疗等需要高效高质融合的应用场景。
多模态图像融合(MMIF)旨在整合不同成像模态的互补信息,克服单一传感器的局限性,提升图像质量并支持遥感、医学诊断和机器人等下游应用。尽管进展显著,现有方法仍面临模态错位、高频细节丢失和任务专用性等问题。为此,本文提出AdaSFFuse框架,通过自适应跨域协同融合学习实现任务通用的多模态图像融合。该框架创新性地引入自适应近似小波变换(AdaWAT)进行频率解耦,能自适应分离不同场景下多模态图像的高低频成分,实现各模态特征的细粒度提取与对齐;同时设计空间-频率马比纳块(Spatial-Frequency Mamba Blocks),在空间与频率域实现高效跨域融合,并通过可学习映射动态调整,增强对多样化模态的鲁棒性。实验在四个任务上验证:红外可见光融合(IVF)、多焦点融合(MFF)、多曝光融合(MEF)和医学图像融合(MIF),结果表明该方法在保持低计算开销与紧凑网络结构的同时,显著优于现有方法,兼顾性能与效率。代码将开源于https://github.com/Zhen-yu-Liu/AdaSFFuse。
原文摘要 · Abstract (English)
Multimodal Image Fusion (MMIF) aims to integrate complementary information from different imaging modalities to overcome the limitations of individual sensors. It enhances image quality and facilitates downstream applications such as remote sensing, medical diagnostics, and robotics. Despite significant advancements, current MMIF methods still face challenges such as modality misalignment, high-frequency detail destruction, and task-specific limitations. To address these challenges, we propose AdaSFFuse, a novel framework for task-generalized MMIF through adaptive cross-domain co-fusion learning. AdaSFFuse introduces two key innovations: the Adaptive Approximate Wavelet Transform (AdaWAT) for frequency decoupling, and the Spatial-Frequency Mamba Blocks for efficient multimodal fusion. AdaWAT adaptively separates the high- and low-frequency components of multimodal images from different scenes, enabling fine-grained extraction and alignment of distinct frequency characteristics for each modality. The Spatial-Frequency Mamba Blocks facilitate cross-domain fusion in both spatial and frequency domains, enhancing this process. These blocks dynamically adjust through learnable mappings to ensure robust fusion across diverse modalities. By combining these components, AdaSFFuse improves the alignment and integration of multimodal features, reduces frequency loss, and preserves critical details. Extensive experiments on four MMIF tasks -- Infrared-Visible Image Fusion (IVF), Multi-Focus Image Fusion (MFF), Multi-Exposure Image Fusion (MEF), and Medical Image Fusion (MIF) -- demonstrate AdaSFFuse's superior fusion performance, ensuring both low computational cost and a compact network, offering a strong balance between performance and efficiency. The code will be publicly available at https://github.com/Zhen-yu-Liu/AdaSFFuse.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。