提出统一框架,让红外可见光图像融合与语义分割相互促进。
MAFS: Masked Autoencoder for Infrared-Visible Image Fusion and Semantic Segmentation
- 双分支并行结构,融合与分割任务协同优化。
- 多阶段Transformer decoder高效聚合多尺度融合特征。
- 动态权重分配机制保障多任务训练稳定,适合跨模态感知场景。
红外-可见光图像融合旨在生成视觉质量优良的融合图像,并提升高层任务性能。现有语义驱动方法虽考虑了语义信息注入下游应用,但尚未从宏观任务层面探究像素级图像融合与跨模态特征融合感知任务之间的相互促进关系。为此,本文提出一个统一网络MAFS,采用并行结构,包含融合子网络和分割子网络。一方面,设计异构特征融合策略以增强图像融合的语义感知能力;另一方面,通过级联融合子网络与分割骨干网络,将分割知识传递至特征级融合以提升分割性能。在框架内,设计新型多阶段Transformer解码器,高效聚合细粒度多尺度融合特征。此外,引入基于最大最小公平分配原则的动态因子,生成两任务自适应权重,确保多任务训练平稳进行。大量实验表明,本方法在性能上优于现有先进方法。代码已开源:https://github.com/Abraham-Einstein/MAFS/。
原文摘要 · Abstract (English)
Infrared-visible image fusion methods aim at generating fused images with good visual quality and also facilitate the performance of high-level tasks. Indeed, existing semantic-driven methods have considered semantic information injection for downstream applications. However, none of them investigates the potential for reciprocal promotion between pixel-wise image fusion and cross-modal feature fusion perception tasks from a macroscopic task-level perspective. To address this limitation, we propose a unified network for image fusion and semantic segmentation. MAFS is a parallel structure, containing a fusion sub-network and a segmentation sub-network. On the one hand, We devise a heterogeneous feature fusion strategy to enhance semantic-aware capabilities for image fusion. On the other hand, by cascading the fusion sub-network and a segmentation backbone, segmentation-related knowledge is transferred to promote feature-level fusion-based segmentation. Within the framework, we design a novel multi-stage Transformer decoder to aggregate fine-grained multi-scale fused features efficiently. Additionally, a dynamic factor based on the max-min fairness allocation principle is introduced to generate adaptive weights of two tasks and guarantee smooth training in a multi-task manner. Extensive experiments demonstrate that our approach achieves competitive results compared with state-of-the-art methods. The code is available at https://github.com/Abraham-Einstein/MAFS/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。