让融合图像直接服务于下游任务,自动学习最优融合方式。
Task-driven Image Fusion with Learnable Fusion Loss

- 用可学习的损失函数替代固定规则,由任务目标反向指导融合过程。
- 在四个数据集上融合效果提升,语义分割与目标检测任务性能显著增强。
- 无需预设损失,适配任意任务和网络结构,灵活性强。
多模态图像融合通过整合多个传感器源的信息,相比单源图像能获得更优的视觉质量和感知特征,通常有助于提升下游任务表现。然而,现有融合方法仍采用预定义的融合目标,可能与下游任务不匹配,限制了自适应引导能力并降低模型灵活性。为此,我们提出任务驱动的图像融合(TDFusion),其融合损失由神经网络生成的可学习参数构成,该模块通过元学习方式以下游任务损失进行监督。优化目标是使融合图像在融合模块优化后,其任务损失最小化。融合模块与损失模块的迭代更新确保融合网络逐步趋向最小化任务损失,从而将融合过程导向具体任务目标。TDFusion的训练完全依赖下游任务损失,可灵活适配任意任务与网络架构。实验在四个不同数据集上进行融合测试,并评估了语义分割与目标检测任务表现,验证了其有效性。
原文摘要 · Abstract (English)
Multi-modal image fusion aggregates information from multiple sensor sources, achieving superior visual quality and perceptual features compared to single-source images, often improving downstream tasks. However, current fusion methods for downstream tasks still use predefined fusion objectives that potentially mismatch the downstream tasks, limiting adaptive guidance and reducing model flexibility. To address this, we propose Task-driven Image Fusion (TDFusion), a fusion framework incorporating a learnable fusion loss guided by task loss. Specifically, our fusion loss includes learnable parameters modeled by a neural network called the loss generation module. This module is supervised by the downstream task loss in a meta-learning manner. The learning objective is to minimize the task loss of fused images after optimizing the fusion module with the fusion loss. Iterative updates between the fusion module and the loss module ensure that the fusion network evolves toward minimizing task loss, guiding the fusion process toward the task objectives. TDFusion's training relies entirely on the downstream task loss, making it adaptable to any specific task. It can be applied to any architecture of fusion and task networks. Experiments demonstrate TDFusion's performance through fusion experiments conducted on four different datasets, in addition to evaluations on semantic segmentation and object detection tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。