用流匹配统一多模态图像融合,提速并提升质量。
FusionFM: All-in-One Multi-Modal Image Fusion with Flow Matching
- 用流匹配直接建模模态到融合图像的分布传输。
- 通过伪标签筛选与精修模块,解决高质量监督数据不足问题。
- 支持多任务持续学习,兼顾效率与模型稳定性。
现有多模态图像融合方法多依赖特定任务模型,训练成本高且可扩展性差。生成式方法虽提供统一建模视角,但因复杂的从噪声到图像的采样路径导致推理缓慢。为此,本文将图像融合建模为从源模态到融合图像分布的直接概率传输,利用流匹配范式提升采样效率与结构一致性。为缓解高质量融合图像监督数据匮乏问题,我们收集多个前沿模型的融合结果作为先验,并设计任务感知选择函数,为每项任务挑选最可靠的伪标签。进一步引入融合精修模块,采用分而治之策略系统识别、分解并增强选定伪标签中的退化成分。针对多任务场景,集成弹性权重固化与经验回放机制,在参数稳定性和记忆保留两方面提升持续学习能力。本方法在多种融合任务中表现优异,显著提升采样效率,同时保持轻量模型设计。代码将于 https://github.com/Ist-Zhy/FusionFM 公开。
原文摘要 · Abstract (English)
Current multi-modal image fusion methods typically rely on task-specific models, leading to high training costs and limited scalability. While generative methods provide a unified modeling perspective, they often suffer from slow inference due to the complex sampling trajectories from noise to image. To address this, we formulate image fusion as a direct probabilistic transport from source modalities to the fused image distribution, leveraging the flow matching paradigm to improve sampling efficiency and structural consistency. To mitigate the lack of high-quality fused images for supervision, we collect fusion results from multiple state-of-the-art models as priors, and employ a task-aware selection function to select the most reliable pseudo-labels for each task. We further introduce a Fusion Refiner module that employs a divide-and-conquer strategy to systematically identify, decompose, and enhance degraded components in selected pseudo-labels. For multi-task scenarios, we integrate elastic weight consolidation and experience replay mechanisms to preserve cross-task performance and enhance continual learning ability from both parameter stability and memory retention perspectives. Our approach achieves competitive performance across diverse fusion tasks, while significantly improving sampling efficiency and maintaining a lightweight model design. The code will be available at: https://github.com/Ist-Zhy/FusionFM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。