arXiv:2503.14944cs.CV2025-03ICCV被引 10

一个能处理多种退化、多任务且支持语言引导的图像融合框架。

MMAIF: Multi-task and Multi-degradation All-in-One for Image Fusion with Language Guidance

  • 基于潜在空间的统一扩散变换器,支持多退化输入
  • 在真实退化数据上超越现有方法,提升融合质量
  • 支持用户语言指令交互,适合实际应用

图像融合是低层视觉中的基础任务,旨在将多幅图像序列整合为单一输出,尽可能保留输入信息。然而现有方法存在四大局限:1)需针对特定任务或数据集训练模型;2)忽略真实世界图像退化(如噪声),导致处理退化输入时失败;3)在像素空间操作,注意力机制计算开销大;4)缺乏用户交互能力。为此,我们提出一个统一框架,实现多任务、多退化、语言引导的图像融合。框架包含两个核心组件:1)模拟真实退化并生成交互提示的实用退化流水线;2)在潜在空间运行的全功能扩散变换器(DiT),以退化输入和生成提示为条件融合清晰图像。此外,我们对原始DiT架构进行了合理改进以适配融合任务。基于此框架,开发了基于回归与流匹配的两种模型变体。大量定性与定量实验表明,该方法有效克服前述限制,显著优于现有恢复+融合及全一体化方案。代码已开源。

原文摘要 · Abstract (English)

Image fusion, a fundamental low-level vision task, aims to integrate multiple image sequences into a single output while preserving as much information as possible from the input. However, existing methods face several significant limitations: 1) requiring task- or dataset-specific models; 2) neglecting real-world image degradations (\textit{e.g.}, noise), which causes failure when processing degraded inputs; 3) operating in pixel space, where attention mechanisms are computationally expensive; and 4) lacking user interaction capabilities. To address these challenges, we propose a unified framework for multi-task, multi-degradation, and language-guided image fusion. Our framework includes two key components: 1) a practical degradation pipeline that simulates real-world image degradations and generates interactive prompts to guide the model; 2) an all-in-one Diffusion Transformer (DiT) operating in latent space, which fuses a clean image conditioned on both the degraded inputs and the generated prompts. Furthermore, we introduce principled modifications to the original DiT architecture to better suit the fusion task. Based on this framework, we develop two versions of the model: Regression-based and Flow Matching-based variants. Extensive qualitative and quantitative experiments demonstrate that our approach effectively addresses the aforementioned limitations and outperforms previous restoration+fusion and all-in-one pipelines. Codes are available at https://github.com/294coder/MMAIF.

图像融合扩散模型多退化语言引导

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。