用语言提示控制图像融合,自动修复复杂退化问题
ControlFusion: A Controllable Image Fusion Framework with Language-Vision Degradation Prompts
- 通过物理模型模拟真实场景复合退化,从数据层面提升鲁棒性
- 结合文本提示与视觉感知,动态调节融合过程,支持用户定制
- 在真实退化场景下优于现有方法,适合需要灵活调控的图像处理任务
当前图像融合方法难以应对真实成像中复杂的复合退化,且缺乏满足用户个性化需求的灵活性。为此,我们提出一种基于语言-视觉退化提示的可控图像融合框架ControlFusion,可自适应消除复合退化。一方面,构建融合Retinex理论与大气散射原理的退化成像模型,模拟真实复杂退化,为数据级修复提供可能;另一方面,设计提示调制的恢复与融合网络,通过退化提示动态增强特征,实现对不同等级退化的灵活适配。针对用户感知差异,引入文本编码器将用户指定的退化类型和严重程度转化为退化提示;同时设计空间-频率协同视觉适配器,自主感知源图退化,减少对人工指令的依赖。大量实验表明,ControlFusion在融合质量与退化处理能力上均优于现有最先进方法,尤其在真实世界复合退化场景下表现突出。代码已公开于https://github.com/Linfeng-Tang/ControlFusion。
原文摘要 · Abstract (English)
Current image fusion methods struggle to address the composite degradations encountered in real-world imaging scenarios and lack the flexibility to accommodate user-specific requirements. In response to these challenges, we propose a controllable image fusion framework with language-vision prompts, termed ControlFusion, which adaptively neutralizes composite degradations. On the one hand, we develop a degraded imaging model that integrates physical imaging mechanisms, including the Retinex theory and atmospheric scattering principle, to simulate composite degradations, thereby providing potential for addressing real-world complex degradations from the data level. On the other hand, we devise a prompt-modulated restoration and fusion network that dynamically enhances features with degradation prompts, enabling our method to accommodate composite degradation of varying levels. Specifically, considering individual variations in quality perception of users, we incorporate a text encoder to embed user-specified degradation types and severity levels as degradation prompts. We also design a spatial-frequency collaborative visual adapter that autonomously perceives degradations in source images, thus eliminating the complete dependence on user instructions. Extensive experiments demonstrate that ControlFusion outperforms SOTA fusion methods in fusion quality and degradation handling, particularly in countering real-world and compound degradations with various levels. The source code is publicly available at https://github.com/Linfeng-Tang/ControlFusion.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。