arXiv:2410.15385cs.CV2024-10被引 26

用低秩专家动态适配复杂图像退化,实现高效全能修复。

LoRA-IR: Taming Low-Rank Experts for Efficient All-in-One Image Restoration

  • 通过低秩适配和专家路由机制,动态组合修复模块。
  • 在14项任务、29个基准上达到顶尖性能,计算开销小。
  • 适合需要快速适应多种退化场景的图像修复应用。

基于提示的全能图像修复框架通过将退化特异性信息融入提示模块取得了显著效果。然而,在真实场景中面对复杂多样的退化问题仍具挑战。为此,我们提出LoRA-IR,一个灵活的高效框架,通过动态调用紧凑的低秩专家实现全能图像修复。该框架包含两个训练阶段:退化引导预训练与参数高效微调。预训练阶段,通过简单机制将预训练CLIP模型扩展至更高分辨率,以提取鲁棒的退化表征,自适应引导修复网络。微调阶段,采用低秩适配(LoRA)优化预训练修复网络。基于混合专家(MoE)架构,LoRA-IR通过退化引导路由器动态集成多个低秩修复专家,显著提升对复杂现实场景中多样且未知退化的适应能力。大量实验表明,LoRA-IR在14项图像修复任务和29个基准上均达到当前最优性能,同时保持计算高效。代码与预训练模型将在https://github.com/shallowdream204/LoRA-IR发布。

原文摘要 · Abstract (English)

Prompt-based all-in-one image restoration (IR) frameworks have achieved remarkable performance by incorporating degradation-specific information into prompt modules. Nevertheless, handling the complex and diverse degradations encountered in real-world scenarios remains a significant challenge. To tackle this, we propose LoRA-IR, a flexible framework that dynamically leverages compact low-rank experts to facilitate efficient all-in-one image restoration. Specifically, LoRA-IR consists of two training stages: degradation-guided pre-training and parameter-efficient fine-tuning. In the pre-training stage, we enhance the pre-trained CLIP model by introducing a simple mechanism that scales it to higher resolutions, allowing us to extract robust degradation representations that adaptively guide the IR network. In the fine-tuning stage, we refine the pre-trained IR network through low-rank adaptation (LoRA). Built upon a Mixture-of-Experts (MoE) architecture, LoRA-IR dynamically integrates multiple low-rank restoration experts through a degradation-guided router. This dynamic integration mechanism significantly enhances our model's adaptability to diverse and unknown degradations in complex real-world scenarios. Extensive experiments demonstrate that LoRA-IR achieves SOTA performance across 14 IR tasks and 29 benchmarks, while maintaining computational efficiency. Code and pre-trained models will be available at: https://github.com/shallowdream204/LoRA-IR.

图像修复低秩适配专家混合高效模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。