arXiv:2409.19403cs.CV2024-09ECCV被引 30

用掩码建模让模型专注图像内容,统一修复各种模糊、噪点等缺陷。

Restore Anything with Masks: Leveraging Mask Image Modeling for Blind All-in-One Image Restoration

论文配图:Restore Anything with Masks: Leveraging Mask Image Modeling for Blind All-in-One Image Restoration
图 1 · 摘自论文原文
  • 通过掩码图像建模提取图像内在信息,不区分具体退化类型。
  • 在多个数据集上达到当前最优性能,跨任务表现更均衡。
  • 适合需要统一修复多种图像问题的实用场景,如老旧照片修复。

全功能图像修复旨在使用单一模型处理多种退化类型。本文提出一种名为 Restore Anything with Masks(RAM)的简单流水线,专注于利用掩码图像建模提取图像内容本质信息,而非像其他方法那样区分退化类型。该流水线包含两个阶段:掩码图像预训练和基于掩码属性导通的微调。设计了一种专为全功能图像修复定制的简单掩码预训练方法,提升网络从各类退化中提取图像内容先验的能力,从而在不同修复任务间实现更均衡的表现并取得更强的整体效果。为在保持学习到的图像先验的同时弥补输入完整性差距,选择性地微调部分层。具体而言,通过提出的掩码属性导通(MAC)对各层重要性排序,选取贡献更高的层进行微调。大量实验表明,本方法达到最先进性能。代码与模型将发布于 https://github.com/Dragonisss/RAM。

原文摘要 · Abstract (English)

All-in-one image restoration aims to handle multiple degradation types using one model. This paper proposes a simple pipeline for all-in-one blind image restoration to Restore Anything with Masks (RAM). We focus on the image content by utilizing Mask Image Modeling to extract intrinsic image information rather than distinguishing degradation types like other methods. Our pipeline consists of two stages: masked image pre-training and fine-tuning with mask attribute conductance. We design a straightforward masking pre-training approach specifically tailored for all-in-one image restoration. This approach enhances networks to prioritize the extraction of image content priors from various degradations, resulting in a more balanced performance across different restoration tasks and achieving stronger overall results. To bridge the gap of input integrity while preserving learned image priors as much as possible, we selectively fine-tuned a small portion of the layers. Specifically, the importance of each layer is ranked by the proposed Mask Attribute Conductance (MAC), and the layers with higher contributions are selected for finetuning. Extensive experiments demonstrate that our method achieves state-of-the-art performance. Our code and model will be released at \href{https://github.com/Dragonisss/RAM}{https://github.com/Dragonisss/RAM}.

图像修复掩码建模统一修复

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。