用4步生成即可完成零样本图像修复,速度快且效果好
Zero-Shot Image Restoration Using Few-Step Guidance of Consistency Models (and Beyond)
- 结合初始化优化与新型噪声注入机制,仅需4次函数评估
- 在超分辨率、去模糊和补全任务上均优于现有方法
- 适合追求快速修复且无需微调的开发者或应用者
近年来,使用单个预训练扩散模型(DM)和数据保真度引导来解决图像修复任务成为趋势,而无需为每个任务单独训练深度网络。然而,现有的“零样本”修复方法通常需要大量神经函数评估(NFE),这主要源于原始生成功能中高NFE需求。近期提出的更快版本扩散模型包括一致性模型(CM),可在少数几步内生成样本。但现有基于引导的一致性模型在修复任务中仍需数十次NFE,或需针对每项任务微调模型,若假设不准确则性能下降。本文提出一种零样本修复方案,利用一致性模型,仅需4次NFE即可良好运行。该方法融合了更优的初始化、反投影引导以及关键的新型噪声注入机制。我们在图像超分辨率、去模糊和图像补全任务上验证了该方法的优势。有趣的是,我们发现这种噪声注入技术不仅适用于一致性模型,还能缓解现有引导扩散模型在减少NFE时的性能下降。
原文摘要 · Abstract (English)
In recent years, it has become popular to tackle image restoration tasks with a single pretrained diffusion model (DM) and data-fidelity guidance, instead of training a dedicated deep neural network per task. However, such "zero-shot" restoration schemes currently require many Neural Function Evaluations (NFEs) for performing well, which may be attributed to the many NFEs needed in the original generative functionality of the DMs. Recently, faster variants of DMs have been explored for image generation. These include Consistency Models (CMs), which can generate samples via a couple of NFEs. However, existing works that use guided CMs for restoration still require tens of NFEs or fine-tuning of the model per task that leads to performance drop if the assumptions during the fine-tuning are not accurate. In this paper, we propose a zero-shot restoration scheme that uses CMs and operates well with as little as 4 NFEs. It is based on a wise combination of several ingredients: better initialization, back-projection guidance, and above all a novel noise injection mechanism. We demonstrate the advantages of our approach for image super-resolution, deblurring and inpainting. Interestingly, we show that the usefulness of our noise injection technique goes beyond CMs: it can also mitigate the performance degradation of existing guided DM methods when reducing their NFE count.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。