用少量优化步骤提升通用模型的修复能力，无需重新训练。
SONIC: Spectral Optimization of Noise for Inpainting with Consistency
- 在频域用Adam优化初始噪声，快速逼近真实图像。
- 仅需数十步优化，就能让通用模型实现最优修复效果。
- 适合希望不训练模型却获得高质量修复的开发者使用。
我们提出一种无需训练的新方法，利用现成的文生图模型进行图像修复。尽管基于引导的方法理论上可使通用模型解决修复等逆问题，但实际效果受限，导致仍需专用修复模型。本文认为，问题的关键在于初始噪声样本的优化不足。我们通过最少数十次优化步骤，将初始噪声调整为近似未遮挡图像，再结合传统无训练修复方法。核心创新包括：(i) 采用线性近似避免昂贵且不切实际的反向传播计算，使这一关系首次可被有效利用；(ii) 在频域中使用Adam优化初始噪声，通过谱预处理稳定优化过程。我们在多种修复任务上验证了该方法，性能超越现有最先进水平。
原文摘要 · Abstract (English)
We propose a novel training-free method for inpainting with off-the-shelf text-to-image models. While guidance-based methods in theory allow generic models to be used for inverse problems such as inpainting -- in practice their effectiveness is limited, leading to the necessity of specialized inpainting-specific models. In this work, we argue the missing ingredient for training-free generic model usage is proper optimization of the initial noise sample. We optimize the initial noise to approximately reproduce the unmasked image, in as few as tens of optimization steps, then use it with a conventional training-free inpainting method. Critically, we propose two core ideas that make this possible: (i) we perform linear approximation that avoids the costly and often impractical unrolling required to relate the initial noise sample to model output -- which potentially is why this relationship was previously overlooked; and (ii) perform spectral preconditioning by optimizing the initial noise sample in the spectral domain with Adam, which stabilizes the optimization. We demonstrate our method on various inpainting tasks, outperforming the state of the art. Project website: https://ubc-vision.github.io/sonic/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。