通过重定向注意力机制,让扩散模型精准擦除物体并自然补全背景。
Attentive Eraser: Unleashing Diffusion Model's Object Removal Potential via Self-Attention Redirection Guidance
- 基于掩码重写注意力机制,优先关注背景区域。
- 生成内容更合理,无随机伪影,补全效果自然。
- 无需微调,适配多种模型,适合图像编辑场景。
最近,扩散模型在图像生成领域表现出色,但在物体移除任务中仍存在生成随机伪影、无法合理补全前景区域等问题。为此,本文提出 Attentive Eraser,一种无需微调的方法,可使预训练扩散模型稳定高效地完成物体移除。首先,观察到自注意力图影响图像结构与细节,提出注意力激活与抑制(ASS),根据给定掩码重构预训练扩散模型中的自注意力机制,使反向生成过程更关注背景而非前景物体。进一步提出自注意力重定向引导(SARG),利用 ASS 重定向后的注意力指导生成,有效移除掩码内物体的同时,生成合理且连贯的内容。实验表明,Attentive Eraser 在多种预训练扩散模型上均表现稳定有效,性能甚至超越部分训练方法。该方法可适配多种扩散模型架构与检查点,具备良好可扩展性。代码已开源:https://github.com/Anonym0u3/AttentiveEraser。
原文摘要 · Abstract (English)
Recently, diffusion models have emerged as promising newcomers in the field of generative models, shining brightly in image generation. However, when employed for object removal tasks, they still encounter issues such as generating random artifacts and the incapacity to repaint foreground object areas with appropriate content after removal. To tackle these problems, we propose Attentive Eraser, a tuning-free method to empower pre-trained diffusion models for stable and effective object removal. Firstly, in light of the observation that the self-attention maps influence the structure and shape details of the generated images, we propose Attention Activation and Suppression (ASS), which re-engineers the self-attention mechanism within the pre-trained diffusion models based on the given mask, thereby prioritizing the background over the foreground object during the reverse generation process. Moreover, we introduce Self-Attention Redirection Guidance (SARG), which utilizes the self-attention redirected by ASS to guide the generation process, effectively removing foreground objects within the mask while simultaneously generating content that is both plausible and coherent. Experiments demonstrate the stability and effectiveness of Attentive Eraser in object removal across a variety of pre-trained diffusion models, outperforming even training-based methods. Furthermore, Attentive Eraser can be implemented in various diffusion model architectures and checkpoints, enabling excellent scalability. Code is available at https://github.com/Anonym0u3/AttentiveEraser.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。