arXiv:2409.00522cs.CV2024-09被引 4

通过反向擦除图像实现物体分步精准插入,效果更自然。

EraseDraw: Learning to Draw Step-by-Step via Erasing Objects from Images

  • 先擦除再逆向生成,利用现有模型强项构建新数据。
  • 在真实场景图像中插入物体,空间与光影一致性显著提升。
  • 支持多种场景和提示,适合需要精细图像编辑的用户。

创作过程如绘画常需逐个构建图像元素。能否构建计算模型完成此任务?以往方法常全局修改图像,导致物体位置不自然、光照失真。我们发现当前先进模型虽在插入物体上表现不佳,却能高效擦除物体与背景。反向利用此能力,可自动生成高质量的物体插入训练数据。基于此可扩展的数据生成流程,我们构建了用于学习物体插入的专用数据集,并训练出一个文本条件扩散模型。定性与定量实验表明,该模型在物体插入任务上达到当前最佳表现,尤其在真实复杂图像中优势明显。我们展示了多样提示和跨领域图像上的出色结果。此外,通过结合CLIP引导的束搜索,实现了迭代式物体逐步插入的自动化。

原文摘要 · Abstract (English)

Creative processes such as painting often involve creating different components of an image one by one. Can we build a computational model to perform this task? Prior works often fail by making global changes to the image, inserting objects in unrealistic spatial locations, and generating inaccurate lighting details. We observe that while state-of-the-art models perform poorly on object insertion, they can remove objects and erase the background in natural images very well. Inverting the direction of object removal, we obtain high-quality data for learning to insert objects that are spatially, physically, and optically consistent with the surroundings. With this scalable automatic data generation pipeline, we can create a dataset for learning object insertion, which is used to train our proposed text conditioned diffusion model. Qualitative and quantitative experiments have shown that our model achieves state-of-the-art results in object insertion, particularly for in-the-wild images. We show compelling results on diverse insertion prompts and images across various domains.In addition, we automate iterative insertion by combining our insertion model with beam search guided by CLIP.

图像编辑扩散模型生成建模

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。