用简单涂鸦指导图像生成，无需训练即可精准控制物体位置方向。
Scribble-Guided Diffusion for Training-free Text-to-Image Generation
- 用户画涂鸦作为视觉提示，无需训练直接引导生成
- 在PASCAL-Scribble上空间控制准确率显著提升
- 适合需要精细布局控制的创意设计场景
近期文本到图像扩散模型取得了显著进展，但往往难以完全捕捉用户意图。现有方法使用文本结合边界框或区域掩码，难以提供精确的空间引导，常导致物体位置错位或方向错误。为此，我们提出Scribble-Guided Diffusion（ScribbleDiff），一种无需训练的方法，利用用户提供的简单涂鸦作为视觉提示来引导图像生成。然而，涂鸦本身稀疏且细小，给扩散模型带来对齐挑战。为此，我们引入了矩量对齐与涂鸦传播机制，实现生成图像与涂鸦输入之间更有效、灵活的对齐。在PASCAL-Scribble数据集上的实验表明，该方法在空间控制和一致性方面均有显著提升，验证了基于涂鸦引导在扩散模型中的有效性。代码已开源：https://github.com/kaist-cvml-lab/scribble-diffusion。
原文摘要 · Abstract (English)
Recent advancements in text-to-image diffusion models have demonstrated remarkable success, yet they often struggle to fully capture the user's intent. Existing approaches using textual inputs combined with bounding boxes or region masks fall short in providing precise spatial guidance, often leading to misaligned or unintended object orientation. To address these limitations, we propose Scribble-Guided Diffusion (ScribbleDiff), a training-free approach that utilizes simple user-provided scribbles as visual prompts to guide image generation. However, incorporating scribbles into diffusion models presents challenges due to their sparse and thin nature, making it difficult to ensure accurate orientation alignment. To overcome these challenges, we introduce moment alignment and scribble propagation, which allow for more effective and flexible alignment between generated images and scribble inputs. Experimental results on the PASCAL-Scribble dataset demonstrate significant improvements in spatial control and consistency, showcasing the effectiveness of scribble-based guidance in diffusion models. Our code is available at https://github.com/kaist-cvml-lab/scribble-diffusion.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。