用扩散模型生成可控制的渐进式素描,支持用户迭代编辑。
CoProSketch: Controllable and Progressive Sketch Generation with Diffusion Model
- 用无符号距离场表示素描,提升生成清晰度
- 支持从框选区域+文本生成粗略草图并逐步细化
- 首个大规模图文配对素描数据集,适合艺术创作工具开发
素描是艺术创作的基础蓝图,因其比像素级图像编辑更易操作而受青睐,但素描生成仍鲜有研究。本文提出 CoProSketch 框架,利用扩散模型实现高可控性与细节丰富的素描生成。通过微调预训练图像生成扩散模型,使用二值化素描图像进行训练,但发现模型难以生成清晰二值图像,导致输出混乱。为此,我们采用无符号距离场(UDF)表示素描,其为连续表示,可通过轻量网络高效解码为素描。用户可基于边界框与文本提示生成初始粗略草图,手动编辑后反馈至模型进行迭代优化,最终输出精细化素描。此外,我们构建了首个大规模文本-素描配对数据集用于训练。实验表明,该方法在语义一致性与可控性上优于基线,为生成流程中融入用户反馈提供了实用解决方案。
原文摘要 · Abstract (English)
Sketches serve as fundamental blueprints in artistic creation because sketch editing is easier and more intuitive than pixel-level RGB image editing for painting artists, yet sketch generation remains unexplored despite advancements in generative models. We propose a novel framework CoProSketch, providing prominent controllability and details for sketch generation with diffusion models. A straightforward method is fine-tuning a pretrained image generation diffusion model with binarized sketch images. However, we find that the diffusion models fail to generate clear binary images, which makes the produced sketches chaotic. We thus propose to represent the sketches by unsigned distance field (UDF), which is continuous and can be easily decoded to sketches through a lightweight network. With CoProSketch, users generate a rough sketch from a bounding box and a text prompt. The rough sketch can be manually edited and fed back into the model for iterative refinement and will be decoded to a detailed sketch as the final result. Additionally, we curate the first large-scale text-sketch paired dataset as the training data. Experiments demonstrate superior semantic consistency and controllability over baselines, offering a practical solution for integrating user feedback into generative workflows.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。