arXiv:2512.22525cs.CV2025-12被引 1

用手绘草图+文字,实现更精准的图像编辑与生成。

DreamOmni3: Scribble-based Editing and Generation

  • 引入手绘草图作为输入,结合文本和图像进行交互式编辑。
  • 在多个任务上超越现有方法,支持复杂多区域修改。
  • 适合需要精细控制图像创作的设计师或开发者使用。

近期统一生成与编辑模型在指令驱动的图像操作中表现优异,但主要依赖文本提示,难以准确捕捉用户意图的编辑位置和细粒度视觉细节。为此,我们提出基于手绘草图的编辑与生成两项新任务,支持在图形界面中融合文本、图像和自由手绘草图,实现更灵活的创作。我们构建了DreamOmni3,解决数据构建与框架设计两大挑战。数据合成包含两部分:基于草图的编辑任务(四类)与基于草图的生成任务(三类)。基于DreamOmni2数据集,通过提取可编辑区域并叠加手绘矩形、圆形、涂鸦或裁剪图像,构建训练数据。框架方面,摒弃传统二值掩码,采用联合输入机制,将原始图与草图同时输入模型,并以不同颜色区分区域,简化处理流程。通过共享索引与位置编码,模型可精确定位草图区域,保持编辑准确性。我们建立了全面基准评估体系,实验表明DreamOmni3性能显著领先,相关模型与代码将公开发布。

原文摘要 · Abstract (English)

Recently unified generation and editing models have achieved remarkable success with their impressive performance. These models rely mainly on text prompts for instruction-based editing and generation, but language often fails to capture users intended edit locations and fine-grained visual details. To this end, we propose two tasks: scribble-based editing and generation, that enables more flexible creation on graphical user interface (GUI) combining user textual, images, and freehand sketches. We introduce DreamOmni3, tackling two challenges: data creation and framework design. Our data synthesis pipeline includes two parts: scribble-based editing and generation. For scribble-based editing, we define four tasks: scribble and instruction-based editing, scribble and multimodal instruction-based editing, image fusion, and doodle editing. Based on DreamOmni2 dataset, we extract editable regions and overlay hand-drawn boxes, circles, doodles or cropped image to construct training data. For scribble-based generation, we define three tasks: scribble and instruction-based generation, scribble and multimodal instruction-based generation, and doodle generation, following similar data creation pipelines. For the framework, instead of using binary masks, which struggle with complex edits involving multiple scribbles, images, and instructions, we propose a joint input scheme that feeds both the original and scribbled source images into the model, using different colors to distinguish regions and simplify processing. By applying the same index and position encodings to both images, the model can precisely localize scribbled regions while maintaining accurate editing. Finally, we establish comprehensive benchmarks for these tasks to promote further research. Experimental results demonstrate that DreamOmni3 achieves outstanding performance, and models and code will be publicly released.

图像编辑草图生成多模态交互设计

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。