提升手绘稿图像编辑的泛化能力,让模型更懂用户指令。
Rethinking Scribble-Guided Image Editing: Generalization, Instruction Adherence, and Multi-Tasking

- 分两阶段训练:先用合成数据学指令,再用真实数据调真实感。
- 用多任务拼接生成训练样本,低成本提升跨任务泛化能力。
- 聚焦编辑区域优化损失函数,提高编辑精度和效率。
手绘稿引导的图像编辑允许用户通过简单笔触与文本提示,精确指定图像修改的位置与方式,实现灵活交互。然而现有模型在该范式下性能仍不稳定,尤其在多任务场景中表现不佳。我们基于开源模型进行实证研究,发现指令层面的泛化(如跨任务、单任务到多任务)比图像领域泛化(如合成到真实、马赛克到常规图像)更难,表明主要瓶颈在于对多样化编辑指令的学习不足。为此提出三项策略:(a) 覆盖-真实感课程,先构建大规模合成指令数据进行广泛任务监督,再用少量真实数据优化生成真实感;(b) 多任务马赛克,通过近乎零成本拼接单任务样本来构造多任务训练样本,使模型能泛化至非马赛克图像;(c) 编辑聚焦损失,利用合成数据中输入输出变化区域,专注训练编辑区域,提升学习效率与编辑准确率。结合这些策略,在VIBE基准上显著提升单任务与多任务编辑性能,达到当前最优。数据集与模型将公开发布。
原文摘要 · Abstract (English)
Scribble-guided image editing allows users to combine simple scribble annotations with text prompts to specify both where and how an image should be edited, enabling flexible interaction with precise spatial control. However, existing models still exhibit unstable performance under this paradigm, especially in multi-task scenarios. To improve performance, we conduct empirical studies using an open-source editing model and reveal an asymmetry in generalization: instruction-level generalization, including across editing tasks and from single-task to multi-task settings, is more challenging than image-domain generalization, such as from synthetic to real-world images or from mosaicked to regular images. This suggests that the primary bottleneck lies in insufficient learning for diverse editing instructions rather than in the image domain gap. Motivated by this insight, we propose three strategies: (a) a Coverage-then-Realism Curriculum, a two-stage pipeline that first builds large-scale synthetic, instruction-rich data for broad task supervision, then curates a small set of real-world data to refine generation realism; (b) Multi-Task Mosaicking, which constructs multi-task training samples by concatenating single-task examples at nearly zero cost while enabling the learned capability to generalize to non-mosaicked images; and (c) an Edit-Focused Loss, which leverages the changed regions between input and output images in synthetic data to focus training on edited regions, improving both learning efficiency and editing accuracy. With these strategies, we substantially improve both single-task and multi-task scribble-guided editing on the VIBE benchmark, achieving state-of-the-art results. We will publicly release our dataset and model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。