让理解模块负责设计,生成模块专注绘画,提升图像编辑精度。
Draw-In-Mind: Rebalancing Designer-Painter Roles in Unified Multimodal Models Benefits Image Editing
- 将设计任务交给理解模块,实现角色再平衡
- 在ImgEdit和GEdit-Bench上超越更大模型
- 适合关注图像编辑性能优化的研究者
近年来,将多模态理解与生成统一于一个模型成为趋势。尽管文本到图像生成表现良好,但精确图像编辑仍存挑战。我们发现根源在于职责分配失衡:理解模块仅作指令翻译,而生成模块需同时完成布局推断、目标区域识别与内容渲染。这违背了理解模块训练数据量远超生成模块的现实。为此,我们提出Draw-In-Mind(DIM)数据集,包含1400万长上下文图文对(DIM-T2I)用于增强指令理解,以及23.3万条由GPT-4o生成的思维链图像构想(DIM-Edit)作为显式设计蓝图。通过轻量级两层MLP连接冻结的Qwen2.5-VL-3B与可训练的SANA1.5-1.6B,在DIM数据集上训练出DIM-4.6B-T2I/Edit。尽管参数量小,其在ImgEdit和GEdit-Bench基准上达到或超过更大型模型如UniWorld-V1和Step1X-Edit的表现,证明将设计责任明确赋予理解模块能显著提升图像编辑能力。相关数据集与模型已开源。
原文摘要 · Abstract (English)
In recent years, integrating multimodal understanding and generation into a single unified model has emerged as a promising paradigm. While this approach achieves strong results in text-to-image (T2I) generation, it still struggles with precise image editing. We attribute this limitation to an imbalanced division of responsibilities. The understanding module primarily functions as a translator that encodes user instructions into semantic conditions, while the generation module must simultaneously act as designer and painter, inferring the original layout, identifying the target editing region, and rendering the new content. This imbalance is counterintuitive because the understanding module is typically trained with several times more data on complex reasoning tasks than the generation module. To address this issue, we introduce Draw-In-Mind (DIM), a dataset comprising two complementary subsets: (i) DIM-T2I, containing 14M long-context image-text pairs to enhance complex instruction comprehension; and (ii) DIM-Edit, consisting of 233K chain-of-thought imaginations generated by GPT-4o, serving as explicit design blueprints for image edits. We connect a frozen Qwen2.5-VL-3B with a trainable SANA1.5-1.6B via a lightweight two-layer MLP, and train it on the proposed DIM dataset, resulting in DIM-4.6B-T2I/Edit. Despite its modest parameter scale, DIM-4.6B-Edit achieves SOTA or competitive performance on the ImgEdit and GEdit-Bench benchmarks, outperforming much larger models such as UniWorld-V1 and Step1X-Edit. These findings demonstrate that explicitly assigning the design responsibility to the understanding module provides significant benefits for image editing. Our dataset and models are available at https://github.com/showlab/DIM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。