用掩码生成Transformer实现精准图像编辑,避免非目标区域被误改。
EditMGT: Unleashing Potentials of Masked Generative Transformers in Image Editing
- 通过多层注意力融合实现编辑区域精确定位
- 采用区域保留采样策略,提升修改的局部可控性
- 在少于10亿参数下实现6倍加速,效果优于扩散模型
扩散模型(DMs)虽在图像编辑中表现优异,但其全局去噪机制常将局部编辑目标与全图上下文混淆,导致非目标区域意外修改。本文转向掩码生成Transformer(MGT),利用其局部解码特性,天然具备保留无关区域的能力。提出首个基于MGT的图像编辑框架EditMGT:首先利用跨注意力图提供编辑区域定位信号,设计多层注意力融合机制实现细粒度定位;在此基础上引入区域保留采样,限制低注意力区域的令牌翻转,抑制无效修改,确保仅在目标区域进行更新。为训练该模型,构建了涵盖七类编辑任务的高分辨率数据集CrispEdit-2M。无需额外参数,通过注意力注入将预训练文本到图像MGT转化为编辑模型。在四个标准基准上实验表明,仅需不到10亿参数,即可实现与现有方法相当甚至更优的编辑质量,其中风格迁移任务提升17.6%,风格改变任务提升3.6%,且推理速度提升6倍。
原文摘要 · Abstract (English)
Recent advances in diffusion models (DMs) have achieved exceptional visual quality in image editing tasks. However, the global denoising dynamics of DMs inherently conflate local editing targets with the full-image context, leading to unintended modifications in non-target regions. In this paper, we shift our attention beyond DMs and turn to Masked Generative Transformers (MGTs) as an alternative approach to tackle this challenge. By predicting multiple masked tokens rather than holistic refinement, MGTs exhibit a localized decoding paradigm that endows them with the inherent capacity to explicitly preserve non-relevant regions during the editing process. Building upon this insight, we introduce the first MGT-based image editing framework, termed EditMGT. We first demonstrate that MGT's cross-attention maps provide informative localization signals for localizing edit-relevant regions and devise a multi-layer attention consolidation scheme that refines these maps to achieve fine-grained and precise localization. On top of these adaptive localization results, we introduce region-hold sampling, which restricts token flipping within low-attention areas to suppress spurious edits, thereby confining modifications to the intended target regions and preserving the integrity of surrounding non-target areas. To train EditMGT, we construct CrispEdit-2M, a high-resolution dataset spanning seven diverse editing categories. Without introducing additional parameters, we adapt a pre-trained text-to-image MGT into an image editing model through attention injection. Extensive experiments across four standard benchmarks demonstrate that, with fewer than 1B parameters, our model achieves similarity performance while enabling 6 times faster editing. Moreover, it delivers comparable or superior editing quality, with improvements of 3.6% and 17.6% on style change and style transfer tasks, respectively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。