arXiv:2604.23763cs.CV2026-04

让扩散模型精准局部编辑,无需用户画遮罩。

Edit Where You Mean: Region-Aware Adapter Injection for Mask-Free Local Image Editing

论文配图:Edit Where You Mean: Region-Aware Adapter Injection for Mask-Free Local Image Editing
图 1 · 摘自论文原文
  • 用轻量适配器分离指令与位置信息,指导修改区域。
  • 在魔笔数据集上像素保留率超95%,编辑准确率领先。
  • 适合需要精准局部修改且不想画遮罩的用户。

大型扩散变换器(DiTs)能良好执行全局编辑指令,但常将局部编辑泄露到无关区域,因联合注意力架构缺乏明确信号指示修改位置。我们提出AdaptEdit,一种协同训练的、指令与区域感知的适配器框架,可在不修改骨干权重的前提下,将冻结的DiT转化为精确的局部编辑器。每个Transformer块中加入轻量级块适配器,注入结构化条件流,将要编辑的内容(指令语义)与位置(空间掩码)解耦;学习的SpatialGate仅将信号路由至目标区域,其余图像几乎保持不变;区域感知损失聚焦于变化像素的训练目标。由于这些组件使骨干内部表示具备掩码感知能力,可联合训练一个薄的掩码预测头,直接从指令和源图定位编辑区域——部署时无需用户掩码。我们在两个互补基准上评估:MagicBrush(有真实目标图像)衡量像素级保真度与编辑准确性,Emu-Edit Test(无真实图像,9种不同编辑类别)测试指令遵循与跨类型泛化能力。AdaptEdit在两者上均达当前最佳性能,同时优于无掩码与理想掩码基线。七组消融实验清晰分离各组件贡献。

原文摘要 · Abstract (English)

Large diffusion transformers (DiTs) follow global editing instructions well but consistently leak local edits into unrelated regions, because joint-attention architectures offer no explicit channel telling the network where to apply the edit. We introduce AdaptEdit, a co-trained, instruction- and region-aware adapter framework that retro-fits a frozen DiT into a precise local editor without modifying its backbone weights. A lightweight Block Adapter at every transformer block injects a structured condition stream that factorizes what to edit (instruction semantics) from where to edit (spatial mask); a learned SpatialGate routes the adapter signal selectively into the edit region while keeping the rest of the image near-identical to the source; and a Region-Aware Loss focuses the training objective on the changing pixels. Because these components make the backbone's internal representation mask-aware end-to-end, a thin MaskPredictor head trained jointly with the editor can ground the edit region directly from the instruction and source image -- eliminating any user-mask requirement at deployment. We evaluate on two complementary benchmarks: MagicBrush (paired ground-truth targets) to measure pixel-level preservation and edit accuracy, and Emu-Edit Test (no ground-truth images, 9 diverse edit categories) to stress-test instruction following and generalization across edit types. On both, AdaptEdit achieves state-of-the-art results, simultaneously outperforming mask-free and oracle-mask baselines. A seven-variant ablation cleanly isolates the contribution of each component.

图像编辑扩散模型局部修改无遮罩

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。