arXiv:2506.04158cs.CV2025-06NeurIPS被引 8

用程序化方式让扩散模型精准执行复杂图像编辑。

Image Editing As Programs with Diffusion Models

  • 将编辑指令拆解为原子操作序列,由轻量适配器实现。
  • 在多步复杂编辑任务中,准确率与语义保真度显著优于现有方法。
  • 适合需要结构大变动的高级图像编辑场景。

尽管扩散模型在文本到图像生成方面取得显著进展,但在指令驱动的图像编辑中仍面临重大挑战。本研究指出关键问题:模型对涉及显著布局变化的结构不一致编辑尤为困难。为此,我们提出基于扩散Transformer(DiT)架构的统一图像编辑框架IEAP。IEAP从还原论视角出发,将复杂编辑指令分解为一系列原子操作,每个操作由共享相同DiT主干的轻量适配器实现,并针对特定类型编辑进行专业化设计。由视觉-语言模型(VLM)驱动的代理程序化调度这些操作,协同完成任意且结构不一致的变换。通过模块化与序列化编辑,IEAP在多种编辑任务上表现出强泛化能力,涵盖从简单调整到大规模结构变化。大量实验表明,该框架在标准基准测试中显著优于现有最优方法,在各类编辑场景下均展现出更高精度与更强语义保真度。代码已开源于https://github.com/YujiaHu1109/IEAP。

原文摘要 · Abstract (English)

While diffusion models have achieved remarkable success in text-to-image generation, they encounter significant challenges with instruction-driven image editing. Our research highlights a key challenge: these models particularly struggle with structurally inconsistent edits that involve substantial layout changes. To mitigate this gap, we introduce Image Editing As Programs (IEAP), a unified image editing framework built upon the Diffusion Transformer (DiT) architecture. At its core, IEAP approaches instructional editing through a reductionist lens, decomposing complex editing instructions into sequences of atomic operations. Each operation is implemented via a lightweight adapter sharing the same DiT backbone and is specialized for a specific type of edit. Programmed by a vision-language model (VLM)-based agent, these operations collaboratively support arbitrary and structurally inconsistent transformations. By modularizing and sequencing edits in this way, IEAP generalizes robustly across a wide range of editing tasks, from simple adjustments to substantial structural changes. Extensive experiments demonstrate that IEAP significantly outperforms state-of-the-art methods on standard benchmarks across various editing scenarios. In these evaluations, our framework delivers superior accuracy and semantic fidelity, particularly for complex, multi-step instructions. Codes are available at https://github.com/YujiaHu1109/IEAP.

图像编辑扩散模型程序化生成结构变化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。