arXiv:2508.15772cs.CVcs.MM2025-08被引 20

用自回归模型实现精准指令图像编辑,速度更快且更贴合指令。

Visual Autoregressive Modeling for Instruction-Guided Image Editing

  • 将图像编辑转为多尺度预测,逐像素生成提升控制精度。
  • 在两个基准上得分超越扩散模型,512×512图仅需1.2秒。
  • 提出级联对齐参考模块,解决细粒度特征无法指导粗粒度生成的问题。

扩散模型虽能生成高保真图像,但其全局去噪机制易导致编辑区域与整体上下文混淆,引发非预期修改并削弱指令遵循性。相比之下,自回归模型通过离散视觉标记的序列生成过程,具备天然因果性和可组合性,更利于精确控制。本文提出VAREdit,一种视觉自回归框架,将图像编辑重构为多尺度目标特征的预测任务。给定源图像特征与文本指令,模型逐步生成目标特征。核心挑战在于如何有效利用源图像标记进行条件引导。我们发现,最细粒度的源特征难以有效指导粗粒度目标特征生成。为此,引入尺度对齐参考(SAR)模块,在首个自注意力层注入尺度匹配的条件信息。实验表明,VAREdit在EMU-Edit与PIE-Bench基准上显著优于主流扩散模型,CLIP与GPT评分均领先;且在512×512图像编辑中仅需1.2秒,比同规模的UltraEdit快2.2倍。代码已开源。

原文摘要 · Abstract (English)

Recent advances in diffusion models have brought remarkable visual fidelity to instruction-guided image editing. However, their global denoising process inherently entangles the edited region with the entire image context, leading to unintended spurious modifications and compromised adherence to editing instructions. In contrast, autoregressive models offer a distinct paradigm by formulating image synthesis as a sequential process over discrete visual tokens. Their causal and compositional mechanism naturally circumvents the adherence challenges of diffusion-based methods. In this paper, we present VAREdit, a visual autoregressive (VAR) framework that reframes image editing as a next-scale prediction problem. Conditioned on source image features and text instructions, VAREdit generates multi-scale target features to achieve precise edits. A core challenge in this paradigm is how to effectively condition the source image tokens. We observe that finest-scale source features cannot effectively guide the prediction of coarser target features. To bridge this gap, we introduce a Scale-Aligned Reference (SAR) module, which injects scale-matched conditioning information into the first self-attention layer. VAREdit demonstrates significant advancements in both editing adherence and efficiency. On EMU-Edit and PIE-Bench benchmarks, VAREdit outperforms leading diffusion-based methods by a substantial margin in terms of both CLIP and GPT scores. Moreover, VAREdit completes a 512$\times$512 editing in 1.2 seconds, making it 2.2$\times$ faster than the similarly sized UltraEdit. Code is available at: https://github.com/HiDream-ai/VAREdit.

图像编辑自回归模型多尺度生成指令控制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。