无需训练和反演,用视觉自回归模型实现精准文本图像编辑。
Training-Free Text-Guided Image Editing with Visual Autoregressive Model
- 通过缓存机制捕捉图文关联,避免传统反演误差。
- 动态掩码仅修改相关区域,实现局部精准编辑。
- 训练免用、推理快(1.2秒/1K图像),适合快速原型设计。
文本引导的图像编辑能通过自然语言描述修改图像。现有基于扩散模型和修正流的方法依赖反演技术提取结构化噪声,但反演误差会传播,导致意外修改并损害保真度。即使反演完美,文本与图像特征纠缠也常引发全局变化,难以实现局部编辑。为此,我们提出基于视觉自回归模型(VAR)的新框架,无需显式反演即可实现精确可控的编辑。该方法引入缓存机制,存储原始图像的标记索引与概率分布,捕捉源提示与图像的关联。利用此缓存,设计自适应细粒度掩码策略,动态识别并限制修改区域,防止非预期改动。进一步采用标记重组装策略,提升编辑多样性、保真度与控制力。本框架无需训练,推理速度快,可于1.2秒内完成1024×1024图像编辑。大量实验表明,其性能在定量指标与视觉质量上均达到或超过现有扩散与修正流方法水平。代码将公开。
原文摘要 · Abstract (English)
Text-guided image editing is an essential task that enables users to modify images through natural language descriptions. Recent advances in diffusion models and rectified flows have significantly improved editing quality, primarily relying on inversion techniques to extract structured noise from input images. However, inaccuracies in inversion can propagate errors, leading to unintended modifications and compromising fidelity. Moreover, even with perfect inversion, the entanglement between textual prompts and image features often results in global changes when only local edits are intended. To address these challenges, we propose a novel text-guided image editing framework based on VAR (Visual AutoRegressive modeling), which eliminates the need for explicit inversion while ensuring precise and controlled modifications. Our method introduces a caching mechanism that stores token indices and probability distributions from the original image, capturing the relationship between the source prompt and the image. Using this cache, we design an adaptive fine-grained masking strategy that dynamically identifies and constrains modifications to relevant regions, preventing unintended changes. A token reassembling approach further refines the editing process, enhancing diversity, fidelity, and control. Our framework operates in a training-free manner and achieves high-fidelity editing with faster inference speeds, processing a 1K resolution image in as fast as 1.2 seconds. Extensive experiments demonstrate that our method achieves performance comparable to, or even surpassing, existing diffusion- and rectified flow-based approaches in both quantitative metrics and visual quality. The code will be released.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。