arXiv:2509.23919cs.CV2025-09AAAI被引 2

无需训练即可实现精准文本引导图像修复,保持背景一致性。

Token Painter: Training-Free Text-Guided Image Inpainting via Mask Autoregressive Models

  • 用掩码自回归模型生成掩码区域内容,局部控制更精准。
  • 在多个指标上超越现有最先进方法,修复内容与提示高度匹配。
  • 适合需要快速部署、无需训练的图像修复场景。

文本引导图像修复旨在根据文本提示修复图像中的遮挡区域,同时保持背景一致。尽管基于扩散模型的方法已成为主流,但其在潜在空间中建模整个图像的特性导致结果难以与提示细节对齐且背景一致性差。为此,我们探索使用掩码自回归(MAR)模型解决该问题。MAR通过生成对应掩码区域的潜在标记,天然支持图像修复,实现更好的局部可控性而不改变背景。然而,直接应用MAR会导致修复内容忽略提示或与背景不协调。通过对修复图像注意力图的分析,我们发现背景标记会影响文本标记的生成过程,并据此设计了无需训练的文本引导图像修复方法「Token Painter」。该方法包含两个关键组件:(1) 双流编码器信息融合(DEIF),在频域融合文本与背景的语义和上下文信息,生成新型引导标记,使MAR既能忠实于提示又能与背景和谐;(2) 自适应解码器注意力分数增强(ADAE),自适应增强引导标记与修复标记的注意力分数,进一步提升提示细节对齐与视觉质量。大量实验表明,该无需训练的方法在几乎所有指标上均优于先前最先进方法。

原文摘要 · Abstract (English)

Text-guided image inpainting aims to inpaint masked image regions based on a textual prompt while preserving the background. Although diffusion-based methods have become dominant, their property of modeling the entire image in latent space makes it challenging for the results to align well with prompt details and maintain a consistent background. To address these issues, we explore Mask AutoRegressive (MAR) models for this task. MAR naturally supports image inpainting by generating latent tokens corresponding to mask regions, enabling better local controllability without altering the background. However, directly applying MAR to this task makes the inpainting content either ignore the prompts or be disharmonious with the background context. Through analysis of the attention maps from the inpainting images, we identify the impact of background tokens on text tokens during the MAR generation, and leverage this to design \textbf{Token Painter}, a training-free text-guided image inpainting method based on MAR. Our approach introduces two key components: (1) Dual-Stream Encoder Information Fusion (DEIF), which fuses the semantic and context information from text and background in frequency domain to produce novel guidance tokens, allowing MAR to generate text-faithful inpainting content while keeping harmonious with background context. (2) Adaptive Decoder Attention Score Enhancing (ADAE), which adaptively enhances attention scores on guidance tokens and inpainting tokens to further enhance the alignment of prompt details and the content visual quality. Extensive experiments demonstrate that our training-free method outperforms prior state-of-the-art methods across almost all metrics. Codes: https://github.com/longtaojiang/Token-Painter.

图像修复文本引导自回归模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。