无需训练即可实现多种文本图像操作,支持删改与风格控制。
OmniText: A Training-Free Generalist for Controllable Text-Image Manipulation
- 利用注意力机制反演与重分配,实现无训练文本移除与风格控制。
- 在多个任务上达到当前最优性能,且与专用方法相当。
- 提供首个通用文本图像操作基准数据集,覆盖多种实际场景。
基于扩散模型的文本合成技术在图像内插入和编辑文本方面表现优异,但现有文本修复方法仍存在三大局限:无法移除文本、难以控制渲染风格、易生成重复字符。为此,我们提出OmniText,一种无需训练的通用框架,可执行多样化的文本图像操作(TIM)。通过研究交叉注意力与自注意力机制,我们发现自注意力反演能缓解模型对周围文本的关注,减少幻觉;同时重新分配交叉注意力,提升特定文本标记概率以降低幻觉。针对可控修复,我们在潜在空间优化中引入新损失函数:交叉注意力内容损失提升文本准确性,自注意力风格损失支持风格定制。此外,我们构建了OmniText-Bench基准数据集,包含输入图像、目标文本及掩码、风格参考,涵盖文本移除、缩放、重定位、多风格插入与编辑等应用。OmniText是首个能够完成多样化TIM任务的通用方法,在多项任务与指标上超越现有文本修复方法,媲美专用模型。
原文摘要 · Abstract (English)
Recent advancements in diffusion-based text synthesis have demonstrated significant performance in inserting and editing text within images via inpainting. However, despite the potential of text inpainting methods, three key limitations hinder their applicability to broader Text Image Manipulation (TIM) tasks: (i) the inability to remove text, (ii) the lack of control over the style of rendered text, and (iii) a tendency to generate duplicated letters. To address these challenges, we propose OmniText, a training-free generalist capable of performing a wide range of TIM tasks. Specifically, we investigate two key properties of cross- and self-attention mechanisms to enable text removal and to provide control over both text styles and content. Our findings reveal that text removal can be achieved by applying self-attention inversion, which mitigates the model's tendency to focus on surrounding text, thus reducing text hallucinations. Additionally, we redistribute cross-attention, as increasing the probability of certain text tokens reduces text hallucination. For controllable inpainting, we introduce novel loss functions in a latent optimization framework: a cross-attention content loss to improve text rendering accuracy and a self-attention style loss to facilitate style customization. Furthermore, we present OmniText-Bench, a benchmark dataset for evaluating diverse TIM tasks. It includes input images, target text with masks, and style references, covering diverse applications such as text removal, rescaling, repositioning, and insertion and editing with various styles. Our OmniText framework is the first generalist method capable of performing diverse TIM tasks. It achieves state-of-the-art performance across multiple tasks and metrics compared to other text inpainting methods and is comparable with specialist methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。