提出一种无需训练的采样器,显著提升图像中文本生成的准确性。
AMO Sampler: Enhancing Text Rendering with Overshooting
- 通过交替过模拟微分方程与重引入噪声,实现动态误差校正。
- 在SD3和Flux上分别提升32.3%和35.9%的文本渲染准确率。
- 根据注意力得分自适应调节修正强度,适合对文本精度要求高的场景。
在文生图任务中,精准对齐文本指令与生成图像中的文字内容仍是重大挑战,当前主流模型如Stable Diffusion 3(SD3)、Flux和AuraFlow仍存在拼写错误或文字不一致问题。本文提出一种无需训练、计算开销极小的方法,通过为预训练修正流(Rectified Flow)模型设计过冲采样器,交替进行过模拟学习的常微分方程(ODE)并重引入噪声。相比欧拉采样器,该方法有效引入额外的朗之万动力学项,可纠正连续欧拉步累积的误差,从而提升文本渲染质量。然而过高过冲强度会导致图像过度平滑。为此,我们提出注意力调制过冲采样器(AMO),依据图像块与文本内容的注意力分数自适应调节过冲强度。AMO在SD3和Flux上分别实现32.3%和35.9%的文本渲染准确率提升,且不损害整体图像质量或增加推理成本。代码已公开:https://github.com/hxixixh/amo-release。
原文摘要 · Abstract (English)
Achieving precise alignment between textual instructions and generated images in text-to-image generation is a significant challenge, particularly in rendering written text within images. Sate-of-the-art models like Stable Diffusion 3 (SD3), Flux, and AuraFlow still struggle with accurate text depiction, resulting in misspelled or inconsistent text. We introduce a training-free method with minimal computational overhead that significantly enhances text rendering quality. Specifically, we introduce an overshooting sampler for pretrained rectified flow (RF) models, by alternating between over-simulating the learned ordinary differential equation (ODE) and reintroducing noise. Compared to the Euler sampler, the overshooting sampler effectively introduces an extra Langevin dynamics term that can help correct the compounding error from successive Euler steps and therefore improve the text rendering. However, when the overshooting strength is high, we observe over-smoothing artifacts on the generated images. To address this issue, we propose an Attention Modulated Overshooting sampler (AMO), which adaptively controls the strength of overshooting for each image patch according to their attention score with the text content. AMO demonstrates a 32.3% and 35.9% improvement in text rendering accuracy on SD3 and Flux without compromising overall image quality or increasing inference cost. Code available at: https://github.com/hxixixh/amo-release.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。