无需训练即可精准生成多行文字,解决中文等复杂文本渲染难题。
FreeText: Training-Free Text Rendering in Diffusion Transformers via Attention Localization and Spectral Glyph Injection
- 通过注意力定位确定文字书写区域,利用锚点与拓扑优化生成高置信度掩码。
- 引入频域调制的字形注入,强化字形结构并抑制语义泄露,提升可读性。
- 适合作为插件集成到现有扩散模型,特别适合需要灵活排版的场景。
大规模文本到图像(T2I)扩散模型在开放域生成中表现优异,但在多行布局、密集排版及长尾语言(如中文)的文字渲染上仍存在困难。现有方法通常需昂贵的再训练或刚性外部布局约束,影响美观且灵活性差。本文提出 extbf{FreeText},一种无需训练、即插即用的框架,通过挖掘 extit{Diffusion Transformer (DiT)} 模型内在机制改进文字渲染。该方法将问题分解为 extit{何处书写} 与 extit{写什么}。针对 extit{何处书写},通过解析图像-文本注意力中的词级空间归因,以类汇点(sink-like)token 作为稳定空间锚点,并结合拓扑感知优化生成高置信度掩码。针对 extit{写什么},提出频谱调制字形注入(SGMI),在噪声对齐的字形先验中引入频域带通调制,增强字形结构并抑制语义泄露(避免渲染概念而非文字)。在 Qwen-Image、FLUX.1-dev、SD3 变体上,于 longText-Benchmark、CVTG 及自建 CLT-Bench 上的实验表明,该方法在显著提升文字可读性的同时,基本保持语义一致性与美学质量,仅带来轻微推理开销。
原文摘要 · Abstract (English)
Large-scale text-to-image (T2I) diffusion models excel at open-domain synthesis but still struggle with precise text rendering, especially for multi-line layouts, dense typography, and long-tailed scripts such as Chinese. Prior solutions typically require costly retraining or rigid external layout constraints, which can degrade aesthetics and limit flexibility. We propose \textbf{FreeText}, a training-free, plug-and-play framework that improves text rendering by exploiting intrinsic mechanisms of \emph{Diffusion Transformer (DiT)} models. \textbf{FreeText} decomposes the problem into \emph{where to write} and \emph{what to write}. For \emph{where to write}, we localize writing regions by reading token-wise spatial attribution from endogenous image-to-text attention, using sink-like tokens as stable spatial anchors and topology-aware refinement to produce high-confidence masks. For \emph{what to write}, we introduce Spectral-Modulated Glyph Injection (SGMI), which injects a noise-aligned glyph prior with frequency-domain band-pass modulation to strengthen glyph structure and suppress semantic leakage (rendering the concept instead of the word). Extensive experiments on Qwen-Image, FLUX.1-dev, and SD3 variants across longText-Benchmark, CVTG, and our CLT-Bench show consistent gains in text readability while largely preserving semantic alignment and aesthetic quality, with modest inference overhead.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。