让扩散模型生成文字时既清晰又精准，还能自适应新字体。
FontFusion: Enhancing Generative Text in Diffusion Models with Typographic Conditioning
- 用分层令牌建模文字与字体关系，实现多粒度控制。
- 在装饰字体上比单编码器提升76%表现，字体一致性提高68%-76%。
- 无需重训练即可接入现有模型，适合图文生成与设计自动化场景。
扩散模型中的字体生成长期面临难题：精确控制字体会降低文字可读性，保持可读性则牺牲字体还原度。本文提出FontFusion，一种即插即用的条件框架，用于扩散Transformer（DiT）架构，通过三项核心创新解决该矛盾：(1) 分层令牌表示，在多粒度上建立显式文字-字体关联；(2) 位置感知嵌入，将字体特征与图像内容空间对齐；(3) 多层级令牌丢弃策略，提升计算效率并增强对未见字体的泛化能力。系统评估表明，结合DeepFont与DINOv2的双编码器在字体嵌入空间中表现最优。FontFusion在复杂装饰字体上相较单编码器基线实现76%相对提升，字体一致性超过无条件模型的68%-76%，且可无缝集成至现有DiT架构，无需重新训练。
原文摘要 · Abstract (English)
Typography generation in diffusion models faces a persistent trade-off: enabling precise font control typically degrades text legibility, while maintaining readability often sacrifices typographic fidelity. We present FontFusion, a plug-and-play conditioning framework for Diffusion Transformer (DiT) architectures that resolves this dilemma through three core innovations: (1) a hierarchical token representation establishing explicit text-font relationships at multiple granularities, (2) position-aware embeddings creating spatial bindings between typography and image content, and (3) a multi-level token dropping strategy improving both computational efficiency and generalization to unseen fonts. Our systematic evaluation of font embedding spaces reveals that a dual encoder combining DeepFont and DINOv2 outperforms any single encoder for typography tasks. FontFusion demonstrates 76% relative improvement on challenging decorative fonts over single-encoder baselines and font consistency gains exceeding approximately 68-76% over unconditioned models, while integrating into existing DiT architectures without retraining.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。