让生成图像中的文字既清晰又正确,解决文本生成难题。
TextPixs: Glyph-Conditioned Diffusion with Character-Aware Attention and OCR-Guided Supervision
- 用双流编码器融合语义与字形信息,生成字符感知表示。
- 设计字符注意力机制,降低错字率(字符错误率0.08)。
- 引入OCR反馈微调,提升可读性,适合广告设计等场景。
当前文本到图像的扩散模型虽能生成逼真多样的图像,但难以生成可读、准确且拼写正确的文字,限制了其在广告、教育和创意设计等实际应用中的使用。本文提出一种新框架GCDA,通过三个模块增强扩散模型:首先采用双流文本编码器,同时捕捉语义上下文与显式字形特征,生成富含自然特性的字符感知表示;其次引入字符感知注意力机制,并设计注意力分离损失,使每个字符的注意力分布独立,避免失真伪影;最后在闭环OCR微调阶段,使用完整文本感知损失直接优化模型的可读性与拼写准确性。大规模实验在MARIO-10M和T2I-CompBench数据集上验证,GCDA在所有指标上均达新最佳表现,文本渲染的字符错误率降至0.08(前最优0.21),词错误率为0.15(前最优0.25),人类感知更优,且高保真图像质量保持良好(FID: 14.3)。
原文摘要 · Abstract (English)
The modern text-to-image diffusion models boom has opened a new era in digital content production as it has proven the previously unseen ability to produce photorealistic and stylistically diverse imagery based on the semantics of natural-language descriptions. However, the consistent disadvantage of these models is that they cannot generate readable, meaningful, and correctly spelled text in generated images, which significantly limits the use of practical purposes like advertising, learning, and creative design. This paper introduces a new framework, namely Glyph-Conditioned Diffusion with Character-Aware Attention (GCDA), using which a typical diffusion backbone is extended by three well-designed modules. To begin with, the model has a dual-stream text encoder that encodes both semantic contextual information and explicit glyph representations, resulting in a character-aware representation of the input text that is rich in nature. Second, an attention mechanism that is aware of the character is proposed with a new attention segregation loss that aims to limit the attention distribution of each character independently in order to avoid distortion artifacts. Lastly, GCDA has an OCR-in-the-loop fine-tuning phase, where a full text perceptual loss, directly optimises models to be legible and accurately spell. Large scale experiments to benchmark datasets, such as MARIO-10M and T2I-CompBench, reveal that GCDA sets a new state-of-the-art on all metrics, with better character based metrics on text rendering (Character Error Rate: 0.08 vs 0.21 for the previous best; Word Error Rate: 0.15 vs 0.25), human perception, and comparable image synthesis quality on high-fidelity (FID: 14.3).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。