arXiv:2504.19724cs.CV2025-04被引 6

让AI模型精准复制多语言文字,无需理解语义

RepText: Rendering Visual Text via Replicating

  • 用非语言依赖的字形和位置信息实现文字渲染
  • 在多个字体和语言上优于开源方法,接近闭源模型
  • 适合需要自定义文字内容、字体和位置的设计师

尽管当代文本生成图像模型在生成视觉吸引人的图像方面取得了显著进展,但其在生成精确且灵活的排版元素(尤其是非拉丁字母)方面仍存在局限。为此,我们提出一个朴素假设:理解文本只是文本渲染的充分条件,而非必要条件。基于此,我们提出RepText,旨在使预训练的单语种文本到图像生成模型具备准确复制多语言视觉文字的能力,而无需真正理解这些文字。具体而言,我们借鉴ControlNet框架,并引入与语言无关的字形与位置信息,以实现协调一致的视觉文字生成,支持用户自定义文字内容、字体和位置。为提升准确性,我们采用文本感知损失结合扩散损失;为稳定生成过程,在推理阶段,我们直接使用噪声字形潜在表示初始化,而非随机初始化,并引入区域掩码限制特征注入仅作用于文字区域,避免背景失真。通过大量实验验证,RepText在性能上优于现有开源方法,达到与原生多语言闭源模型相当的水平。同时,我们也全面讨论了其局限性。

原文摘要 · Abstract (English)

Although contemporary text-to-image generation models have achieved remarkable breakthroughs in producing visually appealing images, their capacity to generate precise and flexible typographic elements, especially non-Latin alphabets, remains constrained. To address these limitations, we start from an naive assumption that text understanding is only a sufficient condition for text rendering, but not a necessary condition. Based on this, we present RepText, which aims to empower pre-trained monolingual text-to-image generation models with the ability to accurately render, or more precisely, replicate, multilingual visual text in user-specified fonts, without the need to really understand them. Specifically, we adopt the setting from ControlNet and additionally integrate language agnostic glyph and position of rendered text to enable generating harmonized visual text, allowing users to customize text content, font and position on their needs. To improve accuracy, a text perceptual loss is employed along with the diffusion loss. Furthermore, to stabilize rendering process, at the inference phase, we directly initialize with noisy glyph latent instead of random initialization, and adopt region masks to restrict the feature injection to only the text region to avoid distortion of the background. We conducted extensive experiments to verify the effectiveness of our RepText relative to existing works, our approach outperforms existing open-source methods and achieves comparable results to native multi-language closed-source models. To be more fair, we also exhaustively discuss its limitations in the end.

文本生成多语言字体复制控制生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。