arXiv:2410.10168cs.CV2024-10中稿 · ECAI2024被引 19

先生成背景再融合文字,实现更自然的图文合成。

First Creating Backgrounds Then Rendering Texts: A New Paradigm for Visual Text Blending

  • 分两步:先生成无文字背景图,再精准渲染文字。
  • 新模型GlyphOnly在小规模文本下仍保持高保真与一致性。
  • 适合需要高质量图文合成的应用,如文本检测数据集生成。

扩散模型在图像生成方面表现优异,推动了视觉文本生成的发展。然而,现有方法多聚焦于仅凭文本提示生成完整图像,控制精度差且实用性有限。更具前景的方向是视觉文本融合,即在无文字背景上无缝融合文本。但现有方法受限于合成用背景不足及泛化能力弱,难以生成高保真、多样化的图像。为此,我们提出一种新范式:先生成背景,再渲染文字。具体地,设计了一个背景生成器以产出高保真、无文字的自然图像;同时提出名为GlyphOnly的文本渲染器,基于Stable Diffusion框架,利用字形(glyphs)和背景作为条件,实现精确渲染与一致性控制,并引入自适应文本块探索策略,提升小规模文本的渲染效果。我们还探索了多种下游应用,包括提升场景文本检测器性能的场景文本数据集合成,以及文本图像的定制与编辑。代码与模型将开源于https://github.com/Zhenhang-Li/GlyphOnly。

原文摘要 · Abstract (English)

Diffusion models, known for their impressive image generation abilities, have played a pivotal role in the rise of visual text generation. Nevertheless, existing visual text generation methods often focus on generating entire images with text prompts, leading to imprecise control and limited practicality. A more promising direction is visual text blending, which focuses on seamlessly merging texts onto text-free backgrounds. However, existing visual text blending methods often struggle to generate high-fidelity and diverse images due to a shortage of backgrounds for synthesis and limited generalization capabilities. To overcome these challenges, we propose a new visual text blending paradigm including both creating backgrounds and rendering texts. Specifically, a background generator is developed to produce high-fidelity and text-free natural images. Moreover, a text renderer named GlyphOnly is designed for achieving visually plausible text-background integration. GlyphOnly, built on a Stable Diffusion framework, utilizes glyphs and backgrounds as conditions for accurate rendering and consistency control, as well as equipped with an adaptive text block exploration strategy for small-scale text rendering. We also explore several downstream applications based on our method, including scene text dataset synthesis for boosting scene text detectors, as well as text image customization and editing. Code and model will be available at \url{https://github.com/Zhenhang-Li/GlyphOnly}.

图文融合扩散模型文本渲染

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。