首个零样本自回归手写文本生成模型,可复现未见字体与笔迹。
Zero-Shot Styled Text Image Generation, but Make It Autoregressive
- 用变分自编码器+自回归Transformer,实现风格条件生成。
- 仅用10万+种合成字体训练,零样本复现新字体和手写风格。
- 生成无背景杂讯,适合下游应用,支持任意长度文本。
风格化手写文本生成(HTG)近年来受到计算机视觉与文档分析领域的关注,已有基于GAN或扩散模型的方法取得良好效果。但这些方法难以泛化至新风格,且在最大输出长度和训练效率方面存在技术瓶颈。为此,本文提出名为Emuru的新框架,结合强大的文本图像表示模型(变分自编码器)与自回归Transformer。该方法可基于文本内容和风格示例(如特定字体或手写风格)生成带风格的文本图像。模型仅在包含超10万种印刷体与书法体的英文合成数据集上训练,具备零样本复现未见字体及用户手写风格的能力。据我们所知,Emuru是首个用于HTG的自回归模型,也是首个专为新风格泛化设计的模型。此外,生成图像无背景干扰,更适用于下游任务。在印刷体与手写体、任意长度文本生成场景下的大量评估验证了方法的有效性。
原文摘要 · Abstract (English)
Styled Handwritten Text Generation (HTG) has recently received attention from the computer vision and document analysis communities, which have developed several solutions, either GAN- or diffusion-based, that achieved promising results. Nonetheless, these strategies fail to generalize to novel styles and have technical constraints, particularly in terms of maximum output length and training efficiency. To overcome these limitations, in this work, we propose a novel framework for text image generation, dubbed Emuru. Our approach leverages a powerful text image representation model (a variational autoencoder) combined with an autoregressive Transformer. Our approach enables the generation of styled text images conditioned on textual content and style examples, such as specific fonts or handwriting styles. We train our model solely on a diverse, synthetic dataset of English text rendered in over 100,000 typewritten and calligraphy fonts, which gives it the capability to reproduce unseen styles (both fonts and users' handwriting) in zero-shot. To the best of our knowledge, Emuru is the first autoregressive model for HTG, and the first designed specifically for generalization to novel styles. Moreover, our model generates images without background artifacts, which are easier to use for downstream applications. Extensive evaluation on both typewritten and handwritten, any-length text image generation scenarios demonstrates the effectiveness of our approach.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。