用半监督方法让扩散模型学会新笔迹，生成更逼真的手写文本图像。
Semi-Supervised Adaptation of Diffusion Models for Handwritten Text Generation
- 通过掩码自编码器学习未见笔迹风格，实现风格条件生成
- 在IAM和RIMES数据集上生成图像质量显著提升
- 适合需要生成新字体训练数据的研究者
手写文本生成（HTG）是生成逼真可读手写文本图像的挑战性任务。给定一段文字和某位作者的样本，目标是合成一张展现正确拼写、且具有目标作者书法风格的图像。该技术常用于为下游模型生成训练数据以适应新数据集。当前扩散模型（DMs）已成为HTG的主流方法。本文提出一种扩展的潜在扩散模型，通过掩码自编码器学习未见笔迹风格的条件表示，使模型能生成训练中未出现过的书写风格。所提出的内容编码器支持文本与书法特征的多种条件输入。同时采用无分类器引导，并研究其对生成图像质量的影响。针对新未标注数据集的适配，提出半监督训练方案。在IAM数据集上评估性能，并使用RIMES数据集验证对未见数据生成能力，结果表明该方法在该应用场景下表现优异。
原文摘要 · Abstract (English)
The generation of images of realistic looking, readable handwritten text is a challenging task which is referred to as handwritten text generation (HTG). Given a string and examples from a writer, the goal is to synthesize an image depicting the correctly spelled word in handwriting with the calligraphic style of the desired writer. An important application of HTG is the generation of training images in order to adapt downstream models for new data sets. With their success in natural image generation, diffusion models (DMs) have become the state-of-the-art approach in HTG. In this work, we present an extension of a latent DM for HTG to enable generation of writing styles not seen during training by learning style conditioning with a masked auto encoder. Our proposed content encoder allows for different ways of conditioning the DM on textual and calligraphic features. Additionally, we employ classifier-free guidance and explore the influence on the quality of the generated training images. For adapting the model to a new unlabeled data set, we propose a semi-supervised training scheme. We evaluate our approach on the IAM-database and use the RIMES-database to examine the generation of data not seen during training achieving improvements in this particularly promising application of DMs for HTG.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。