用5次样本实现手写风格可控生成，提升手写识别效果
DiffusionPen: Towards Controlling the Style of Handwritten Text Generation
- 基于潜在扩散模型，融合度量学习与分类的混合风格提取器
- 在IAM数据集上生成效果优于现有方法，且提升手写识别性能
- 适合手写生成与识别研究者，尤其关注少样本风格迁移场景
手写文本生成（HTG）在给定文本和风格条件下极具挑战，源于用户间特征差异大及训练中未见的新词组合。尽管扩散模型在该任务上展现潜力，但仍处于探索阶段。本文提出DiffusionPen（DiffPen），一种基于潜在扩散模型的5次样本风格手写文本生成方法。通过结合度量学习与分类的混合风格提取器，该方法能有效捕捉已见与未见字词及风格的文本与书写特征，生成逼真手写样本。此外，我们探索了多风格混合与噪声嵌入等数据变异策略，增强了生成结果的鲁棒性与多样性。在IAM离线手写数据库上的大量实验表明，本方法在定性和定量上均优于现有方法，且生成的数据可进一步提升手写文本识别（HTR）系统性能。代码已公开于：https://github.com/koninik/DiffusionPen。
原文摘要 · Abstract (English)
Handwritten Text Generation (HTG) conditioned on text and style is a challenging task due to the variability of inter-user characteristics and the unlimited combinations of characters that form new words unseen during training. Diffusion Models have recently shown promising results in HTG but still remain under-explored. We present DiffusionPen (DiffPen), a 5-shot style handwritten text generation approach based on Latent Diffusion Models. By utilizing a hybrid style extractor that combines metric learning and classification, our approach manages to capture both textual and stylistic characteristics of seen and unseen words and styles, generating realistic handwritten samples. Moreover, we explore several variation strategies of the data with multi-style mixtures and noisy embeddings, enhancing the robustness and diversity of the generated data. Extensive experiments using IAM offline handwriting database show that our method outperforms existing methods qualitatively and quantitatively, and its additional generated data can improve the performance of Handwriting Text Recognition (HTR) systems. The code is available at: https://github.com/koninik/DiffusionPen.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。