通过掩码自监督预训练，显著提升文本识别模型性能。
Masked Self-Supervised Pre-Training for Text Recognition Transformers on Large-Scale Datasets
- 逐步增加掩码比例并融合掩码与未掩码信息优化损失函数
- 在5000万无标注文本行上预训练，字符错误率降低最高达30%
- 无需额外标注数据，效果媲美迁移学习，适合资源受限场景
自监督学习已成为利用大规模无标注数据提升模型性能的有效方法。本文探索了针对文本识别变换器的掩码自监督预训练。具体提出两项预训练阶段改进：逐步增加掩码概率，以及修改损失函数以同时包含掩码和非掩码图像块的信息。我们在包含5000万条无标注文本行的数据集上进行预训练，并在四个不同规模的标注数据集上进行微调。实验结果表明，该自监督预训练方法显著优于传统迁移学习，在部分情况下字符错误率相对降低最高达30%。该方法在不依赖额外标注文本的前提下，达到与迁移学习相当的效果。
原文摘要 · Abstract (English)
Self-supervised learning has emerged as a powerful approach for leveraging large-scale unlabeled data to improve model performance in various domains. In this paper, we explore masked self-supervised pre-training for text recognition transformers. Specifically, we propose two modifications to the pre-training phase: progressively increasing the masking probability, and modifying the loss function to incorporate both masked and non-masked patches. We conduct extensive experiments using a dataset of 50M unlabeled text lines for pre-training and four differently sized annotated datasets for fine-tuning. Furthermore, we compare our pre-trained models against those trained with transfer learning, demonstrating the effectiveness of the self-supervised pre-training. In particular, pre-training consistently improves the character error rate of models, in some cases up to 30 % relatively. It is also on par with transfer learning but without relying on extra annotated text lines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。