用Vision Transformer提升手写文本识别，小数据下表现更优。
HTR-VT: Handwritten Text Recognition with Vision Transformer
- 用CNN替代原始图像分块，提升特征提取效率
- 结合SAM优化器和跨度掩码，显著提升小样本性能
- 在19,830条数据的LAM数据集上创了新纪录
我们探索了视觉变换器（ViT）在手写文本识别中的应用。由于该领域标注数据有限，仅依赖ViT难以取得高精度。以往基于Transformer的模型需借助外部数据或大规模预训练才能表现优异。为此，我们提出一种数据高效的ViT方法，仅使用标准Transformer的编码器。通过用卷积神经网络（CNN）替代原始图像分块嵌入，并采用锐度感知最小化（SAM）优化器，使模型收敛至更平坦的极小值，带来显著性能提升。此外，我们引入跨度掩码技术，对特征图中连续区域进行遮蔽，起到有效正则化作用。实验证明，该方法在小型数据集IAM和READ2016上媲美传统CNN模型；并在目前最大的手写文本数据集LAM（含19,830条训练文本行）上建立了新基准。代码已公开于https://github.com/YutingLi0606/HTR-VT。
原文摘要 · Abstract (English)
We explore the application of Vision Transformer (ViT) for handwritten text recognition. The limited availability of labeled data in this domain poses challenges for achieving high performance solely relying on ViT. Previous transformer-based models required external data or extensive pre-training on large datasets to excel. To address this limitation, we introduce a data-efficient ViT method that uses only the encoder of the standard transformer. We find that incorporating a Convolutional Neural Network (CNN) for feature extraction instead of the original patch embedding and employ Sharpness-Aware Minimization (SAM) optimizer to ensure that the model can converge towards flatter minima and yield notable enhancements. Furthermore, our introduction of the span mask technique, which masks interconnected features in the feature map, acts as an effective regularizer. Empirically, our approach competes favorably with traditional CNN-based models on small datasets like IAM and READ2016. Additionally, it establishes a new benchmark on the LAM dataset, currently the largest dataset with 19,830 training text lines. The code is publicly available at: https://github.com/YutingLi0606/HTR-VT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。