通过注入n-gram模型，让Transformer在识别手写文本时动态适应语言分布变化。
N-gram Injection into Transformers for Dynamic Language Model Adaptation in Handwritten Text Recognition
- 在解码器早期注入目标语料的n-gram，实现推理时语言建模动态调整。
- 在三个手写数据集上显著缩小了源域与目标域间的识别性能差距。
- 无需额外训练，仅靠n-gram推理即可提升跨语料泛化能力，适合实际部署。
基于Transformer的编码器-解码器网络在手写文本识别中取得了显著成果，部分得益于其自回归解码器隐式学习的语言模型。然而，当评估目标语料的语言分布与训练时所见源文本不同时，性能会大幅下降。为在语言分布偏移情况下保持识别准确率，我们提出一种外部n-gram注入（NGI）方法，实现在推理时对网络语言建模能力的动态适应。该方法允许切换至在接近目标分布的语料上估计的n-gram语言模型，从而缓解偏差，且无需在目标图像-文本对上进行额外训练。我们选择在Transformer解码器中早期注入n-gram，使网络能以极低的额外开销充分利用纯文本数据。在三个手写数据集上的实验表明，所提NGI显著缩小了源域与目标域之间的性能差距。
原文摘要 · Abstract (English)
Transformer-based encoder-decoder networks have recently achieved impressive results in handwritten text recognition, partly thanks to their auto-regressive decoder which implicitly learns a language model. However, such networks suffer from a large performance drop when evaluated on a target corpus whose language distribution is shifted from the source text seen during training. To retain recognition accuracy despite this language shift, we propose an external n-gram injection (NGI) for dynamic adaptation of the network's language modeling at inference time. Our method allows switching to an n-gram language model estimated on a corpus close to the target distribution, therefore mitigating bias without any extra training on target image-text pairs. We opt for an early injection of the n-gram into the transformer decoder so that the network learns to fully leverage text-only data at the low additional cost of n-gram inference. Experiments on three handwritten datasets demonstrate that the proposed NGI significantly reduces the performance gap between source and target corpora.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。