用合成数据训练语言模型，大幅提升历史文本OCR错误纠正能力。
Scrambled text: training Language Models to correct OCR errors using synthetic data
- 通过字符级马尔可夫过程生成合成错别字数据
- 字符错误率降低55%，词错误率降低32%
- 适合历史文献数字化与自然语言处理研究者
光学字符识别（OCR）错误在数字化历史档案中普遍存在，严重影响其可用性与价值。生成式语言模型（LM）可通过上下文和文化背景信息纠正这些错误，这一过程称为上下文利用的OCR纠错（CLOCR-C）。然而，获取足够的微调数据颇具挑战。本文表明，使用语言模型与字符级马尔可夫污染过程生成合成数据进行微调，可显著提升纠错能力。在合成数据上训练的模型相比基础模型，字符错误率降低55%，词错误率降低32%，且优于在真实数据上训练的模型。关键发现包括：低污染数据比高污染数据更有效；非均匀字符污染优于均匀污染；在固定词元预算下，增加每条样本的词元数优于增加样本数量。本文提供8条有效训练准则、11,000篇19世纪报纸的合成数据集，以及scrambledtext Python库，用于生成合成污染数据。
原文摘要 · Abstract (English)
OCR errors are common in digitised historical archives significantly affecting their usability and value. Generative Language Models (LMs) have shown potential for correcting these errors using the context provided by the corrupted text and the broader socio-cultural context, a process called Context Leveraging OCR Correction (CLOCR-C). However, getting sufficient training data for fine-tuning such models can prove challenging. This paper shows that fine-tuning a language model on synthetic data using an LM and using a character level Markov corruption process can significantly improve the ability to correct OCR errors. Models trained on synthetic data reduce the character error rate by 55% and word error rate by 32% over the base LM and outperform models trained on real data. Key findings include; training on under-corrupted data is better than over-corrupted data; non-uniform character level corruption is better than uniform corruption; More tokens-per-observation outperforms more observations for a fixed token budget. The outputs for this paper are a set of 8 heuristics for training effective CLOCR-C models, a dataset of 11,000 synthetic 19th century newspaper articles and scrambledtext a python library for creating synthetic corrupted data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。