用合成数据破解低资源语言OCR的数据瓶颈
synthocr-gen: A synthetic ocr dataset generator for low-resource languages- breaking the data barrier
- 将数字文本转为带复杂符号的合成训练数据
- 生成60万样本的克什米尔语OCR数据集
- 适合研究低资源语言与文档识别的学者
低资源语言的光学字符识别(OCR)因缺乏大规模标注数据而面临重大挑战。以约700万使用者、使用独特波斯-阿拉伯文字符且带特殊变音符号的克什米尔语为例,现有主流系统(如Tesseract、TrOCR、PaddleOCR)均不支持。人工构建数据集成本高、耗时长且易出错。我们提出SynthOCR-Gen——一个专为低资源语言设计的开源合成OCR数据生成工具。该工具将数字Unicode文本语料转化为可直接使用的训练数据,包含字符、词、n-gram、句子、行级别的分段,支持脚本纯净性校验的Unicode归一化,多字体渲染及25种以上模拟真实文档退化的增强技术(如旋转、模糊、噪声、扫描伪影)。我们成功生成了60万样本的克什米尔语词级分割数据集,并在HuggingFace上公开发布。该工作为低资源语言进入视觉-语言模型时代提供了可行路径,工具对全球研究者和实践者开放。
原文摘要 · Abstract (English)
Optical Character Recognition (OCR) for low-resource languages remains a significant challenge due to the scarcity of large-scale annotated training datasets. Languages such as Kashmiri, with approximately 7 million speakers and a complex Perso-Arabic script featuring unique diacritical marks, currently lack support in major OCR systems including Tesseract, TrOCR, and PaddleOCR. Manual dataset creation for such languages is prohibitively expensive, time-consuming, and error-prone, often requiring word by word transcription of printed or handwritten text. We present SynthOCR-Gen, an open-source synthetic OCR dataset generator specifically designed for low-resource languages. Our tool addresses the fundamental bottleneck in OCR development by transforming digital Unicode text corpora into ready-to-use training datasets. The system implements a comprehensive pipeline encompassing text segmentation (character, word, n-gram, sentence, and line levels), Unicode normalization with script purity enforcement, multi-font rendering with configurable distribution, and 25+ data augmentation techniques simulating real-world document degradations including rotation, blur, noise, and scanner artifacts. We demonstrate the efficacy of our approach by generating a 600,000-sample word-segmented Kashmiri OCR dataset, which we release publicly on HuggingFace. This work provides a practical pathway for bringing low-resource languages into the era of vision-language AI models, and the tool is openly available for researchers and practitioners working with underserved writing systems worldwide.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。