通过问答增强文本图像识别，提升模型对字符细节的理解能力。
An Effective Data Augmentation Method by Asking Questions about Scene Text Images
- 用自然语言提问图像中字符的属性，生成辅助训练任务
- 在WordArt和Esposalles数据集上显著降低CER与WER
- 适合需要精细文本理解的OCR场景，如复杂场景文字识别
场景文本识别（STR）和手写文本识别（HTR）在将图像中文本内容准确转录为机器可读格式时面临挑战。传统OCR模型直接预测文本，限制了对文本结构的深入推理。我们提出一种受视觉问答（VQA）启发的数据增强框架，通过结构化的问题-答案任务强化OCR训练。针对每张图像-文本对，生成询问字符级别属性（如存在性、位置、频次）的自然语言问题，答案来自真实文本。这些辅助任务促使模型进行更细粒度的推理，使OCR模型能够联合理解图像与问题中的视觉特征。在WordArt和Esposalles数据集上的实验表明，该方法持续优于基线模型，显著降低字符错误率（CER）和词错误率（WER）。代码已开源：https://github.com/xuyaooo/DataAugOCR。
原文摘要 · Abstract (English)
Scene text recognition (STR) and handwritten text recognition (HTR) face significant challenges in accurately transcribing textual content from images into machine-readable formats. Conventional OCR models often predict transcriptions directly, which limits detailed reasoning about text structure. We propose a VQA-inspired data augmentation framework that strengthens OCR training through structured question-answering tasks. For each image-text pair, we generate natural-language questions probing character-level attributes such as presence, position, and frequency, with answers derived from ground-truth text. These auxiliary tasks encourage finer-grained reasoning, and the OCR model aligns visual features with textual queries to jointly reason over images and questions. Experiments on WordArt and Esposalles datasets show consistent improvements over baseline models, with significant reductions in both CER and WER. Our code is publicly available at https://github.com/xuyaooo/DataAugOCR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。