测试工业级OCR在真实文档上的检索生成效果,发现高准确率不等于好结果。
When Good OCR Is Not Enough: Benchmarking OCR Robustness for Retrieval-Augmented Generation

- 构建覆盖11类复杂文档的OCR评测基准,涵盖极端版式、水印背景等挑战场景。
- 实测显示:即使字符错误率低,结构语义错误仍导致检索失败率显著上升。
- 适合关注OCR与生成系统联动性能的工业应用研究者使用。
工业级检索增强生成(RAG)系统依赖光学字符识别(OCR)将视觉文档转为文本。现有OCR评测多基于字符级指标,难以反映真实场景下下游RAG的有效性。本文提出面向工业RAG的OCR评测基准,涵盖11类复杂文档类型,包括极端布局、高分辨率页面、复杂或带水印背景、历史文档(非标准阅读顺序)、装饰性文字以及含表格和数学公式的内容。在受控的OCR-first RAG流程中评估近期SOTA OCR模型,结果显示:尽管在常规基准上表现优异,但在真实工业文档上性能明显下降。我们发现,高OCR准确率并不保证下游RAG表现良好:结构与语义错误即使在低词错误率(WER)/字符错误率(CER)下仍会导致严重检索失败。进一步分析表明,这种误差不匹配具有类别依赖性,源于检索端与生成端双重失效,且在典型OCR-first流程中保持稳定。该基准已开源:https://github.com/Qihoo360/InduOCRBench。
原文摘要 · Abstract (English)
Industrial Retrieval-Augmented Generation (RAG) systems depend on optical character recognition (OCR) to transform visual documents into text. Existing OCR benchmarks rely on character-level metrics, which inadequately measure downstream RAG effectiveness under real-world conditions. We introduce an OCR benchmark for industrial RAG systems covering 11 challenging document types, including extreme layouts, high-resolution pages, complex or watermarked backgrounds, historical documents with non-standard reading orders, visually decorated text, and documents containing tables and mathematical formulas. Evaluating recent SOTA OCR models under a controlled OCR-first RAG pipeline shows clear performance degradation on realistic industrial documents despite strong conventional benchmark scores. We find that high OCR accuracy does not necessarily translate into strong downstream RAG performance: structural and semantic errors can cause substantial retrieval failures even when WER/CER remains low. Further analysis shows that this mismatch is category-dependent, arises through both retrieval-side and downstream generation-side failures, and remains stable across representative OCR-first pipeline choices. The benchmark is publicly available at https://github.com/Qihoo360/InduOCRBench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。