OCR错误会严重拖累RAG性能，该研究首次系统评估其影响并发布基准数据集。
OCR Hinders RAG: Evaluating the Cascading Impact of OCR on Retrieval-Augmented Generation
- 构建OHRBench基准，涵盖8561张真实文档图像和8498个问答对
- 发现语义噪声与格式噪声均随程度上升导致RAG准确率下降
- 验证现有OCR方案均无法满足RAG高质量知识库需求
检索增强生成（RAG）通过引入外部知识来减少大语言模型的幻觉并融入更新信息，无需重新训练。作为RAG的关键环节，外部知识库通常通过光学字符识别（OCR）从非结构化PDF文档中提取结构化数据构建。然而，由于OCR预测不完美以及结构化数据固有的非均匀表示，知识库不可避免地包含多种OCR噪声。本文提出OHRBench，首个用于理解OCR对RAG系统级联影响的基准测试。OHRBench包含来自七个真实应用场景的8,561张非结构化文档图像，以及基于文档多模态元素生成的8,498个问答对，挑战当前用于RAG的OCR解决方案。为深入分析OCR影响，我们识别出两类主要噪声：语义噪声与格式噪声，并通过扰动生成具有不同噪声程度的结构化数据集。利用OHRBench，我们全面评估了现有OCR方案，发现无一能胜任构建高质量RAG知识库的任务。进一步系统评估表明，噪声程度与RAG性能呈显著负相关。OHRBench包含原始PDF、问答对及真实结构化数据，已开源：https://github.com/opendatalab/OHR-Bench
原文摘要 · Abstract (English)
Retrieval-augmented Generation (RAG) enhances Large Language Models (LLMs) by integrating external knowledge to reduce hallucinations and incorporate up-to-date information without retraining. As an essential part of RAG, external knowledge bases are commonly built by extracting structured data from unstructured PDF documents using Optical Character Recognition (OCR). However, given the imperfect prediction of OCR and the inherent non-uniform representation of structured data, knowledge bases inevitably contain various OCR noises. In this paper, we introduce OHRBench, the first benchmark for understanding the cascading impact of OCR on RAG systems. OHRBench includes 8,561 carefully selected unstructured document images from seven real-world RAG application domains, along with 8,498 Q&A pairs derived from multimodal elements in documents, challenging existing OCR solutions used for RAG. To better understand OCR's impact on RAG systems, we identify two primary types of OCR noise: Semantic Noise and Formatting Noise and apply perturbation to generate a set of structured data with varying degrees of each OCR noise. Using OHRBench, we first conduct a comprehensive evaluation of current OCR solutions and reveal that none is competent for constructing high-quality knowledge bases for RAG systems. We then systematically evaluate the impact of these two noise types and demonstrate the trend relationship between the degree of OCR noise and RAG performance. Our OHRBench, including PDF documents, Q&As, and the ground truth structured data are released at: https://github.com/opendatalab/OHR-Bench
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。