arXiv:2602.17687cs.IRcs.AI2026-02被引 2

构建科学文档图文检索基准,对比图像与文本方法优劣。

IRPAPERS: A Visual Document Benchmark for Scientific Retrieval and Question Answering

  • 设计图文双版本科学论文页面数据集,支持多模态对比实验。
  • 图文混合检索优于单一模态,最高达95% Recall@20,互补性显著。
  • 闭源模型表现更优,且深度检索提升问答精度,适合信息检索研究者。

AI在文本和关系数据处理上成果显著,但视觉文档处理仍较薄弱。传统系统依赖OCR转换图像为文本,而多模态大模型可直接从图像中检索生成。我们提出IRPAPERS,包含166篇科学论文共3,230页,每页提供图像与OCR文本。通过180个‘大海捞针’式问题,对比图像与文本检索及问答系统。文本检索使用Arctic 2.0嵌入、BM25和混合搜索,分别达到46%、78%、91% Recall@1、@5、@20;图像检索分别为43%、78%、93%。两者失败模式互补,融合后性能提升,达49%、81%、95%。评估了MUVERA效率-性能权衡及多向量图像嵌入模型。闭源模型Cohere Embed v4图像嵌入表现最佳,达58%、87%、97%。问答任务中,文本RAG对齐率(0.82)高于图像系统(0.71),且多文档检索优于单文档最优情况。分析发现两种模态各有短板,特定问题类型需依赖某一模态。数据集与代码开源。

原文摘要 · Abstract (English)

AI systems have achieved remarkable success in processing text and relational data, yet visual document processing remains relatively underexplored. Whereas traditional systems require OCR transcriptions to convert these visual documents into text and metadata, recent advances in multimodal foundation models offer retrieval and generation directly from document images. This raises a key question: How do image-based systems compare to established text-based methods? We introduce IRPAPERS, a benchmark of 3,230 pages from 166 scientific papers, with both an image and an OCR transcription for each page. Using 180 needle-in-the-haystack questions, we compare image- and text-based retrieval and question answering systems. Text retrieval using Arctic 2.0 embeddings, BM25, and hybrid text search achieved 46% Recall@1, 78% Recall@5, and 91% Recall@20, while image-based retrieval reaches 43%, 78%, and 93%, respectively. The two modalities exhibit complementary failures, enabling multimodal hybrid search to outperform either alone, achieving 49% Recall@1, 81% Recall@5, and 95% Recall@20. We further evaluate efficiency-performance tradeoffs with MUVERA and assess multiple multi-vector image embedding models. Among closed-source models, Cohere Embed v4 page image embeddings outperform Voyage 3 Large text embeddings and all tested open-source models, achieving 58% Recall@1, 87% Recall@5, and 97% Recall@20. For question answering, text-based RAG systems achieved higher ground-truth alignment than image-based systems (0.82 vs. 0.71), and both benefit substantially from increased retrieval depth, with multi-document retrieval outperforming oracle single-document retrieval. We analyze the complementary limitations of unimodal text and image representations and identify question types that require one modality over the other. The IRPAPERS dataset and all experimental code are publicly available.

图文检索科学文献多模态基准测试

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。