针对发票类文档视觉相似度高导致检索困难的问题,提出新基准与混合检索方法。
Invoice Haystack: Benchmarking Document Retrieval and Visual Question Answering Under Strong Visual Homogeneity

- 构建视觉高度同质的发票数据集,测试模型在真实企业文档场景下的检索能力
- 提出双流融合检索框架,在同质文档中实现60%的召回率,领先现有方法13.5个百分点
- 适合关注文档智能检索、多模态模型在真实场景表现的研究者与开发者
视觉语言模型在单文档视觉问答上已接近人类水平,但在处理大量视觉同质文档时性能显著下降。现有多文档基准混合多种文档类型,造成嵌入空间人为分离,无法反映企业文档库中数千份共享相同视觉模板的记录。本文识别出该现象为嵌入坍缩,并提出Invoice Haystack基准,包含1500张匿名发票图像和200个具有区分性的问答对,专用于测试强视觉同质性下的检索能力。该基准平均成对余弦相似度达0.73,远高于现有基准(DocHaystack为0.38,InfoHaystack为0.31),构成更严峻的检索挑战。为此,本文提出VL-RAG,一种联合利用文本与视觉嵌入的混合检索增强生成框架,并引入基于VLM的验证过滤器以精确识别文档。VL-RAG在Invoice Haystack-500上达到60.0% Recall@1,相比最先进方法提升最多13.5个百分点;同时在DocHaystack-1000(77.1% vs. 75.2%)和InfoHaystack-1000(84.5% vs. 80.0%)上也显著改善,证明双流融合策略在同质与异质文档中均具优势。
原文摘要 · Abstract (English)
Vision Language Models have achieved near-human performance on single-document Visual Question Answering, yet their effectiveness degrades significantly when retrieving information from large collections of visually homogeneous documents. Existing multi-document benchmarks aggregate diverse document types, creating artificial separation in embedding space that does not reflect enterprise document repositories where thousands of records share identical visual templates. We identify this as embedding collapse and introduce Invoice Haystack, a benchmark with 1,500 anonymized invoice images paired with 200 discriminative question-answer pairs, specifically designed to stress-test retrieval under strong visual homogeneity. Invoice Haystack exhibits a mean pairwise cosine similarity of 0.73, compared to 0.38 (DocHaystack) and 0.31 (InfoHaystack) in existing benchmarks, posing a fundamentally more challenging retrieval problem. Addressing the identified challenge, we propose VL-RAG, a hybrid retrieval-augmented generation framework that jointly leverages text and visual embeddings to harness the complementary strengths of both modalities, followed by a VLM-based verification filter for precise document identification. VL-RAG achieves 60.0\% Recall@1 on Invoice Haystack-500, outperforming existing state-of-the-art method by up to an absolute 13.5 percentage points. It further improves retrieval considerably on DocHaystack-1000 (77.1\% vs.\ 75.2\%) and InfoHaystack-1000 (84.5\% vs.\ 80.0\%), establishing the proposed dual-stream fusion as a consistently superior retrieval strategy across both homogeneous and heterogeneous document collections.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。