无需OCR,用三阶段金字塔索引实现高效视觉文档检索。
Beyond Patch Aggregation: 3-Pass Pyramid Indexing for Vision-Enhanced Document Retrieval
- 三阶段金字塔索引融合页面摘要、标题和视觉热点生成轻量向量。
- 每页仅存17-27个向量,检索效率媲美传统方法,支持多模型兼容。
- 在金融文档数据集上准确率高达0.8051,适合大规模视觉增强检索场景。
以文档为中心的RAG流程通常从OCR开始,随后依赖脆弱的启发式规则进行分块、表格解析和版面重建。这类文本优先方法维护成本高,对版面微小变化敏感,常丢失关键空间线索。视觉优先检索成为有力替代方案,如ColPali和ColQwen直接处理页面图像,保留结构并简化流程,表现优异。但这些晚期交互模型绑定特定视觉主干网络,需存储每页数百个补丁嵌入,导致内存开销大,难以规模化部署。我们提出VisionRAG,一种无OCR、模型无关的多模态检索系统。它直接以图像形式索引文档,保留版面、表格与空间线索,并构建语义向量而不绑定具体提取方式。其三阶段金字塔索引框架利用全局页面摘要、章节标题、视觉热点及事实级线索生成向量,作为轻量检索代理。查询时,VisionRAG基于金字塔索引召回最相关页面,将原始页面图像以base64编码传给多模态LLM进行最终问答。检索中采用互惠排名融合策略整合金字塔信号,实现稳健排序。VisionRAG每页仅存储17至27个向量,效率接近补丁方法,同时兼容多种多模态编码器。在金融文档基准测试中,其FinanceBench@10准确率达0.8051,TAT DQA@100召回率达0.9629。结果表明,无OCR、摘要引导的多模态检索是传统文本抽取流程的实际且可扩展替代方案。
原文摘要 · Abstract (English)
Document centric RAG pipelines usually begin with OCR, followed by brittle heuristics for chunking, table parsing, and layout reconstruction. These text first workflows are costly to maintain, sensitive to small layout shifts, and often lose the spatial cues that contain the answer. Vision first retrieval has emerged as a strong alternative. By operating directly on page images, systems like ColPali and ColQwen preserve structure and reduce pipeline complexity while achieving strong benchmark performance. However, these late interaction models tie retrieval to a specific vision backbone and require storing hundreds of patch embeddings per page, creating high memory overhead and complicating large scale deployment. We introduce VisionRAG, a multimodal retrieval system that is OCR free and model agnostic. VisionRAG indexes documents directly as images, preserving layout, tables, and spatial cues, and builds semantic vectors without committing to a specific extraction. Our three pass pyramid indexing framework creates vectors using global page summaries, section headers, visual hotspots, and fact level cues. These summaries act as lightweight retrieval surrogates. At query time, VisionRAG retrieves the most relevant pages using the pyramid index, then forwards the raw page image encoded as base64 to a multimodal LLM for final question answering. During retrieval, reciprocal rank fusion integrates signals across the pyramid to produce robust ranking. VisionRAG stores only 17 to 27 vectors per page, matching the efficiency of patch based methods while staying flexible across multimodal encoders. On financial document benchmarks, it achieves 0.8051 accuracy at 10 on FinanceBench and 0.9629 recall at 100 on TAT DQA. These results show that OCR free, summary guided multimodal retrieval is a practical and scalable alternative to traditional text extraction pipelines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。