针对多图文档问答，提出智能压缩与选择视觉信息的新方法。
Doc-REFRAG: Rethinking Multimodal Document Retrieval-Augmented Generation
- 用轻量强化学习筛选关键图像，动态压缩无关视觉内容
- 在6个基准上超越11个基线模型，推理延迟降低显著
- 适用于真实多图文档问答，提升准确率与效率
真实世界知识存在于多模态文档中，需要检索增强生成（RAG）以实现精准问答。然而，现有多模态RAG模型主要面向单图或封闭文档场景，在真实多图情境下表现有限，且处理大量检索图像带来大量无关视觉标记的计算开销。为此，我们构建了包含34.3万组问答对的大型数据集DocLongRAG，每条数据平均关联37.4张检索图像，以反映真实的RAG流程。基于此数据集，我们提出Doc-REFRAG，一种问题引导的框架：将视觉标记压缩为粗粒度块，并通过轻量级强化学习选择器有选择地扩展与问题相关的内容。在六个基准上的实验表明，Doc-REFRAG优于十一个强基线模型，在保持更高准确率的同时，显著降低推理延迟。
原文摘要 · Abstract (English)
Real-world knowledge resides in multimodal documents, necessitating retrieval-augmented generation (RAG) for accurate question answering. However, existing multimodal RAG models are primarily designed for single-image or closed-document settings and exhibit limited accuracy in realistic multi-image scenarios. Moreover, processing numerous retrieved images incurs substantial computational overhead from irrelevant visual tokens. To address these challenges, we introduce DocLongRAG, a large-scale dataset of 343K question--answer pairs, each associated with an average of 37.4 retrieved images to reflect authentic RAG workflows. Building on this dataset, we propose Doc-REFRAG, a question-guided framework that compresses visual tokens into coarse chunks and selectively expands question-relevant ones via a lightweight RL-based selector. Experiments on six benchmarks show that Doc-REFRAG outperforms eleven strong baselines, achieving state-of-the-art accuracy with significantly lower inference latency. Our resources are available at https://github.com/Collab-Gen/Doc-REFRAG.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。