arXiv:2607.24799cs.IRcs.AI2026-07

用开源模型提升科学文档问答准确率,减少幻觉。

Multimodal Hybrid Retrieval-Augmented Generation for Scientific Document Understanding using Open-Source SLMs

论文配图:Multimodal Hybrid Retrieval-Augmented Generation for Scientific Document Understanding using Open-Source SLMs
图 1 · 摘自论文原文
  • 融合视觉语言模型生成图表文本摘要,实现多模态信息摄入
  • 混合稀疏与稠密检索,召回率提升157%,延迟仅增50毫秒
  • 适合科研人员在本地部署高精度文档理解系统

大语言模型在未微调的情况下回答科学文档问题时容易产生幻觉。现有检索增强生成方法虽部分缓解此问题,但仍面临上下文知识有限、稀疏与稠密检索差异及检索噪声等挑战。本文提出一种先进的多模态检索增强生成系统,通过开源视觉语言模型Qwen2-VL-2B-Instruct生成表格与图表的文本摘要,构建多模态摄入流程。检索阶段结合基于HNSW的语义搜索与GIN词法搜索,通过倒数排名融合统一,并用交叉编码器重排序以降低噪声。为保障多轮对话连贯性,引入查询压缩模块。评估采用MMLongBench基准、BeIR格式合成数据集及DeepEval框架,结果表明相比基线Naive-RAG,检索质量提升157%,额外延迟仅50毫秒;Qwen2-VL-2B-Instruct在BERTScore上表现接近云端模型。研究验证了开源小型语言模型配合先进检索策略,可在不依赖云服务的前提下实现具有竞争力的文档理解性能。

原文摘要 · Abstract (English)

Large Language Models tend to hallucinate when answering domain-specific ques tions from scientific documents without prior fine-tuning. Currently, methods such as Retrieval-Augmented Generation partially solve this problem but face different challenges: limited context knowledge, difference between sparse and dense retrieval, and retrieval noise. This paper presents an Advanced Multimodal Retrieval-Augmented Generation system that aims to solve those challenges and im prove the accuracy of information extraction. The proposed architecture introduces a multimodal ingestion pipeline that leverages an open-source Vision-Language Model (Qwen2-VL-2B-Instruct) to generate textual summaries of tables and fig ures. The retrieval phase integrates HNSW-based semantic search with GIN-based lexical search, unified through Reciprocal Rank Fusion and refined using Cross Encoder reranking to minimize retrieval noise. To ensure conversational coherence across multi-turn interactions, a Query Condenser module is employed. Evaluation is conducted by independently assessing the ingestion, retrieval and generation stages using the MMLongBench benchmark, a BeIR-format synthetic dataset and the DeepEval framework. Moreover, results demonstrate a 157% improvement in retrieval quality over a Naive-RAG baseline, with only 50 ms additional la tency, while Qwen2-VL-2B-Instruct achieved results comparable to cloud-based models in BERTScore. These findings validate that open-source optimized SLMs, paired with advanced retrieval strategies, can provide competitive performance for document understanding without relying on cloud-based models.

多模态检索增强科学文献开源模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。