用开源大模型提升检索增强推理的逻辑能力，对抗误导信息
Open-RAG: Enhanced Retrieval-Augmented Reasoning with Open-Source Large Language Models
- 将通用大模型转为参数高效稀疏专家混合模型，支持复杂推理
- 在多跳问答和含干扰项任务中表现超越ChatGPT等主流模型
- 适合需要高精度、可解释推理的开源模型应用者
检索增强生成（RAG）能提升大语言模型的事实准确性，但现有方法在使用开源大模型时往往推理能力有限。为此，我们提出Open-RAG框架，将任意密集型大模型转化为参数高效的稀疏专家混合（MoE）模型，以应对单跳与多跳查询等复杂推理任务。该框架特别训练模型识别看似相关实则误导的干扰信息，通过隐式学习动态选择有效专家并整合外部知识，提升响应准确性和上下文相关性。此外，我们设计了一种混合自适应检索方法，平衡性能提升与推理速度。实验表明，基于Llama2-7B的Open-RAG在多种知识密集型任务中优于ChatGPT、Self-RAG和Command R+等先进模型。代码与模型已开源：https://openragmoe.github.io/
原文摘要 · Abstract (English)
Retrieval-Augmented Generation (RAG) has been shown to enhance the factual accuracy of Large Language Models (LLMs), but existing methods often suffer from limited reasoning capabilities in effectively using the retrieved evidence, particularly when using open-source LLMs. To mitigate this gap, we introduce a novel framework, Open-RAG, designed to enhance reasoning capabilities in RAG with open-source LLMs. Our framework transforms an arbitrary dense LLM into a parameter-efficient sparse mixture of experts (MoE) model capable of handling complex reasoning tasks, including both single- and multi-hop queries. Open-RAG uniquely trains the model to navigate challenging distractors that appear relevant but are misleading. As a result, Open-RAG leverages latent learning, dynamically selecting relevant experts and integrating external knowledge effectively for more accurate and contextually relevant responses. In addition, we propose a hybrid adaptive retrieval method to determine retrieval necessity and balance the trade-off between performance gain and inference speed. Experimental results show that the Llama2-7B-based Open-RAG outperforms state-of-the-art LLMs and RAG models such as ChatGPT, Self-RAG, and Command R+ in various knowledge-intensive tasks. We open-source our code and models at https://openragmoe.github.io/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。