用强化学习先推理再提取证据，提升大模型生成质量
Learning to Extract Rational Evidence via Reinforcement Learning for Retrieval-Augmented Generation
- 先推理后提取，避免盲目筛选关键信息
- 在5个数据集上显著提升生成准确率，证据更紧凑高质量
- 适合需要精准生成的问答与智能体系统
检索增强生成（RAG）能有效提升大语言模型（LLM）的准确性。然而，检索噪声会严重降低LLM生成质量，亟需去噪机制。以往方法直接提取证据，缺乏深度思考，可能误删关键线索且泛化能力差。为此，我们提出EviOmni，通过先推理后提取的方式学习理性证据。EviOmni将证据推理与提取整合为统一轨迹，采用知识标记掩码防止信息泄露，并通过可验证奖励（答案、长度、格式）进行在线策略强化学习优化。在五个基准数据集上的大量实验表明，EviOmni能提供紧凑且高质量的证据，提升下游任务准确率，并兼容传统与智能体式RAG系统。
原文摘要 · Abstract (English)
Retrieval-Augmented Generation (RAG) effectively improves the accuracy of Large Language Models (LLMs). However, retrieval noises significantly undermine the quality of LLMs' generation, necessitating the development of denoising mechanisms. Previous works extract evidence straightforwardly without deep thinking, which may risk filtering out key clues and struggle with generalization. To this end, we propose EviOmni, which learns to extract rational evidence via reasoning first and then extracting. Specifically, EviOmni integrates evidence reasoning and evidence extraction into one unified trajectory, followed by knowledge token masking to avoid information leakage, optimized via on-policy reinforcement learning with verifiable rewards in terms of answer, length, and format. Extensive experiments on five benchmark datasets show the superiority of EviOmni, which provides compact and high-quality evidence, enhances the accuracy of downstream tasks, and supports both traditional and agentic RAG systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。