用引用提示的评分机制,让AI搜索更全面、真实、有逻辑链。
Chaining the Evidence: Robust Reinforcement Learning for Deep Search Agents with Citation-Aware Rubric Rewards
- 拆解问题为可验证小目标,要求带引用证据链。
- 在多个基准上超越传统强化学习方法,减少胡编乱造。
- 适合需要严谨推理和文献支持的研究型AI任务。
强化学习已成为提升基于大模型的深度搜索代理的关键技术。然而,现有方法主要依赖二元结果奖励,无法捕捉代理推理过程的全面性与事实性,常导致捷径利用和幻觉等问题。为此,我们提出一种细粒度奖励框架——引用感知评分机制(Citation-aware Rubric Rewards, CaRR),强调推理的全面性、事实依据和证据连贯性。CaRR将复杂问题分解为可验证的单跳评分项,要求代理通过明确识别隐藏实体、提供正确引用并构建完整证据链来满足这些评分项。我们进一步引入引用感知组相对策略优化(C-GRPO),结合CaRR与结果奖励训练鲁棒的深度搜索代理。实验表明,C-GRPO在多个深度搜索基准上均优于标准结果驱动的强化学习基线。分析还验证了其能有效抑制捷径利用,促进全面且基于证据的推理,并在开放式深度研究任务中展现出强泛化能力。代码与数据已公开于 https://github.com/THUDM/CaRR。
原文摘要 · Abstract (English)
Reinforcement learning (RL) has emerged as a critical technique for enhancing LLM-based deep search agents. However, existing approaches primarily rely on binary outcome rewards, which fail to capture the comprehensiveness and factuality of agents' reasoning process, and often lead to undesirable behaviors such as shortcut exploitation and hallucinations. To address these limitations, we propose \textbf{Citation-aware Rubric Rewards (CaRR)}, a fine-grained reward framework for deep search agents that emphasizes reasoning comprehensiveness, factual grounding, and evidence connectivity. CaRR decomposes complex questions into verifiable single-hop rubrics and requires agents to satisfy these rubrics by explicitly identifying hidden entities, supporting them with correct citations, and constructing complete evidence chains that link to the predicted answer. We further introduce \textbf{Citation-aware Group Relative Policy Optimization (C-GRPO)}, which combines CaRR and outcome rewards for training robust deep search agents. Experiments show that C-GRPO consistently outperforms standard outcome-based RL baselines across multiple deep search benchmarks. Our analysis also validates that C-GRPO effectively discourages shortcut exploitation, promotes comprehensive, evidence-grounded reasoning, and exhibits strong generalization to open-ended deep research tasks. Our code and data are available at https://github.com/THUDM/CaRR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。