用动态知识图谱和多奖励强化学习,让搜索代理更准更快不绕路。
DynaSearcher: Dynamic Knowledge Graph Augmented Search Agent via Multi-Reward Reinforcement Learning
- 用动态知识图谱约束中间查询,保证事实一致性。
- 多奖励强化学习提升准确率、效率和答案质量,减少冗余探索。
- 小模型也能达顶尖效果,适合资源受限场景。
基于大语言模型的多步智能体检索系统在复杂信息搜索任务中表现优异,但仍面临生成事实不一致的中间查询和低效搜索路径的问题,易导致推理偏差或冗余计算。为此,我们提出DynaSearcher,一种通过动态知识图谱与多奖励强化学习增强的搜索代理。系统利用知识图谱作为外部结构化知识,显式建模实体关系,引导搜索过程,确保中间查询的事实一致性,并减轻无关信息带来的偏差。同时,采用多奖励强化学习框架,对检索准确率、效率和回答质量等目标进行细粒度控制,促进高质量中间查询和完整最终答案的生成,抑制不必要的探索,减少信息遗漏或重复。实验表明,该方法在六个多跳问答数据集上达到当前最优答案准确率,仅使用小规模模型和有限计算资源即匹配前沿大模型性能。此外,方法在多种检索环境及更大规模模型下均展现出强泛化性与鲁棒性,凸显其广泛适用性。
原文摘要 · Abstract (English)
Multi-step agentic retrieval systems based on large language models (LLMs) have demonstrated remarkable performance in complex information search tasks. However, these systems still face significant challenges in practical applications, particularly in generating factually inconsistent intermediate queries and inefficient search trajectories, which can lead to reasoning deviations or redundant computations. To address these issues, we propose DynaSearcher, an innovative search agent enhanced by dynamic knowledge graphs and multi-reward reinforcement learning (RL). Specifically, our system leverages knowledge graphs as external structured knowledge to guide the search process by explicitly modeling entity relationships, thereby ensuring factual consistency in intermediate queries and mitigating biases from irrelevant information. Furthermore, we employ a multi-reward RL framework for fine-grained control over training objectives such as retrieval accuracy, efficiency, and response quality. This framework promotes the generation of high-quality intermediate queries and comprehensive final answers, while discouraging unnecessary exploration and minimizing information omissions or redundancy. Experimental results demonstrate that our approach achieves state-of-the-art answer accuracy on six multi-hop question answering datasets, matching frontier LLMs while using only small-scale models and limited computational resources. Furthermore, our approach demonstrates strong generalization and robustness across diverse retrieval environments and larger-scale models, highlighting its broad applicability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。