用知识图谱和多轮强化学习提升大模型深度搜索能力
DeepDive: Advancing Deep Search Agents with Knowledge Graphs and Multi-Turn RL
- 从开放知识图谱自动生成复杂难题，构建训练数据
- 通过多轮强化学习增强大模型长期推理与工具使用能力
- 支持测试时并行采样和工具调用扩展，适合研究者复现
将大型语言模型（LLMs）与浏览工具结合可显著提升其解决复杂现实任务的深度搜索能力。然而，由于长程推理能力有限且缺乏足够困难的监督数据，开源LLM在此类任务中表现仍不佳。为此，我们提出DeepDive，以推进深度搜索代理的发展：首先，设计策略从开放知识图谱中自动生成复杂、困难且难以发现的问题；其次，采用端到端多轮强化学习（RL），提升LLM在深度搜索中的长期推理能力；为鼓励多样性并减少冗余，引入冗余惩罚机制，抑制重复相似查询。实验表明，DeepDive-32B在BrowseComp上达到新的开源基准性能，优于WebSailor、DeepSeek-R1-Browse和Search-o1。我们验证了多轮强化学习训练能显著提升深度搜索能力，并在多个基准上带来明显改进。此外，DeepDive实现了测试时的工具调用扩展与并行采样。所有数据集、模型与代码均已公开于https://github.com/THUDM/DeepDive。
原文摘要 · Abstract (English)
Augmenting large language models (LLMs) with browsing tools substantially improves their potential as deep search agents to solve complex, real-world tasks. Yet, open LLMs still perform poorly in such settings due to limited long-horizon reasoning capacity with browsing tools and the lack of sufficiently difficult supervised data. To address these challenges, we present DeepDive to advance deep search agents. First, we propose a strategy to automatically synthesize complex, difficult, and hard-to-find questions from open knowledge graphs. Second, we apply end-to-end multi-turn reinforcement learning (RL) to enhance LLMs' long-horizon reasoning with deep search. To encourage diversity and reduce redundancy, we design a redundancy penalty that discourages repeated similar queries. Experiments show that DeepDive-32B achieves a new open-source competitive result on BrowseComp, outperforming WebSailor, DeepSeek-R1-Browse, and Search-o1. We demonstrate that multi-turn RL training improves deep search ability and significantly contributes to the performance improvements across multiple benchmarks. We observe that DeepDive enables test-time scaling of tool calls and parallel sampling. All datasets, models, and code are publicly available at https://github.com/THUDM/DeepDive.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。