用大规模异步强化学习，让搜索代理实现百次操作的长程智能搜索。
Beyond Ten Turns: Unlocking Long-Horizon Agentic Search with Large-Scale Asynchronous RL
- 采用全异步强化学习，支持超长对话轮次的搜索策略训练。
- 在xBench和GAIA测试中，准确率分别提升至78.0%和58.7%，超越现有开源模型。
- 无需外部大模型，仅靠提示词自动生成高质量数据，适合研究长程智能搜索者。
基于大语言模型的智能体在整合外部工具处理复杂知识任务方面已取得显著进展。搜索工具作为获取外部知识的核心手段，但开源智能体仍难以达到专家级搜索能力，主要受限于训练方法的可扩展性、效率与数据质量。现有在线强化学习方法通常将对话轮次限制在10轮以内，制约了复杂策略的学习。本文提出ASearcher，一个用于搜索智能体的大规模强化学习开源项目。核心贡献包括：(1) 可扩展的完全异步强化学习训练机制，支持长时序搜索并保持高效率；(2) 基于提示词的LLM智能体，可自主生成高质量且具有挑战性的问答对，构建大规模数据集。通过强化学习训练，其提示式QwQ-32B智能体在xBench和GAIA上分别获得78.0%和58.7%的Avg@4得分，显著优于现有开源32B模型。该智能体在训练过程中实现了超过100次工具调用及超过40万输出词元的长程探索。最终，不依赖外部大模型，仅用简单设计的ASearcher-Web-QwQ即达成51.1（xBench）和58.7（GAIA）的性能，媲美商业系统，并可在零样本迁移和测试时搜索中实现类似效果。项目代码、模型与数据已开源：https://github.com/inclusionAI/ASearcher。
原文摘要 · Abstract (English)
Recent advancements in LLM-based agents have demonstrated remarkable capabilities in handling complex, knowledge-intensive tasks by integrating external tools. Among diverse choices of tools, search tools play a pivotal role in accessing vast external knowledge. However, open-source agents still fall short of achieving expert-level Search Intelligence, the ability to resolve ambiguous queries, generate precise searches, analyze results, and conduct thorough exploration. Existing approaches fall short in scalability, efficiency, and data quality. For example, small turn limits in existing online RL methods, e.g. <=10, restrict complex strategy learning. This paper introduces ASearcher, an open-source project for large-scale RL training of search agents. Our key contributions include: (1) Scalable fully asynchronous RL training that enables long-horizon search while maintaining high training efficiency. (2) A prompt-based LLM agent that autonomously synthesizes high-quality and challenging QAs, creating a large-scale QA dataset. Through RL training, our prompt-based QwQ-32B agent achieves substantial improvements, with 78.0% and 34.3% Avg@4 gains on xBench and GAIA, respectively. Notably, our agent exhibits extreme long-horizon search, with tool calls exceeding 100 turns and output tokens exceeding 400k during training time. With a simple agent design and no external LLMs, ASearcher-Web-QwQ achieves Avg@4 scores of 51.1 on xBench and 58.7 on GAIA, surpassing existing open-source 32B agents. Finally, we also show that ASearcher-Web-QwQ could achieve performance of commercial systems using external summary tool in a zero-shot transfer manner and test-time search. We open-source our models, training data, and codes in https://github.com/inclusionAI/ASearcher.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。