自动生成真实多跳搜索任务，让大模型自我训练提升搜索能力。
SearchMaster: Grounded and Regulated Self-Play for Search Agents
- 用显式证据链生成任务，避免虚假多跳问题。
- 按成功搜索深度评分，确保任务具有挑战性。
- 惩罚过度打开文档，引导精准信息获取，适合想练搜索的开发者。
训练基于大语言模型的搜索代理需要高质量的搜索数据：要求真正多跳检索的任务和有效使用搜索工具的轨迹。现有方法常依赖人工编写任务、专家示范或更强的教师模型。我们提出SearchMaster，一种自对弈框架，让单一LLM在本地搜索环境中自主生成、求解并验证搜索任务。主要挑战在于自生成任务和轨迹可能带来误导信号：伪多跳问题、仅以成功率评估难度忽略搜索深度、以及大量打开文档但缺乏针对性证据获取。SearchMaster通过三项控制机制解决：证据链生成器（ECG）通过显式跨文档证据链约束任务生成，减少伪多跳；搜索深度奖励（SDR）根据成功轨迹的搜索深度而非单纯成功率评估任务难度，保证任务具备搜索强度；过开惩罚（OOP）通过抑制过度文档打开来规范工具使用，防止长而浅的浏览。经验证的提议与求解轨迹采用GRPO联合优化。在六个深度搜索基准上，SearchMaster将Qwen3.5-9B基线模型平均准确率从38.19%提升至51.52%，在BrowseComp-Plus上提升30.1个百分点。结果表明，有依据且受控的自对弈可无需人工标注问答对或专家示范，生成有效的搜索代理训练数据。代码已开源。
原文摘要 · Abstract (English)
Training LLM-based search agents requires high-quality search data: tasks that demand genuine multi-hop retrieval and trajectories that use search tools effectively. Existing pipelines often depend on human-written tasks, expert demonstrations, or stronger teacher models. We present SearchMaster, a self-play framework that trains a single LLM from search tasks it generates, solves, and verifies in a local search environment. The key challenge is that self-generated tasks and rollouts can yield misleading signals: pseudo multi-hop questions, success-rate difficulty estimates that ignore search depth, and rollouts with excessive opening but little targeted evidence acquisition. SearchMaster addresses these failure modes with three controls. An Evidence-Chain Generator (ECG) grounds task generation in explicit cross-document evidence chains to reduce pseudo multi-hop questions. A Search-Depth Reward (SDR) scores task difficulty by the search depth of successful rollouts rather than success rate alone, keeping retained tasks search-intensive. An Over-Opening Penalty (OOP) regulates tool use by discouraging excessive document opening, avoiding long but shallow browsing. Verified Proposer and Solver rollouts are then jointly optimized with GRPO. Across six deep-search benchmarks, SearchMaster improves a Qwen3.5-9B backbone from 38.19% to 51.52% average accuracy, with a 30.1-point gain on BrowseComp-Plus. These results show that grounded and regulated self-play can provide effective search-agent training data without human-labeled QA pairs or expert demonstrations. The code is available at https://github.com/WentaoTan/SearchMaster.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。