通过训练模型具备四项关键推理能力,显著提升智能搜索表现。
Beneficial Reasoning Behaviors in Agentic Search and Effective Post-training to Obtain Them
- 识别并强化信息验证、权威评估等四种有效推理行为
- 在三个网页搜索基准上比直接强化学习提升37.2%
- 适合需要多步推理的复杂信息检索任务研究者
智能搜索要求大语言模型执行多步搜索以解决复杂信息查询任务,对推理能力提出独特挑战。本文通过对比成功与失败轨迹,识别出四种有益推理行为:信息验证、权威评估、自适应搜索和错误恢复。基于此,提出行为引导(Behavior Priming)训练方法,在强化学习前通过监督微调已识别行为轨迹来培养这些能力,再进行标准强化学习优化。在Qwen3-1.7B和Llama3.2-3B-Instruct上的实验表明,该方法在三个网页搜索基准上相对直接强化学习提升37.2%,在七个多跳问答基准上提升6.2%,优于使用结果正确轨迹的SFT-then-RL基线。进一步分析显示,行为引导阶段的行为质量比结果正确性更重要,且增强了探索能力(pass@8)和测试时扩展性(搜索步数),为强化学习提供稳健基础。代码已开源。
原文摘要 · Abstract (English)
Agentic search requires large language models (LLMs) to perform multi-step search to solve complex information-seeking tasks, imposing unique challenges on their reasoning capabilities. However, what constitutes effective reasoning for agentic search and how it can be learned remains unclear. In this work, we first investigate the reasoning behaviors that enable success in agentic search. By comparing successful and failed trajectories via an LLM-based analysis pipeline, we identify four beneficial behaviors: Information Verification, Authority Evaluation, Adaptive Search, and Error Recovery. Building on this, we propose Behavior Priming, a training approach that equips agentic search models with these reasoning behaviors before reinforcement learning (RL). Specifically, it first performs supervised fine-tuning (SFT) on collected trajectories exhibiting the identified behaviors to cultivate these behaviors, and then applies standard RL to further improve task performance. Experiments on Qwen3-1.7B and Llama3.2-3B-Instruct show that Behavior Priming yields relative improvements over direct RL by 37.2\% on three web benchmarks and 6.2\% on seven multi-hop QA benchmarks, and outperforms the SFT-then-RL baseline using outcome-correct trajectories for fine-tuning. Crucially, we show that these reasoning behaviors matter more than outcome correctness in the priming stage prior to RL. Further analysis reveals that Behavior Priming enhances exploration (pass@8) and test-time scaling (search step number), providing a robust foundation for RL. Our code are avalible at https://github.com/cxcscmu/Behavior-Priming-for-Agentic-Search.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。