让搜索代理先规划再检索,提升多跳问答准确率
Plan Before Search: Search Agents Need Plan

- 先分解问题为有序子问题,避免检索时受无关文档干扰
- 在3B到14B模型上验证,相同奖励导致不同失败模式
- 用小模型自生成轨迹激活规划能力,无需外部强模型蒸馏
将大语言模型训练为检索增强型推理代理,通常结合强化学习与从更强模型蒸馏的SFT冷启动。然而,这一范式忽略了两个基本因素:子技能间的依赖结构,以及蒸馏并非获取能力的唯一路径。我们通过Plan——一种多跳检索的结构化代理行为——研究此问题:在任何检索前,将问题分解为有序子问题,使每一步检索都能锚定在预设子问题上,而非受早期部分相关文档影响而漂移。在涵盖3B至14B参数的三个模型家族中,我们发现相同的奖励信号会引发质性不同的强化学习失败模式。这表明成功训练不仅取决于奖励设计,还依赖于模型特定的可行性条件:足够的初始熵、训练稳定性及前置子技能。为此,我们提出一种自启动范式,由小型种子模型生成过滤后的轨迹,在任意目标模型中激活Plan,无需从外部更强模型蒸馏。我们的流程在所有测试模型上均激活Plan,并在多跳问答基准上持续优于竞争基线。
原文摘要 · Abstract (English)
Training large language models as retrieval-augmented reasoning agents typically combines reinforcement learning with an SFT cold start distilled from a stronger model. However, this paradigm overlooks two fundamental factors: the dependency structure among sub-skills, and the possibility that distillation is not the only route to capability acquisition. We study this through Plan, a structured agentic behavior for multi-hop retrieval that decomposes a question into ordered sub-questions before any retrieval is performed, so that each search step can be anchored to a pre-designed sub-question instead of drifting under the influence of partially relevant documents retrieved earlier. However, across three model families spanning 3B to 14B parameters, we find that an identical reward signal induces qualitatively different RL failure modes. This phenomenon indicates that successful training hinges not only on reward design but also on model-specific feasibility conditions: sufficient initial entropy, training stability, and prerequisite sub-skills. Motivated by this, we propose a self-bootstrapping paradigm in which a small-scale seed model generates filtered trajectories that activate Plan in any target model, eliminating the need for distillation from an external stronger model. Our pipeline activates Plan across every tested model and consistently outperforms competitive baselines on multi-hop QA benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。