用搜索深度动态分配训练资源,让AI更高效学习如何检索信息。
CuSearch: Curriculum Rollout Sampling via Search Depth for Agentic RAG

- 根据搜索深度智能分配训练样本,优先聚焦更深的检索路径。
- 在零搜索任务上提升11.8%的精确匹配率,性能显著优于传统方法。
- 无需人工标注,自动适应训练过程中的深度分布变化,适合强化学习型RAG系统。
基于可验证奖励的强化学习(RLVR)已成为从结果监督中训练智能体增强生成(RAG)系统的一种有前景的方法。现有方法通常对轨迹进行均匀采样,隐含假设所有轨迹具有同等信息量。然而,不同轨迹的搜索深度差异显著:更深的轨迹包含更多检索决策点,能为检索子策略提供更密集的直接监督。随着训练进行,批次内深度分布向更高值偏移,但均匀采样无法感知这一变化。为此,我们提出CuSearch,一种基于搜索深度贪婪分配(SDGA)的课程化轨迹采样框架。SDGA-Auto始终将更新预算集中于当前批次中最深的轨迹,随深度分布上升形成隐式训练对齐的课程;SDGA-Phase则显式推进课程阈值,当足够多深搜轨迹出现时触发。跨模型类型与检索框架的实验表明,CuSearch持续提升性能,在零搜索任务上相比标准GRPO最高提升11.8个精确匹配点。结果确立了每条轨迹的搜索深度作为RLVR驱动的RAG训练中检索监督密度的可靠、无需标注的代理指标。
原文摘要 · Abstract (English)
Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a promising paradigm for training agentic retrieval-augmented generation (RAG) systems from outcome-only supervision. Most existing methods optimize policies from uniformly sampled rollouts, implicitly treating all trajectories as equally informative. However, trajectories differ substantially in search depth and are therefore not equally informative: deeper-search trajectories contain more retrieval decision points and provide denser direct supervision for the retrieval sub-policy. Moreover, this heterogeneity grows over training as the within-batch depth distribution shifts toward higher values, yet uniform rollout sampling remains blind to this shift. To address this, we propose CuSearch, a curriculum rollout sampling framework built on Search-Depth Greedy Allocation (SDGA), a batch-level operator that reallocates a fixed update budget toward deeper-search trajectories. SDGA-Auto always targets the deepest available trajectories in the current batch, yielding an implicit training-aligned curriculum as the depth distribution shifts upward. SDGA-Phase explicitly advances the curriculum threshold as deeper trajectories become sufficiently abundant. Experiments across model types and retrieval frameworks show that CuSearch consistently improves performance, achieving up to 11.8 exact-match points over standard GRPO on ZeroSearch. These results establish per-trajectory search depth as a reliable, annotation-free proxy for retrieval supervision density in RLVR-based agentic RAG training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。