用可复用的评分规则库，让搜索模型从过程中学到更好策略。
ARBOR: Online Process Rewards via a Reusable Rubric Buffer for Search Agents
- 构建共享的评分规则缓冲区，跨查询复用过程评判标准。
- 在结果相同的训练组中仍能生成有效梯度，提升4.2分准确率。
- 适合需要高效训练搜索代理的研究者和开发者。
基于大模型的搜索代理通常仅通过最终结果奖励进行训练，导致搜索过程缺乏监督。当样本轨迹结果一致时，这种信号会退化为零梯度。现有方法要么需昂贵验证器，要么生成一次性使用的查询专属评分标准。我们提出 ARBOR（自适应可复用评分缓冲区），通过维护跨查询共享的规则记忆，将对比轨迹生成的草稿整合为通用评分规则，并随策略演化逐步淘汰。少量活跃规则以稀疏成对判断打分，得分加入基础奖励，使即使在结果一致的情况下也能提供过程级梯度。在四个多跳问答基准上，ARBOR持续优于GRPO和DAPO基线，平均提升大模型判别准确率4.2点，将最多42%原本无梯度的训练组转化为有信息量的组。
原文摘要 · Abstract (English)
LLM-based search agents are trained predominantly with outcome-only reward, leaving the search process itself unsupervised. This signal degenerates on outcome-homogeneous groups where all sampled trajectories share the same correctness, yielding zero within-group advantage and no gradient. Existing process supervision either trains a costly verifier or generates per-query rubrics that are inconsistent across queries and discarded after one use. We propose ARBOR (Adaptive Rubric Buffer for Online Reward), a reusable process-reward framework that maintains a rubric memory shared across queries. Query-local drafts induced from contrastive trajectories are admitted, consolidated into cross-query common rubrics, and retired as the policy evolves. A small active subset of common rubrics scores trajectories via sparse pairwise judging, and the resulting scores are added to the base reward, providing process-level gradient even when outcome reward is uniform. ARBOR consistently outperforms GRPO and DAPO baselines on four multi-hop QA benchmarks, raising average LLM-judge accuracy by up to 4.2 points and converting up to 42% of otherwise-zero-gradient training groups into informative ones.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。