arXiv:2603.21383cs.AI2026-03被引 5

用少量计算提升长序列智能体任务的准确率,兼顾泛化能力。

PivotRL: High Accuracy Agentic Post-Training at Low Compute Cost

  • 基于SFT轨迹做局部策略优化,筛选高价值中间步骤作为训练信号。
  • 在4个智能体领域内平均提升4.17%精度,非智能体任务上提升10.04%。
  • 仅需E2E RL四分之一的推演轮次,即可达到相近编码任务表现。

长时序智能体任务的后训练面临计算效率与泛化能力之间的矛盾:监督微调(SFT)虽计算高效,但易出现域外(OOD)性能下降;端到端强化学习(E2E RL)保留域外能力,却因大量在线推演带来高计算开销。本文提出PivotRL框架,利用已有SFT轨迹,结合SFT的计算效率与E2E RL的域外准确性。其核心机制包括:1)执行局部在线推演,筛选出动作结果方差大的关键中间步骤(称为pivots);2)采用功能等价动作的奖励信号,而非严格匹配SFT示范文本。理论证明,该机制可激发高自然梯度幅值的学习信号,同时最大程度保持无关任务动作的概率排序。在相同数据下,相较于标准SFT,PivotRL在四个智能体领域平均提升4.17%的域内准确率,在非智能体任务上提升10.04%的域外准确率。在智能体编码任务中,仅需E2E RL四分之一的推演轮次即达相近精度。该方法已被NVIDIA Nemotron-3-Super-120B-A12B采纳,用于生产级智能体后训练。

原文摘要 · Abstract (English)

Post-training for long-horizon agentic tasks has a tension between compute efficiency and generalization. While supervised fine-tuning (SFT) is compute efficient, it often suffers from out-of-domain (OOD) degradation. Conversely, end-to-end reinforcement learning (E2E RL) preserves OOD capabilities, but incurs high compute costs due to many turns of on-policy rollout. We introduce PivotRL, a novel framework that operates on existing SFT trajectories to combine the compute efficiency of SFT with the OOD accuracy of E2E RL. PivotRL relies on two key mechanisms: first, it executes local, on-policy rollouts and filters for pivots: informative intermediate turns where sampled actions exhibit high variance in outcomes; second, it utilizes rewards for functional-equivalent actions rather than demanding strict string matching with the SFT data demonstration. We theoretically show that these mechanisms incentivize strong learning signals with high natural gradient norm, while maximally preserving policy probability ordering on actions unrelated to training tasks. In comparison to standard SFT on identical data, we demonstrate that PivotRL achieves +4.17% higher in-domain accuracy on average across four agentic domains, and +10.04% higher OOD accuracy in non-agentic tasks. Notably, on agentic coding tasks, PivotRL achieves competitive accuracy with E2E RL with 4x fewer rollout turns. PivotRL is adopted by NVIDIA's Nemotron-3-Super-120B-A12B, acting as the workhorse in production-scale agentic post-training.

智能体强化学习后训练高效算法

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。