改进异步智能体强化学习的优化方法，提升训练效率。
SPO++: Stream-Aligned Policy Optimization for Asynchronous Agentic RL
- 用动作词元度量标准化优势，修正原方法偏差
- 在ALFWorld和Math-TIR上实现更高效的在线学习
- 适合研究异步强化学习与智能体系统的设计者
组相对强化学习需等待同一提示的子轨迹完成，对长而多变的工具使用轨迹成本高昂。单流策略优化（SPO）通过持久的提示级价值估计消除了这一依赖，但其方案在优化逐词均值演员损失前会抹去每条轨迹的一个优势。我们发现轨迹中心化通常无法使演员消耗的词元加权量中心化，因此通过动作词元度量下的终局回报优势标准化来修复这一偏差。此外，我们按生成该提示证据的策略事件组织数据，而非按学习者接收顺序。在两个模型规模下对ALFWorld及Math-TIR的匹配实验表明，SPO++相比SPO提升了在线学习效率。配对消融实验确认动作词元度量归一化为最强有效组件。
原文摘要 · Abstract (English)
Group-relative reinforcement learning waits for sibling rollouts of the same prompt, which is costly for long and variable tool-use trajectories. Single-stream Policy Optimization (SPO) removes this dependency with a persistent prompt-level value estimate, but its recipe whitens one advantage per trajectory before optimizing a token-mean actor loss. We show that trajectory centering generally does not center the token-weighted quantity consumed by the actor, and fix the mismatch by standardizing terminal-outcome advantages under the action-token measure. We additionally organize prompt evidence by the policy event that generated it rather than learner receipt order. Across matched runs on ALFWorld at two model scales and on Math-TIR, SPO++ improves online learning efficiency over SPO. A paired ablation identifies action-token-measure normalization as the strongest tested component.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。