arXiv:2606.03070cs.LGcs.AI2026-06

无需行为信息,用当前策略概率稳定异步强化学习训练。

ASymPO: Asymmetric-Scale Policy Optimization for Asynchronous LLM Post-Training Without Behavior Information

论文配图:ASymPO: Asymmetric-Scale Policy Optimization for Asynchronous LLM Post-Training Without Behavior Information
图 1 · 摘自论文原文
  • 仅用当前策略概率,通过尺度归一化解决响应过时问题。
  • 在数学推理任务中,训练速度提升3.2倍且性能不降。
  • 适合追求高吞吐、无行为日志系统的模型后训练场景。

异步强化学习可通过解耦生成与策略优化提升语言模型后训练效率,但过时响应会导致分布漂移。传统行为修正方法依赖对齐的、版本化的、数值一致的行为策略概率,包括概率比、重要性权重或截断。本文探究是否可仅用当前策略概率稳定异步组相对强化学习。发现存在尺度失衡失效模式:当过时响应在当前策略下评估时,正负损失项处于不同负对数概率尺度,零和优势不再对应平衡损失贡献。为此提出非对称尺度策略优化(ASymPO),将每个响应的词元损失除以其当前平均词元负对数概率。ASymPO无需行为策略概率,恢复响应级零和平衡并保留非零学习信号。同时引入固定负尺度基线(SPO),在异步数学推理后训练中评估两者仅用当前策略的目标。

原文摘要 · Abstract (English)

Asynchronous reinforcement learning can improve language-model post-training throughput by decoupling response generation from policy optimization, but stale responses introduce distribution drift. Standard behavior-corrected methods control this drift with behavior-policy probabilities, importance ratios, or clipping, which requires token-aligned, versioned, and numerically consistent behavior log-probabilities across rollout and learner systems. We ask whether asynchronous group-relative RL can instead be stabilized using only current-policy probabilities. We identify a scale-imbalance failure mode: when stale responses are evaluated under the current policy, positive and negative loss terms can appear at different negative log-probability scales, so zero-sum advantages no longer imply balanced loss contributions. We propose Asymmetric-Scale Policy Optimization (ASymPO), which normalizes each response's token loss by its current average token negative log-probability. ASymPO requires no behavior-policy probabilities, restores response-level zero-sum balance, and preserves a nonzero learning signal. We also introduce Scaled Policy Optimization (SPO), a fixed negative-scaling baseline, and evaluate both current-policy-only objectives in asynchronous mathematical reasoning post-training.

强化学习异步训练语言模型后训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。