arXiv:2607.22186cs.AI2026-07被引 1

提出熵缩放信任域,解决异步强化学习中数据过时导致的策略崩溃问题。

Deconstructing Off-Policy Ratios: Entropy-Scaled Trust Regions for Asynchronous Reinforcement Learning

论文配图:Deconstructing Off-Policy Ratios: Entropy-Scaled Trust Regions for Asynchronous Reinforcement Learning
图 1 · 摘自论文原文
  • 根据局部词元熵动态调整重要性比率,避免统一阈值误判
  • 在长序列任务上实现37.34(BrowseComp-Plus)和95.69(GSM8K)得分
  • 无需额外前向传播,支持2.6倍加速,适合大模型后训练场景

异步强化学习通过重叠采样生成与策略优化提升大语言模型后训练效率,但产生的滞后非策略数据会引发优化不稳定甚至策略崩溃。现有方法仅依据重要性比率大小决定保留或丢弃,对所有位置使用统一阈值。本文发现,重要性比率的自然尺度随词元熵系统性变化:低熵时训练-推理差异被大幅放大为显著采样噪声;高熵时运行中的权重更新会自然引发明显且合理的探索偏差。单纯基于幅度的修正会无意引入放大噪声,同时压制由更新触发的必要探索。为此,我们提出熵缩放信任域(ESTR),按局部熵缩放每个词元的非策略偏差,无需额外前向传播或版本切换检测。在长时序代理任务和数学推理基准上,ESTR持续优于现有异步方法,在BrowseComp-Plus达37.34平均@1,在多轮GSM8K达95.69,媲美同步GRPO,同时获得2.6倍加速。

原文摘要 · Abstract (English)

Asynchronous reinforcement learning (RL) accelerates large language model (LLM) post-training by overlapping rollout generation with policy optimization, but the resulting stale, off-policy data can destabilize optimization and ultimately cause policy collapse. Existing methods typically retain or discard tokens based solely on the magnitude of their importance ratios, applying the same threshold uniformly across token positions. In this work, we reveal that the natural scale of the importance ratio varies systematically with token entropy. Under asynchronous dynamics, this entropy-ratio scaling dictates two distinct phenomena: at low entropy, the inherent train-inference discrepancy is drastically amplified into substantial sampling noise; at high entropy, in-flight weight updates naturally induce pronounced, legitimate exploratory deviations. Consequently, magnitude-only correction inadvertently admits the amplified noise while strictly masking out the essential exploration triggered by in-flight updates. To address this, we propose the Entropy-Scaled Trust Region (ESTR), which scales each token's off-policy deviation by its local entropy, requiring no auxiliary forward passes or explicit version-switch detection. Across long-horizon agentic tasks and mathematical reasoning benchmarks, ESTR consistently outperforms existing asynchronous methods and achieves the best train-inference consistency. It reaches $37.34$ avg@1 on BrowseComp-Plus and $95.69$ on multi-turn GSM8K, matching synchronous GRPO while achieving a $2.6\times$ speedup.

强化学习异步训练大模型后训练熵缩放

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。