针对大规模并行训练，提出自适应稳定策略的强化学习算法
WarpSAC: Towards the Pinnacle of Scalable Off-policy RL by Rethinking Exploration and Exploitation
- 根据数据规模动态调整探索与利用策略，设计双版本算法
- 在多种环境上提升性能，最高加速比达36.4%且成功率从19.8%升至96.4%
- 适合高并发模拟场景下的强化学习部署，尤其适用于硬件资源充足时
大规模并行仿真改变了离线策略强化学习的数据格局，挑战了为数据有限场景设计的稳定性机制。通过在八个基准家族上的控制实验，我们发现这些稳定器具有数据格局依赖性：参数归一化在回放覆盖窄时有效，但在数据丰富时限制价值拟合；剪裁双Q可被放宽于高吞吐量操作中。年龄偏倚回放缓冲加权在各数据格局下均提升学习效率，尤其在网络容量受限时。基于此，我们提出WarpSAC，一种面向数据格局自适应的离线策略强化学习算法族。WarpSAC采用样本权重衰减实现高效利用，并提供两个变体：WarpSAC-L（归一化开启，剪裁双Q）用于数据受限的CPU规模训练，WarpSAC-A（归一化关闭，单Q）用于数据丰富的GPU并行训练。WarpSAC在九个CPU规模环境中使归一化得分-步数AUC较FlashSAC提升4.5%，在十四个GPU并行环境中提升23.1%。其将UnitreeG1TransportBox-v1的成功率从19.8%提升至96.4%，使MuJoCo Playground的平均归一化墙时AUC提高19.1%，并在Unitree G1上实现36.4%更快速的仿真到现实部署。结果表明，可扩展的离线策略强化学习应根据可用数据格局调整其稳定机制。
原文摘要 · Abstract (English)
Massively parallel simulation changes the data regime in which off-policy reinforcement learning (RL) is trained, challenging stabilizers designed for data-limited replay. Through controlled experiments across eight benchmark families, we show that these stabilizers are data-regime-dependent: parameter normalization helps with narrow replay coverage but restricts value fitting when data are abundant, while clipped double-Q can be relaxed in high-throughput manipulation. Age-biased replay weighting improves learning efficiency across regimes, especially with limited network capacity. Based on these findings, we propose WarpSAC, a regime-aware family of off-policy RL algorithms. WarpSAC uses Sample Weight Decay for efficient exploitation and provides two variants: WarpSAC-L (Norm ON, clipped double-Q) for data-limited CPU-scale training, and WarpSAC-A (Norm OFF, single-Q) for data-abundant GPU-parallel training. WarpSAC improves normalized score--step AUC over FlashSAC by 4.5% across nine CPU-scale environments and 23.1% across fourteen GPU-parallel environments. It increases UnitreeG1TransportBox-v1 success rate from 19.8% to 96.4%, improves mean normalized wall-time AUC on MuJoCo Playground by 19.1%, and achieves 36.4% faster sim-to-real deployment on Unitree G1 than FlashSAC. These results show that scalable off-policy RL should adapt its stabilizers to the available data regime.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。