SPEQ通过周期性离线优化，显著降低高更新率强化学习的计算开销。
SPEQ: Offline Stabilization Phases for Efficient Q-Learning in High Update-To-Data Ratio Reinforcement Learning
- 将在线训练与周期性离线调优结合，分阶段更新策略
- 相比顶尖方法减少40%~99%梯度更新，节省27%~78%训练时间
- 适合计算资源受限的真实场景，兼顾效率与性能
高更新率比（UTD）的强化学习算法虽提升样本效率，但计算成本高昂，限制实际应用。本文提出离线稳定化阶段高效Q学习（SPEQ），结合低UTD在线训练与周期性离线稳定阶段。在稳定阶段，使用固定回放缓冲区以高UTD比例对Q函数进行微调，避免对次优数据重复更新。该结构化训练策略在计算与样本效率间取得平衡，克服了高低UTD方法的局限。实验表明，SPEQ在MuJoCo连续控制基准测试中，相比当前最优高UTD方法，梯度更新次数减少40%至99%，训练时间缩短27%至78%，同时性能相当或更优。研究揭示周期性稳定化阶段是传统训练流程的有效替代方案，为计算资源受限场景下的可扩展强化学习提供新路径。
原文摘要 · Abstract (English)
High update-to-data (UTD) ratio algorithms in reinforcement learning (RL) improve sample efficiency but incur high computational costs, limiting real-world scalability. We propose Offline Stabilization Phases for Efficient Q-Learning (SPEQ), an RL algorithm that combines low-UTD online training with periodic offline stabilization phases. During these phases, Q-functions are fine-tuned with high UTD ratios on a fixed replay buffer, reducing redundant updates on suboptimal data. This structured training schedule optimally balances computational and sample efficiency, addressing the limitations of both high and low UTD ratio approaches. We empirically demonstrate that SPEQ requires from 40% to 99% fewer gradient updates and 27% to 78% less training time compared to state-of-the-art high UTD ratio methods while maintaining or surpassing their performance on the MuJoCo continuous control benchmark. Our findings highlight the potential of periodic stabilization phases as an effective alternative to conventional training schedules, paving the way for more scalable reinforcement learning solutions in real-world applications where computational resources are constrained.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。