通过智能选择更新时机，提升强化学习训练稳定性与性能。
Constant in an Ever-Changing World
- 保留代表策略与当前策略，仅在当前策略更优时更新代表策略
- 在五个MuJoCo环境中显著提升传统算法性能，且无额外计算开销
- 适合追求训练稳定性的强化学习研究者与工程应用
强化学习的训练过程常因严重震荡导致不稳定和性能下降。本文提出恒定于变化世界（CIC）框架，以增强算法稳定性并提升性能。CIC同时维护一个代表策略和一个当前策略，不盲目更新代表策略，仅当当前策略表现更优时才进行更新。此外，CIC采用自适应调节机制，使代表策略与当前策略协同促进价值函数估计的训练。我们在五个MuJoCo环境上评估了CIC，结果表明其在不增加额外计算成本的前提下，有效提升了传统算法的性能。
原文摘要 · Abstract (English)
The training process of reinforcement learning often suffers from severe oscillations, leading to instability and degraded performance. In this paper, we propose a Constant in an Ever-Changing World (CIC) framework that enhances algorithmic stability to improve performance. CIC maintains both a representative policy and a current policy. Instead of updating the representative policy blindly, CIC selectively updates it only when the current policy demonstrates superiority. Furthermore, CIC employs an adaptive adjustment mechanism, enabling the representative and current policies to jointly facilitate critic training. We evaluate CIC on five MuJoCo environments, and the results show that CIC improves the performance of conventional algorithms without incurring additional computational cost.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。