研究强化学习中两种正则化如何协同提升算法稳定性
On the Effect of Regularization in Policy Mirror Descent
- 通过大规模实验分析两种正则化机制的交互作用
- 发现二者可部分替代，但组合方式直接影响性能鲁棒性
- 适合关注算法稳定性和超参数敏感性的研究人员
策略镜面下降（PMD）作为强化学习中的统一框架，将策略梯度方法与一阶优化方法镜面下降相联系。其核心包含两个关键正则化成分：(i) 距离项，用于在策略更新中建立信任区域以保证稳定性；(ii) MDP 正则项，通过增强奖励函数来促进结构和鲁棒性。尽管 PMD 在理论上已得到广泛研究，但其在实践中的验证仍较少。本文对这两种正则化技术的相互作用进行了大规模实证分析，在小型强化学习环境中运行超过 50 万次训练种子。结果表明，虽然两个正则项可部分相互替代，但它们的精确组合对于实现鲁棒性能至关重要。这些发现凸显了在强化学习中推进更鲁棒算法研究的潜力，特别是针对超参数敏感性问题。
原文摘要 · Abstract (English)
Policy Mirror Descent (PMD) has emerged as a unifying framework in reinforcement learning (RL) by linking policy gradient methods with a first-order optimization method known as mirror descent. At its core, PMD incorporates two key regularization components: (i) a distance term that enforces a trust region for stable policy updates and (ii) an MDP regularizer that augments the reward function to promote structure and robustness. While PMD has been extensively studied in theory, empirical investigations remain scarce. This work provides a large-scale empirical analysis of the interplay between these two regularization techniques, running over 500k training seeds on small RL environments. Our results demonstrate that, although the two regularizers can partially substitute each other, their precise combination is critical for achieving robust performance. These findings highlight the potential for advancing research on more robust algorithms in RL, particularly with respect to hyperparameter sensitivity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。