提升价值估计精度能让基础策略梯度算法媲美先进方法。
Improving Value Estimation Critically Enhances Vanilla Policy Gradient
- 增加每轮价值函数更新步数，改进价值估计
- 基础策略梯度在连续控制任务中达到或超越PPO性能
- 对超参数更鲁棒，利于实际应用
现代策略梯度算法（如TRPO、PPO）在多数强化学习任务中优于基础策略梯度。我们质疑了‘近似信任区域保证稳定策略提升’这一常见认知，发现更关键的因素是每轮迭代中更多价值更新步带来的价值估计精度提升。实验表明，仅通过增加每轮价值更新次数，基础策略梯度在所有标准连续控制基准环境中的表现即可达到或超过PPO水平。更重要的是，这种简单修改使算法对超参数选择更为鲁棒，暗示强化学习算法仍有潜力变得更高效且更易使用。
原文摘要 · Abstract (English)
Modern policy gradient algorithms, such as TRPO and PPO, outperform vanilla policy gradient in many RL tasks. Questioning the common belief that enforcing approximate trust regions leads to steady policy improvement in practice, we show that the more critical factor is the enhanced value estimation accuracy from more value update steps in each iteration. To demonstrate, we show that by simply increasing the number of value update steps per iteration, vanilla policy gradient itself can achieve performance comparable to or better than PPO in all the standard continuous control benchmark environments. Importantly, this simple change to vanilla policy gradient is significantly more robust to hyperparameter choices, opening up the possibility that RL algorithms may still become more effective and easier to use.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。