arXiv:2606.06746cs.LG2026-06被引 2

提出新方法评估强化学习训练稳定性,揭示算法间差异。

Performance Variation in Deep Reinforcement Learning

论文配图:Performance Variation in Deep Reinforcement Learning
图 1 · 摘自论文原文
  • 用分位数统计替代传统方差,更准确反映训练波动
  • 发现PPO和SAC在不同归一化下性能波动差异明显
  • 适合关注算法可靠性与可复现性的研究者参考

深度强化学习算法常表现出低运行间鲁棒性,即相同配置下多次独立训练结果差异显著。尽管该问题在研究与实践中带来广泛挑战,但现有研究多仅报告平均性能的不确定性,缺乏对变异性的系统评估。本文指出传统不确定性与变异度量的局限性,尤其其目的错位及低估风险。为此,我们提出基于分位数的统计指标(min-max IPR)与运行层面分位数可视化方法,二者均基于样本分位数的标准性质,直观且信息丰富。通过三个案例验证:第一,层归一化与倒数第二层归一化能缩小PPO的性能波动,而SAC则基本不变;第二,对比PPO、SAC、TD-MPC与TD-MPC2,TD-MPC波动最小且数据效率最高;第三,在五个Atari环境中比较DQN与Rainbow,两者性能波动水平相近。

原文摘要 · Abstract (English)

Deep reinforcement learning (RL) algorithms often suffer from low run-to-run robustness, manifesting as significant performance variation across independent runs of identically configured agents. Although this issue poses a spectrum of challenges across research and practice, relatively few studies develop methods to evaluate it; RL research instead often reports uncertainty in the estimated mean performance. In this paper, we outline the limitations of conventional uncertainty and variation estimates, particularly their misalignment with purpose and the risk of underreporting. We then propose an alternative percentile-based statistic and visualization method, min-max IPR and run-wise percentile highlighting, respectively. These percentile-based tools are easy to interpret and rely on standard properties of sample percentiles, providing rich information about run-to-run performance variation. We demonstrate this through three case studies. First, we show that LayerNorm and penultimate-layer normalizations narrow performance variation in PPO, whereas the variation is mostly unchanged in SAC. Second, we compare PPO, SAC, TD-MPC, and TD-MPC2, and show TD-MPC exhibits the least variation while being the most data efficient among the four. Finally, in a comparison of DQN and Rainbow on five Atari environments, we show that both algorithms exhibit similar levels of performance variation.

强化学习稳定性可复现性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。