同一算法不同实现差异巨大，盲目替换会颠覆研究结论。
On the Mistaken Assumption of Interchangeable Deep Reinforcement Learning Implementations
- 通过差分测试对比多个PPO实现，发现代码细节导致性能天差地别。
- 5个PPO实现中，3个在56个游戏中超人类表现达50%，另2个不足15%。
- 实证表明：代码不一致会改变实验结果，需警惕算法实现的可比性。
深度强化学习（DRL）近年来在驾驶模拟器、3D机器人控制和多人在线竞技游戏等复杂环境中取得显著进展。尽管已有众多先进算法（如DQN、PPO）的实现版本，但研究普遍错误假设相同算法的不同实现具有可比性且可互换。本文通过差分测试揭示了实现间的显著差异及其对性能和先前研究结论的影响。在56个游戏上测试5个PPO实现时，3个实现有50%的试验达到超人类水平，而其余2个不足15%。手动分析源码发现，代码级差异是主因。此外，我们复现一项研究，证明该互换假设足以反转实验结果。因此，必须重新审视算法实现的使用方式。
原文摘要 · Abstract (English)
Deep Reinforcement Learning (DRL) is a paradigm of artificial intelligence where an agent uses a neural network to learn which actions to take in a given environment. DRL has recently gained traction from being able to solve complex environments like driving simulators, 3D robotic control, and multiplayer-online-battle-arena video games. Numerous implementations of the state-of-the-art algorithms responsible for training these agents, like the Deep Q-Network (DQN) and Proximal Policy Optimization (PPO) algorithms, currently exist. However, studies make the mistake of assuming implementations of the same algorithm to be consistent and thus, interchangeable. In this paper, through a differential testing lens, we present the results of studying the extent of implementation inconsistencies, their effect on the implementations' performance, as well as their impact on the conclusions of prior studies under the assumption of interchangeable implementations. The outcomes of our differential tests showed significant discrepancies between the tested algorithm implementations, indicating that they are not interchangeable. In particular, out of the five PPO implementations tested on 56 games, three implementations achieved superhuman performance for 50% of their total trials while the other two implementations only achieved superhuman performance for less than 15% of their total trials. As part of a meticulous manual analysis of the implementations' source code, we analyzed implementation discrepancies and determined that code-level inconsistencies primarily caused these discrepancies. Lastly, we replicated a study and showed that this assumption of implementation interchangeability was sufficient to flip experiment outcomes. Therefore, this calls for a shift in how implementations are being used.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。