发现像素强化学习缩放失效的根源是编码器与全连接层间的瓶颈
Mind the GAP! The Challenges of Scale in Pixel-based Deep Reinforcement Learning
- 提出用全局平均池化直接突破编码器-全连接层瓶颈
- 在多种环境中实现性能随规模提升而稳定增长
- 适合关注强化学习可扩展性的研究者参考
在基于像素的强化学习中，模型规模扩大常导致性能下降。本文揭示了问题根源：编码器（多层卷积）输出与后续全连接层之间的连接存在瓶颈，该瓶颈限制了模型扩展能力。我们证明此前多数方法实际上都在间接缓解这一瓶颈。基于此分析，提出使用全局平均池化（Global Average Pooling）这一简单有效的方法，直接打破瓶颈，避免了复杂架构设计，显著提升了模型在不同环境下的可扩展性。
原文摘要 · Abstract (English)
Scaling deep reinforcement learning in pixel-based environments presents a significant challenge, often resulting in diminished performance. While recent works have proposed algorithmic and architectural approaches to address this, the underlying cause of the performance drop remains unclear. In this paper, we identify the connection between the output of the encoder (a stack of convolutional layers) and the ensuing dense layers as the main underlying factor limiting scaling capabilities; we denote this connection as the bottleneck, and we demonstrate that previous approaches implicitly target this bottleneck. As a result of our analyses, we present global average pooling as a simple yet effective way of targeting the bottleneck, thereby avoiding the complexity of earlier approaches.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。