用平均池化替代全连接，提升图像强化学习泛化能力
Impoola: The Power of Average Pooling for Image-Based Deep Reinforcement Learning
- 用全局平均池化替代特征图展平，简化网络结构
- 在Procgen基准上超越更大更复杂的模型，泛化性能显著提升
- 适合关注模型效率与泛化能力的强化学习研究者
随着图像强化学习任务日益复杂，模型规模扩大成为提升性能的关键。当前主流采用15层受ResNet启发的Impala-CNN作为图像编码器，但其设计仍有优化空间。本文发现，将Impala-CNN中特征图的展平操作替换为全局平均池化，可显著提升性能。该改进模型称为Impoola-CNN，在Procgen基准测试中表现优于更大更复杂的模型，尤其在无中心观测的游戏场景中提升最为明显。我们认为网络的平移敏感性降低是性能提升的关键原因。结果表明，模型扩展不仅依赖增大规模，高效网络设计同样重要。代码已开源：https://github.com/raphajaner/impoola。
原文摘要 · Abstract (English)
As image-based deep reinforcement learning tackles more challenging tasks, increasing model size has become an important factor in improving performance. Recent studies achieved this by focusing on the parameter efficiency of scaled networks, typically using Impala-CNN, a 15-layer ResNet-inspired network, as the image encoder. However, while Impala-CNN evidently outperforms older CNN architectures, potential advancements in network design for deep reinforcement learning-specific image encoders remain largely unexplored. We find that replacing the flattening of output feature maps in Impala-CNN with global average pooling leads to a notable performance improvement. This approach outperforms larger and more complex models in the Procgen Benchmark, particularly in terms of generalization. We call our proposed encoder model Impoola-CNN. A decrease in the network's translation sensitivity may be central to this improvement, as we observe the most significant gains in games without agent-centered observations. Our results demonstrate that network scaling is not just about increasing model size - efficient network design is also an essential factor. We make our code available at https://github.com/raphajaner/impoola.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。