研究如何将动作信息融入RNN状态更新，提升强化学习性能
Investigating Action Encodings in Recurrent Neural Networks in Reinforcement Learning
- 在RNN状态更新中引入动作信息作为输入
- 实验证明该设计可显著提升多任务环境下的学习效率
- 适合关注RNN在强化学习中应用的算法研究者
在真实世界部署强化学习（RL）智能体时，构建和维护状态是关键。循环神经网络（RNN）已成为解决状态构建问题的重要方法，许多大规模强化学习系统都采用RNN结构。尽管如此，影响性能的关键设计选择与实现细节往往未被充分报告。本文聚焦于一种可调整的RNN架构设计维度：如何将动作信息整合到递归单元的状态更新函数中。我们探讨了多种动作信息的使用方式，并在多个典型任务环境中对改进后的架构进行实验评估。结果表明，合理融合动作信息能有效提升策略学习与价值函数估计的表现。最后，本文讨论了未来在强化学习场景下开发新型递归单元的挑战与方向。
原文摘要 · Abstract (English)
Building and maintaining state to learn policies and value functions is critical for deploying reinforcement learning (RL) agents in the real world. Recurrent neural networks (RNNs) have become a key point of interest for the state-building problem, and several large-scale reinforcement learning agents incorporate recurrent networks. While RNNs have become a mainstay in many RL applications, many key design choices and implementation details responsible for performance improvements are often not reported. In this work, we discuss one axis on which RNN architectures can be (and have been) modified for use in RL. Specifically, we look at how action information can be incorporated into the state update function of a recurrent cell. We discuss several choices in using action information and empirically evaluate the resulting architectures on a set of illustrative domains. Finally, we discuss future work in developing recurrent cells and discuss challenges specific to the RL setting.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。