RTU让RNN在线强化学习更高效且表现更好
Real-Time Recurrent Learning using Trace Units in Reinforcement Learning
- 在L RU基础上引入迹单元，提升实时循环学习效率
- 在多个部分可观测环境中性能超越其他循环结构
- 计算量少，适合资源受限的在线学习场景
循环神经网络（RNN）常用于部分可观测环境中的表征学习。对于持续与环境交互的在线学习智能体，使用实时循环学习（RTRL）训练RNN是理想选择，但标准RNN的RTRL计算成本过高。一种有前景的方法是采用线性循环架构（LRUs），用复数对角递归权重替代密集权重，使RTRL高效化。本文在此基础上提出一种轻量级改进：循环迹单元（RTUs），仅对LRUs进行微小调整，但在使用RTRL训练时展现出显著性能提升。实验表明，RTUs在多个部分可观测环境任务中显著优于其他循环结构，同时计算开销更低。
原文摘要 · Abstract (English)
Recurrent Neural Networks (RNNs) are used to learn representations in partially observable environments. For agents that learn online and continually interact with the environment, it is desirable to train RNNs with real-time recurrent learning (RTRL); unfortunately, RTRL is prohibitively expensive for standard RNNs. A promising direction is to use linear recurrent architectures (LRUs), where dense recurrent weights are replaced with a complex-valued diagonal, making RTRL efficient. In this work, we build on these insights to provide a lightweight but effective approach for training RNNs in online RL. We introduce Recurrent Trace Units (RTUs), a small modification on LRUs that we nonetheless find to have significant performance benefits over LRUs when trained with RTRL. We find RTUs significantly outperform other recurrent architectures across several partially observable environments while using significantly less computation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。