提出可实时计算的循环单元，让在线强化学习在部分可观测环境下仍能高效运行。
Streaming Reinforcement Learning under Partial Observability with Real-Time Recurrent Learning
- 设计对角循环结构，实现精确实时递归学习，时间与内存复杂度线性增长。
- 在链式记忆任务中，性能远超传统方法，链长达128时仍稳定表现。
- 无需经验回放或批量更新，适合资源受限的实时智能体应用。
流式强化学习是一种符合自然学习智能体特性的在线学习范式，要求数据以单批次增量处理且无经验回放缓冲区。尽管流式强化学习已能在全可观测场景下结合深度函数逼近实现扩展，但部分可观测环境仍难以应对。截断时间反向传播（TBPTT）在流式设置下退化为一步梯度，而精确的实时递归学习（RTRL）成本过高。本文提出循环痕迹单元（Recurrent Trace Units, RTUs），一种对角循环架构，使精确RTRL具备参数量的线性时间与内存复杂度。实验表明，该方法可无缝集成至现有流式算法，在离散与连续控制任务中均有效。在链长2至128的MemoryChain诊断任务中，其性能显著优于使用前馈网络、GRU和RTU的流式TBPTT(1)基线。在五个POPGym任务及部分可观测的MuJoCo连续控制任务上，流式方法性能接近批处理PPO，在遮蔽的MuJoCo任务中恢复了大部分批处理性能，且未使用任何经验回放或批量更新。
原文摘要 · Abstract (English)
Streaming reinforcement learning has emerged as an online learning paradigm that conforms to the restrictions of natural learning agents that process data incrementally, i.e. with a batch size of 1 and no replay buffer. While streaming RL has recently been shown to scale with deep function approximation with full observability, partially observable settings have remained out of reach. Truncated backpropagation through time collapses to a one-step gradient horizon under the streaming setting, and exact real-time recurrent learning is prohibitively expensive. We close this gap using recurrent trace units, a diagonal recurrent architecture that enables exact RTRL with linear time and memory complexity in the parameter count, and show that they integrate cleanly into existing streaming algorithms across both discrete and continuous control. On a MemoryChain diagnostic with chain lengths from 2 to 128, our method sustains performance where streaming TBPTT(1) baselines using feedforward, GRU, and RTU networks collapse. On five POPGym tasks and on partially observable MuJoCo continuous control, the streaming approach is competitive with batched PPO on POPGym and recovers a substantial fraction of batched performance on masked MuJoCo, despite using no replay buffer or batched updates.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。