线性循环网络能有效处理部分可观测强化学习问题。
Why Linear Recurrent Memory Works in Partially Observable Reinforcement Learning
- 构建两种线性滤波器，可精确或近似还原隐藏状态信息
- 在确定性转移下，滤波器使状态解码误差趋近于零
- 适用于动作控制的隐马尔可夫模型，适合状态不完全可观测场景
线性递归神经网络在部分可观测强化学习中表现优异。本文通过构造并研究两种线性滤波器提供理论解释：(i) 在确定性转移矩阵下，首个滤波器能精确重现隐马尔可夫模型（HMM）中预归一化对数几率，成为最优策略学习的充分统计量；(ii) 第二个滤波器在近乎确定性转移下实现状态解码误差趋于零，显著降低状态模糊性。该结果推广至动作控制的HMM，此时对应滤波器变为依赖动作的时变系统。数值实验验证了核心结论，并表明所构造的线性滤波器在小型强化学习游戏中具有强大特征提取能力。
原文摘要 · Abstract (English)
The family of linear recurrent neural networks has shown strong performance as recurrent memory units in partially observable reinforcement learning. We provide a theoretical justification for their empirical effectiveness by constructing and studying two linear filters: (i) the first exactly reproduces the pre-softmax logits of the belief vector in a hidden Markov model (HMM) under a deterministic transition matrix, thereby serving as a sufficient statistic for optimal policy learning, (ii) the second achieves vanishing state-decoding error under a nearly deterministic transition matrix, thus reducing state ambiguity to near zero. The results extend to action-controlled HMMs, where the corresponding linear filters become time-varying with action-dependent dynamics. We illustrate our main results through numerical experiments and further show that the constructed linear filter serves as a strong feature extractor in a small reinforcement learning game.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。