用信念状态提升强化学习在部分可观测环境下的表现
Belief-State RWKV for Reinforcement Learning under Partial Observability
- 将循环状态显式定义为信念状态，包含均值与方差
- 在最难的分布内场景下回报提升，噪声漂移时也更稳健
- 适合研究部分可观测强化学习的学者与工程师
我们提出一种基于RWKV型递归序列模型的强化学习新范式，将固定大小的递归状态明确解释为信念状态而非黑箱隐藏向量。不同于仅依赖单一摘要向量h_t的策略与价值函数，我们维护一个由RWKV式递归统计推导出的紧凑不确定性感知状态b_t = (μ_t, Σ_t)，使控制决策同时依赖记忆与不确定性。该设计针对普通固定状态策略在部分可观测设置中的关键缺陷：虽可存储证据，但未必体现置信度。本文给出方法、理论框架及一次带有隐藏层级观测噪声的初步强化学习实验，并进行测试时噪声扫描。结果表明，信念状态策略整体接近最优递归基线，且在最困难的分布内情形和保留噪声漂移条件下略有回报提升。额外消融实验显示，此简单信念读出机制优于两种更结构化的扩展——门控记忆控制与特权信念目标，凸显当前需更丰富的基准测试。
原文摘要 · Abstract (English)
We propose a stronger formulation of RL on top of RWKV-style recurrent sequence models, in which the fixed-size recurrent state is explicitly interpreted as a belief state rather than an opaque hidden vector. Instead of conditioning policy and value on a single summary h_t, we maintain a compact uncertainty-aware state b_t = (μ_t, Σ_t) derived from RWKV-style recurrent statistics and let control depend on both memory and uncertainty. This design targets a key weakness of plain fixed-state policies in partially observed settings: they may store evidence, but not necessarily confidence. We present the method, a theoretical program, and a pilot RL experiment with hidden episode-level observation noise together with a test-time noise sweep. The pilot shows that belief-state policies nearly match the best recurrent baseline overall while slightly improving return on the hardest in-distribution regime and under a held-out noise shift. Additional ablations show that this simple belief readout is currently stronger than two more structured extensions, namely gated memory control and privileged belief targets, underscoring the need for richer benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。