将分布强化学习扩展到部分可观测环境，实现可证明收敛的值迭代。
Provable Distributional Value Iteration under Partial Observability
- 提出适用于部分可观测决策过程的新分布贝尔曼算子。
- 在超度量下证明了算法收敛性，且在风险中性时还原经典解法。
- 用psi向量表示回报分布，适合处理不确定环境下的规划问题。
在许多现实世界的规划任务中，智能体需应对环境状态的不确定性以及随机动态和奖励带来的结果变异性。受世界模型方法进展启发，该文将分布强化学习（DistRL）从完全可观测领域扩展至部分可观测马尔可夫决策过程（POMDPs）。文中引入新的分布贝尔曼算子，并在上确界p-Wasserstein度量下证明其收敛性。同时提出通过psi向量对回报分布进行有限表示，推广了传统POMDP求解器中的alpha向量。基于此，构建了分布点基值迭代（DPBVI），将psi向量融入标准点基备份过程，实现了分布强化学习与POMDP规划的融合。实验表明，当风险中性时，DPBVI可还原经典点基值迭代（PBVI），验证了该分布扩展的有效性。
原文摘要 · Abstract (English)
In many real-world planning tasks, agents must tackle uncertainty about the environment's state and variability in the outcomes induced by stochastic dynamics and rewards. Motivated by recent progress in world model approaches, where latent models approximate beliefs and support planning, we extend Distributional Reinforcement Learning (DistRL), which models the entire return distribution for fully observable domains, to Partially Observable Markov Decision Processes (POMDPs). Concretely, we introduce new distributional Bellman operators for partial observability and prove their convergence under the supremum p-Wasserstein metric. We also propose a finite representation of these return distributions via psi-vectors, generalizing the classical alpha-vectors in POMDP solvers. Building on this, we develop Distributional Point-Based Value Iteration (DPBVI), which integrates psi-vectors into a standard point-based backup procedure, bridging DistRL and POMDP planning. Our experiments demonstrate that DPBVI recovers classical Point-Based Value Iteration (PBVI) in the risk-neutral case, validating the distributional extension.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。