用神经网络替代传统方法，高效求解超大规模部分可观测马尔可夫决策过程。
Neural Value Iteration
- 将价值函数表示为神经网络集合，利用其泛化能力避免全量计算
- 在不可行的超大规模问题上仍接近最优解，突破传统方法瓶颈
- 适合处理状态空间巨大、传统方法无法应对的复杂决策任务
部分可观测马尔可夫决策过程（POMDP）的价值函数具有分段线性凸（PWLC）特性，可由有限个超平面（即α-向量）表示。现有主流离线规划器采用基于点的价值迭代法，在可达信念点上对α-向量执行贝尔曼备份直至收敛。但由于每个α-向量是|S|-维的，贝尔曼备份的计算成本极高，导致该方法在大规模问题上迅速变得不可行。本文提出一种新思路：利用PWLC性质，将价值函数改用有限个神经网络表示。由此构建的新型规划算法——神经价值迭代（Neural Value Iteration），结合了神经网络的泛化能力与经典价值迭代框架。实验表明，该方法在现有离线求解器无法处理的极大规模POMDP中仍能获得近似最优解。
原文摘要 · Abstract (English)
The value function of a POMDP exhibits the piecewise-linear-convex (PWLC) property and can be represented as a finite set of hyperplanes, known as $α$-vectors. Most state-of-the-art POMDP solvers (offline planners) follow the point-based value iteration scheme, which performs Bellman backups on $α$-vectors at reachable belief points until convergence. However, since each $α$-vector is $|S|$-dimensional, these methods quickly become intractable for large-scale problems due to the prohibitive computational cost of Bellman backups. In this work, we demonstrate that the PWLC property allows a POMDP's value function to be alternatively represented as a finite set of neural networks. This insight enables a novel POMDP planning algorithm called \emph{Neural Value Iteration}, which combines the generalization capability of neural networks with the classical value iteration framework. Our approach achieves near-optimal solutions even in extremely large POMDPs that are intractable for existing offline solvers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。