提出无需记忆的策略迭代方法，高效求解部分可观测决策问题。
Memoryless Policy Iteration for Episodic POMDPs
- 通过周期性单步优化与评估交替，实现无记忆策略的单调改进。
- 在多个测试环境上，计算速度比基线快数倍，且不依赖模型。
- 适合需要快速决策的实时系统，如机器人控制、智能体交互场景。
无记忆和有限记忆策略为求解部分可观测马尔可夫决策过程（POMDPs）提供了实用替代方案，因其直接作用于输出空间而非高维信念空间。然而，将经典策略迭代方法扩展至该设定仍具挑战性：输出过程非马尔可夫性导致各阶段的策略改进相互依赖。本文提出一类新的单调提升策略迭代算法，其交替执行基于输出的单阶段策略改进与按预定周期模式进行的策略评估。我们证明该族算法存在最优周期模式以最大化自然计算效率指标，并识别出周期最小的最简模式。在此结构基础上，进一步开发了一种无需模型的变体，通过数据估计值并直接学习无记忆策略。在多个POMDP示例中，本方法在基于模型与无模型设置下均显著优于策略梯度基线及近期专用算法，实现显著计算加速。
原文摘要 · Abstract (English)
Memoryless and finite-memory policies offer a practical alternative for solving partially observable Markov decision processes (POMDPs), as they operate directly in the output space rather than in the high-dimensional belief space. However, extending classical methods such as policy iteration to this setting remains difficult; the output process is non-Markovian, making policy-improvement steps interdependent across stages. We introduce a new family of monotonically improving policy-iteration algorithms that alternate between single-stage output-based policy improvements and policy evaluations according to a prescribed periodic pattern. We show that this family admits optimal patterns that maximize a natural computational-efficiency index, and we identify the simplest pattern with minimal period. Building on this structure, we further develop a model-free variant that estimates values from data and learns memoryless policies directly. Across several POMDPs examples, our method achieves significant computational speedups over policy-gradient baselines and recent specialized algorithms in both model-based and model-free settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。