将缺失数据理论融入强化学习，让智能体在观测不全时仍能做出最优决策。
Missingness-MDPs: Bridging the Theory of Missing Data and POMDPs
- 用缺失机制建模观测不全，把缺失类型分为随机、非随机等三类
- 从轨迹数据中学习缺失模式，确保策略近似最优（误差可控制）
- 适合处理真实场景中部分信息缺失的强化学习任务
我们提出缺失性马尔可夫决策过程（miss-MDP），一种新型部分可观测马尔可夫决策过程（POMDP），将缺失数据理论纳入其中。miss-MDP 的观测函数即为缺失函数，刻画每个状态特征在时间步上缺失的概率。文献中区分三类经典缺失机制：完全随机缺失（MCAR）、随机缺失（MAR）和非随机缺失（MNAR）。我们的规划目标是，在给定动作-观测轨迹数据集的前提下，为未知缺失函数的 miss-MDP 计算近似最优策略。实现该最优性保证需从数据中学习缺失函数，这对一般 POMDP 不可行。为此，我们利用不同缺失类型的结构特性，推导出概率近似正确（PAC）的学习算法，从而获得一个近似但完整的 miss-MDP，再使用现成规划方法求解。我们证明：以高概率，所得策略在真实 miss-MDP 中为 ε-最优。实验验证了理论，并显示本方法优于两种无模型 POMDP 方法。
原文摘要 · Abstract (English)
We introduce missingness-MDPs (miss-MDPs), a novel subclass of partially observable Markov decision processes (POMDPs) that incorporates the theory of missing data. A miss-MDP is a POMDP whose observation function is a missingness function, specifying the probability that individual state features are missing (i.e., unobserved) at a time step. The literature distinguishes three canonical missingness types: missing (1) completely at random (MCAR), (2) at random (MAR), and (3) not at random (MNAR). Our planning problem is to compute near-optimal policies for a miss-MDP with an unknown missingness function, given a dataset of action-observation trajectories. Achieving such optimality guarantees for policies requires learning the missingness function from data, which is infeasible for general POMDPs. To overcome this challenge, we exploit the structural properties of different missingness types to derive probably approximately correct (PAC) algorithms for learning the missingness function. These algorithms yield an approximate but fully specified miss-MDP that we solve using off-the-shelf planning methods. We prove that, with high probability, the resulting policies are epsilon-optimal in the true miss-MDP. Empirical results confirm the theory and demonstrate superior performance of our approach over two model-free POMDP methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。