arXiv:2604.21256cs.AI2026-04被引 1

研究POMDP策略在观测偏差下的鲁棒性,给出可保证性能的偏差上限。

Robustness Analysis of POMDP Policies to Observation Perturbations

  • 将观测鲁棒性问题建模为双层优化,利用单调性设计高效求解算法。
  • 非粘滞情形下只需考虑有限状态控制器节点的观测,显著降低复杂度。
  • 算法有收敛性保障,适用于机器人与运筹学中的大规模实际问题。

部分可观测马尔可夫决策过程(POMDP)策略通常基于理想系统模型设计。但在部署中,因校准漂移或传感器退化,模型可能偏离真实系统,导致性能意外下降。本文研究策略对观测模型偏差的鲁棒性,提出“策略观测鲁棒性问题”:确定在何种最大观测偏差下,策略价值仍高于给定阈值。分析两种变体——粘滞型(偏差依赖于状态和动作)与非粘滞型(偏差可依赖历史)。证明该问题可表述为双层优化,内层优化关于偏差大小单调,从而可在外层使用根查找算法高效求解。对于非粘滞情形,当策略用有限状态控制器(FSC)表示时,仅需考虑依赖于FSC节点的观测,而非完整历史。提出具有完备性与收敛性保证的鲁棒区间搜索算法,适用于粘滞与非粘滞情形。非粘滞情形下为多项式时间复杂度,粘滞情形下至多为指数时间。实验验证了算法在含数万状态的POMDP问题上的可扩展性,并通过机器人与运筹学案例展示了其实际应用价值。

原文摘要 · Abstract (English)

Policies for Partially Observable Markov Decision Processes (POMDPs) are often designed using a nominal system model. In practice, this model can deviate from the true system during deployment due to factors such as calibration drift or sensor degradation, leading to unexpected performance degradation. This work studies policy robustness against deviations in the POMDP observation model. We introduce the Policy Observation Robustness Problem: to determine the maximum tolerable deviation in a POMDP's observation model that guarantees the policy's value remains above a specified threshold. We analyze two variants: the sticky variant, where deviations are dependent on state and actions, and the non-sticky variant, where they can be history-dependent. We show that the Policy Observation Robustness Problem can be formulated as a bi-level optimization problem in which the inner optimization is monotonic in the size of the observation deviation. This enables efficient solutions using root-finding algorithms in the outer optimization. For the non-sticky variant, we show that when policies are represented with finite-state controllers (FSCs) it is sufficient to consider observations which depend on nodes in the FSC rather than full histories. We present Robust Interval Search, an algorithm with soundness and convergence guarantees, for both the sticky and non-sticky variants. We show this algorithm has polynomial time complexity in the non-sticky variant and at most exponential time complexity in the sticky variant. We provide experimental results validating and demonstrating the scalability of implementations of Robust Interval Search to POMDP problems with tens of thousands of states. We also provide case studies from robotics and operations research which demonstrate the practical utility of the problem and algorithms.

POMDP鲁棒性策略优化机器人

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。