arXiv:2503.00788cs.GTcs.AI2025-03被引 1

提出简洁策略表示方法,解决无限状态马尔可夫决策问题的验证与合成难题。

Taming Infinity one Chunk at a Time: Concisely Represented Strategies in One-Counter MDPs

  • 基于计数器区间划分,设计可压缩表示的策略结构。
  • 所有问题均在PSPACE内可判定,支持精确概率验证与策略合成。
  • 适合形式化验证、自动推理及理论计算机科学方向研究者。

马尔可夫决策过程(MDPs)是建模随机环境中决策的经典框架。本文研究一类基本的无限MDPs:一计数器MDPs(OC-MDPs)。它们通过一个取自然数值的计数器扩展有限MDPs,从而在状态与计数器值组合构成的配置空间上诱导出无限MDP。考虑两类典型目标:到达目标状态(状态可达性),以及以计数器值为零时到达目标状态(选择性终止)。后者的合成问题目前尚未被证明是可判定的,且与数论中的重大开放问题相关。此外,即使看似简单的策略(如无记忆策略)在OC-MDP中也可能因配置空间无限而无法实际构造:需要有限且尽可能小的表示。为此,我们引入两类基于计数器值区间划分的简洁策略表示类。对这两类策略和两类目标,我们研究了验证问题(给定策略是否保证足够高的目标达成概率),以及两个合成问题(是否存在此类策略):一个是区间划分作为输入固定,另一个是仅参数化区间划分。我们提出一种基于压缩诱导无限MDP的通用方法,在所有情况下实现可判定性,且复杂度均处于PSPACE内。

原文摘要 · Abstract (English)

Markov decision processes (MDPs) are a canonical model to reason about decision making within a stochastic environment. We study a fundamental class of infinite MDPs: one-counter MDPs (OC-MDPs). They extend finite MDPs via an associated counter taking natural values, thus inducing an infinite MDP over the set of configurations (current state and counter value). We consider two characteristic objectives: reaching a target state (state-reachability), and reaching a target state with counter value zero (selective termination). The synthesis problem for the latter is not known to be decidable and connected to major open problems in number theory. Furthermore, even seemingly simple strategies (e.g., memoryless ones) in OC-MDPs might be impossible to build in practice (due to the underlying infinite configuration space): we need finite, and preferably small, representations. To overcome these obstacles, we introduce two natural classes of concisely represented strategies based on a (possibly infinite) partition of counter values in intervals. For both classes, and both objectives, we study the verification problem (does a given strategy ensure a high enough probability for the objective?), and two synthesis problems (does there exist such a strategy?): one where the interval partition is fixed as input, and one where it is only parameterized. We develop a generic approach based on a compression of the induced infinite MDP that yields decidability in all cases, with all complexities within PSPACE.

马尔可夫决策形式化验证可判定性策略合成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。