arXiv:2608.26763cs.AIcs.FL2026-08

用有限状态机对折扣收益分类,实现高效策略合成。

Categorizer Automata for Discounted-Sum Payoffs

  • 构建线性状态空间的分类自动机,替代指数级传统方法。
  • 对分段常数效用可精确求解,对分段Lipschitz效用得ε最优策略。
  • 适用于不连续效用函数,适合形式化验证与强化学习场景。

将连续数据划分为离散区间是人工智能中的基础操作。本文提出分类自动机——一种确定性自动机,能读取无限奖励序列并识别其折扣和所属的有限个区间之一。该模型推广了比较器自动机(仅两个区间),后者已在定量合成中证明有效。主要技术贡献是构造出状态空间与区间数成线性关系的分类自动机,而非传统交叉积带来的指数增长。随后将其应用于马尔可夫决策过程,实现对任意折扣收益期望效用的最大化,即使效用函数不连续亦可。对于分段常数效用函数,算法为精确解,运行时间为伪多项式时间;对分段Lipschitz类效用(包含有限跳跃间有界斜率的函数),同样在伪多项式时间内运行,并给出ε-最优策略。此外,我们证明该合成问题对分段常数效用已为PSPACE-hard。

原文摘要 · Abstract (English)

Categorizing continuous data into discrete bins is a fundamental operation in artificial intelligence. We introduce the categorizer automaton, a deterministic automaton that reads an infinite sequence of rewards and identifies which of finitely many bins contains its discounted sum. Categorizer automata generalize comparator automata, the special case of two bins, which have already proven useful in quantitative synthesis. Our main technical contribution is the construction of a categorizer automaton whose state space is linear in the number of bins, rather than exponential as obtained by a cross-product of comparator automata. We then apply categorizer automata to Markov decision processes, where they allow one to synthesize policies that maximize the expected utility of a discounted-sum payoff for utility functions that may be discontinuous. For piecewise-constant utility functions, the resulting algorithm is exact and runs in pseudo-polynomial time. For piecewise-Lipschitz utility functions, a class that includes any utility with bounded slope between finitely many jumps, it again runs in pseudo-polynomial time and yields an $\varepsilon$-optimal policy. We also show that the synthesis problem considered is PSPACE-hard already for piecewise-constant utilities.

自动机强化学习决策优化形式化验证

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。