arXiv:2508.18252cs.AI2025-08被引 1

用有理函数高效计算黑威尔最优策略,突破传统算法效率瓶颈。

Efficient Computation of Blackwell Optimal Policies using Rational Functions

  • 通过有理函数排序替代数值计算,实现符号化求解
  • 首次提出确定性MDP的强多项式时间算法,一般MDP达亚指数级速度
  • 适用于需要长期与短期均衡决策的研究者,如强化学习优化

马尔可夫决策过程(MDPs)是序列决策建模的基础框架,常用折扣奖励和平均奖励作为优化标准。然而,前者可能过度关注短期收益,后者依赖强结构假设。黑威尔最优性克服这些缺陷,提供在两种框架下均成立的稳健准则。尽管理论优势明显,现有计算黑威尔最优(BO)策略的算法仍存在计算成本高或难以实现的问题。本文提出基于邻近1处有理函数排序的计算方法,将最先进的确定性和一般性MDP算法改进为符号运算,避免位复杂度依赖。对确定性MDP,首次实现强多项式时间算法;对一般MDP,获得首个亚指数时间算法。同时推广多个策略迭代算法,将最优上界从折扣准则扩展至黑威尔准则。

原文摘要 · Abstract (English)

Markov Decision Problems (MDPs) provide a foundational framework for modelling sequential decision-making across diverse domains, guided by optimality criteria such as discounted and average rewards. However, these criteria have inherent limitations: discounted optimality may overly prioritise short-term rewards, while average optimality relies on strong structural assumptions. Blackwell optimality addresses these challenges, offering a robust and comprehensive criterion that ensures optimality under both discounted and average reward frameworks. Despite its theoretical appeal, existing algorithms for computing Blackwell Optimal (BO) policies are computationally expensive or hard to implement. In this paper we describe procedures for computing BO policies using an ordering of rational functions in the vicinity of $1$. We adapt state-of-the-art algorithms for deterministic and general MDPs, replacing numerical evaluations with symbolic operations on rational functions to derive bounds independent of bit complexity. For deterministic MDPs, we give the first strongly polynomial-time algorithms for computing BO policies, and for general MDPs we obtain the first subexponential-time algorithm. We further generalise several policy iteration algorithms, extending the best known upper bounds from the discounted to the Blackwell criterion.

强化学习最优策略算法优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。