研究非矩形平均奖励鲁棒MDP，给出最优策略与有限时间表现评估方法。
Non-Rectangular Average-Reward Robust MDPs: Optimal Policies and Their Transient Values
- 基于对抗性转移概率耦合建模，无需矩形性假设
- 证明平均奖励最优无法保证短期表现，存在任意差的瞬时值
- 提出分段策略，实现常数阶瞬时性能，适合长期鲁棒决策场景
我们研究在平均奖励准则下的非矩形鲁棒马尔可夫决策过程，其中模糊集耦合了跨状态的转移概率，对手在整个时序上承诺采用一个平稳核。我们证明，任何在模糊集上均匀实现次线性期望后悔的历史依赖策略都是鲁棒最优的；且鲁棒价值可表示为模糊集中经典最优收益的下确界，无需矩形性或鲁棒动态规划原理。在弱连通性假设下，我们通过将平均奖励强化学习文献中的高概率后悔界转化为期望后悔准则，建立了此类策略的存在性。随后引入瞬时值框架，评估鲁棒最优策略的有限时间性能，证明仅平均奖励最优可能掩盖任意差的瞬时表现，并推导出基于后悔的瞬时值下界。最后，我们构造了一种分段策略，结合最坏情况模型的最优平稳策略、任意时刻有效的序列检验以及在线学习回退机制，实现了常数阶瞬时值。
原文摘要 · Abstract (English)
We study non-rectangular robust Markov decision processes under the average-reward criterion, where the ambiguity set couples transition probabilities across states and the adversary commits to a stationary kernel for the entire horizon. We show that any history-dependent policy achieving sublinear expected regret uniformly over the ambiguity set is robust-optimal, and that the robust value admits a minimax representation as the infimum over the ambiguity set of the classical optimal gains, without requiring any form of rectangularity or robust dynamic programming principle. Under the weak communication assumption, we establish the existence of such policies by converting high-probability regret bounds from the average-reward reinforcement learning literature into the expected-regret criterion. We then introduce a transient-value framework to evaluate finite-time performance of robust optimal policies, proving that average-reward optimality alone can mask arbitrarily poor transients and deriving regret-based lower bounds on transient values. Finally, we construct an epoch-based policy that combines an optimal stationary policy for the worst-case model with an anytime-valid sequential test and an online learning fallback, achieving a constant-order transient value.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。