arXiv:2602.01903cs.LGstat.ML2026-02中稿 · ICML

新算法自适应不同环境,实现更优的强化学习后悔界。

Data- and Variance-dependent Regret Bounds for Online Tabular MDPs

  • 基于乐观正则化策略,融合全局与策略优化方法。
  • 在随机环境中实现方差感知的亚线性后悔,可到对数级增长。
  • 适用于复杂度变化多端的在线马尔可夫决策问题,适合研究者参考。

本文研究已知转移概率的在线周期性表格型马尔可夫决策过程(MDPs),设计了兼具最优性能的算法,在对抗性环境下实现数据依赖的后悔界,在随机环境下实现方差依赖的后悔界。通过引入一阶量、二阶量和路径长度等新数据依赖度量刻画MDP复杂度,并利用方差相关度量描述随机环境。算法基于带对数障碍正则化的乐观跟随规则化领导者,分别采用全局优化与策略优化框架。全局优化方法在对抗性环境下获得一阶、二阶及路径长度的后悔界;在随机环境下,实现不依赖差距的方差感知后悔界,以及关于回合数为多项式对数的差距感知后悔界。策略优化方法达到相同适应性,仅在回合长度上略有损失,得益于一种新的乐观Q函数估计器。最后,我们建立了基于数据依赖复杂度的对抗性后悔下界和基于方差的随机后悔下界,表明全局优化方法的上界近乎最优。

原文摘要 · Abstract (English)

This work studies online episodic tabular Markov decision processes (MDPs) with known transitions and develops best-of-both-worlds algorithms that achieve refined data-dependent regret bounds in the adversarial regime and variance-dependent regret bounds in the stochastic regime. We quantify MDP complexity using a first-order quantity and several new data-dependent measures for the adversarial regime, including a second-order quantity and a path-length measure, as well as variance-based measures for the stochastic regime. To adapt to these measures, we develop algorithms based on global optimization and policy optimization, both built on optimistic follow-the-regularized-leader with log-barrier regularization. For global optimization, our algorithms achieve first-order, second-order, and path-length regret bounds in the adversarial regime, and in the stochastic regime, they achieve a variance-aware gap-independent bound and a variance-aware gap-dependent bound that is polylogarithmic in the number of episodes. For policy optimization, our algorithms achieve the same data- and variance-dependent adaptivity, up to a factor of the episode horizon, by exploiting a new optimistic $Q$-function estimator. Finally, we establish regret lower bounds in terms of data-dependent complexity measures for the adversarial regime and a variance measure for the stochastic regime, implying that the regret upper bounds achieved by the global-optimization approach are nearly optimal.

强化学习在线学习后悔界马尔可夫决策

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。