arXiv:2510.06647stat.MLcs.LG2025-10被引 3

提出细粒度的后悔上界,让强化学习算法更精准地衡量性能差距。

Q-Learning with Fine-Grained Gap-Dependent Regret

  • 区分最优与次优状态动作对,构建新分析框架
  • 首次实现非UCB类算法的精细后悔上界,性能优于原版AMB
  • 适合关注算法理论精度与性能优化的研究者

本文研究在回合制表格马尔可夫决策过程中的模型无关强化学习的细粒度间隙依赖后悔上界。现有模型无关算法虽达到极小极大最坏情况后悔上界,但其间隙依赖边界仍较粗糙,未能充分捕捉次优性间隙结构。为此,本文为基于UCB和非UCB的算法建立了细粒度间隙依赖后悔上界。在基于UCB的设定中,提出一种新分析框架,显式分离最优与次优状态动作对的分析,首次获得UCB-Hoeffding(Jin等,2018)的细粒度后悔上界。为展示该框架普适性,引入新型基于UCB的ULCB-Hoeffding算法,受AMB(Xu等,2021)启发但结构简化,兼具细粒度后悔保证且实验表现优于AMB。在非UCB设定中,重新审视唯一已知算法AMB,发现其算法设计与分析中存在两处关键问题:Q值更新中不当截断、集中性论证违反鞅差条件。提出改进版AMB,解决上述问题,首次为非UCB方法建立严格细粒度间隙依赖后悔上界,实验验证其性能优于原始AMB。

原文摘要 · Abstract (English)

We study fine-grained gap-dependent regret bounds for model-free reinforcement learning in episodic tabular Markov Decision Processes. Existing model-free algorithms achieve minimax worst-case regret, but their gap-dependent bounds remain coarse and fail to fully capture the structure of suboptimality gaps. We address this limitation by establishing fine-grained gap-dependent regret bounds for both UCB-based and non-UCB-based algorithms. In the UCB-based setting, we develop a novel analytical framework that explicitly separates the analysis of optimal and suboptimal state-action pairs, yielding the first fine-grained regret upper bound for UCB-Hoeffding (Jin et al., 2018). To highlight the generality of this framework, we introduce ULCB-Hoeffding, a new UCB-based algorithm inspired by AMB (Xu et al.,2021) but with a simplified structure, which enjoys fine-grained regret guarantees and empirically outperforms AMB. In the non-UCB-based setting, we revisit the only known algorithm AMB, and identify two key issues in its algorithm design and analysis: improper truncation in the $Q$-updates and violation of the martingale difference condition in its concentration argument. We propose a refined version of AMB that addresses these issues, establishing the first rigorous fine-grained gap-dependent regret for a non-UCB-based method, with experiments demonstrating improved performance over AMB.

强化学习后悔上界理论分析Q-learning

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。