改进Q学习更新顺序与初始化，显著提升回合制强化学习效率
ReversedQ: Opportunities for Faster Q-Learning in Episodic Online Reinforcement Learning
- 调整值函数更新顺序、频率和初始值以加速学习
- 在双向锁和链式MDP上，累积奖励提升至78.78%和61.81%
- 适合追求高效模型无关强化学习的科研与工程人员
我们研究有限时域回合制马尔可夫决策过程中的无模型Q学习，其跨回合动态保持平稳。我们识别出新兴无模型后验采样方法的核心问题：依赖延迟学习以获得理论保证。具体发现三个加速学习的机会：(i) 值函数更新顺序，(ii) 更新频率，(iii) 值函数初始化。基于Wang等人的RandomizedQ，我们在多个实验中验证这些改进及其组合效应。结果表明，所提方法ReversedQ在双向魔鬼组合锁（BDCL）上将标准化平均累积奖励从9.53%提升至78.78%，在链式MDP上从21.76%提升至61.81%。
原文摘要 · Abstract (English)
We study model-free Q-learning in finite-horizon episodic Markov Decision Processes (MDPs) with stationary dynamics across episodes. We identify a central issue in nascent model-free posterior-sampling works: the reliance on delayed learning in order to prove theoretical guarantees. In particular, we identify three opportunities for faster learning - (i) value-function update order, (ii) update frequencies, and (iii) value-function initialization. Using Wang et al.'s RandomizedQ as a basis, we illustrate these changes and their individual (as well as cumulative) impact in multiple empirical studies. We find that our combined modifications, termed ReversedQ, improve scaled mean cumulative reward compared to RandomizedQ, from 9.53% to 78.78% in the Bidirectional Diabolical Combination Lock (BDCL), and from 21.76% to 61.81% in a chain MDP.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。