提出更快收敛的强化学习算法，用于解决动态资源分配难题。
Faster Q-Learning Algorithms for Restless Bandits
- 用双时标方法结合Q-learning与索引学习，提升决策效率
- 使用UCB探索策略的PhaseQL收敛速度最快，优于ε-greedy
- 适合研究动态调度、在线优化的科研人员参考
我们研究了用于随机多臂老虎机（RMAB）的Whittle索引学习算法。首先提出Q-learning及其变体——快速Q-learning（SQL）、广义快速Q-learning（GSQL）和阶段Q-learning（PhaseQL），并讨论ε-greedy和上置信界（UCB）等探索策略。进一步将Q-learning及其变体与UCB策略结合进行研究。通过数值实验表明，采用UCB探索的Q-learning具有更快收敛性，而使用UCB的PhaseQL收敛速度最优。随后将这些算法扩展至索引学习框架，该框架为双时标随机逼近：在慢时标上更新索引学习方案，在快时标上以固定索引值进行Q-learning更新。研究常步长双时标随机逼近算法，并通过数值示例验证性能。结果表明，结合UCB的索引学习比ε-greedy收敛更快；且无论是否采用UCB，PhaseQL均表现出最佳收敛性能。
原文摘要 · Abstract (English)
We study the Whittle index learning algorithm for restless multi-armed bandits (RMAB). We first present Q-learning algorithm and its variants -- speedy Q-learning (SQL), generalized speedy Q-learning (GSQL) and phase Q-learning (PhaseQL). We also discuss exploration policies -- $ε$-greedy and Upper confidence bound (UCB). We extend the study of Q-learning and its variants with UCB policy. We illustrate using numerical example that Q-learning with UCB exploration policy has faster convergence and PhaseQL with UCB have fastest convergence rate. We next extend the study of Q-learning variants for index learning to RMAB. The algorithm of index learning is two-timescale variant of stochastic approximation, on slower timescale we update index learning scheme and on faster timescale we update Q-learning assuming fixed index value. We study constant stepsizes two timescale stochastic approximation algorithm. We describe the performance of our algorithms using numerical example. It illustrate that index learning with Q learning with UCB has faster convergence that $ε$ greedy. Further, PhaseQL (with UCB and $ε$ greedy) has the best convergence than other Q-learning algorithms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。