提出可检测分布漂移的Q-learning算法，提升非平稳环境下的决策稳定性。
Q-Learning with Shift-Aware Upper Confidence Bound in Non-Stationary Reinforcement Learning
- 用转移密度函数检测环境分布变化
- 在10个任务中实现比基线更低的累计损失（regret）
- 适合动态环境中的在线学习应用
本文研究在有限时长远期和无限时长远期马尔可夫决策过程中的非平稳强化学习问题，其中环境的转移函数可能在特定回合或任意时间点突然改变。现有基于置信上界（QUCB）的Q-learning方法虽能学习有效策略，但在分布漂移后仍可能因过度利用而获得次优回报。为此，本文提出密度-置信上界Q-learning（DQUCB），通过转移密度函数识别分布漂移，并利用其似然值改进置信度估计，实现探索与利用的更好平衡。理论上，我们证明了理想情形下DQUCB的累积损失优于QUCB；实验表明，该算法在10个基准任务及使用深度Q网络的新冠疫情患者医院分配任务中，均以更低成本的计算开销显著降低累计损失，表现优于基线方法。
原文摘要 · Abstract (English)
We study the Non-Stationary Reinforcement Learning (RL) under distribution shifts in both finite-horizon episodic and infinite-horizon discounted Markov Decision Processes (MDPs). In the finite-horizon case, the transition functions may suddenly change at a particular episode. In the infinite-horizon setting, such changes can occur at an arbitrary time step during the agent's interaction with the environment. While the Q-learning Upper Confidence Bound algorithm (QUCB) can discover a proper policy during learning, due to the distribution shifts, this policy can exploit sub-optimal rewards after the shift happens. To address this issue, we propose Density-QUCB (DQUCB), a shift-aware Q-learning UCB algorithm, which uses a transition density function to detect distribution shifts, then leverages its likelihood to enhance the uncertainty estimation quality of Q-learning UCB, resulting in a balance between exploration and exploitation. Theoretically, we prove that our oracle DQUCB achieves a better regret guarantee than QUCB. Empirically, our DQUCB enjoys the computational efficiency of model-free RL and outperforms QUCB baselines by having a lower regret across RL tasks, as well as a COVID-19 patient hospital allocation task using a Deep-Q-learning architecture.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。