提出周期正则化Q学习，解决函数逼近下Q学习的收敛难题。
Periodic Regularized Q-Learning
- 在投影算子层面引入正则化，构造收缩型值迭代
- 理论证明算法在线性函数逼近下具有有限时间收敛性
- 适合研究函数逼近下RL收敛性的学者参考
强化学习中，Q-learning在表格设置下有收敛保证，但在线性函数逼近下不成立。为此，已有研究通过正则化技术提升稳定性。本文提出周期正则化Q学习（PRQ），首先在投影算子层面引入正则化，显式构建正则化投影值迭代（RP-VI），并进一步扩展为基于样本的强化学习算法。通过合理正则化投影算子，使投影值迭代成为压缩映射。将该正则化投影推广至随机情形，建立了PRQ算法，并提供了严格理论分析，证明了在線性函數近似下PRQ的有限時間收斂保證。
原文摘要 · Abstract (English)
In reinforcement learning (RL), Q-learning is a fundamental algorithm whose convergence is guaranteed in the tabular setting. However, this convergence guarantee does not hold under linear function approximation. To overcome this limitation, a significant line of research has introduced regularization techniques to ensure stable convergence under function approximation. In this work, we propose a new algorithm, periodic regularized Q-learning (PRQ). We first introduce regularization at the level of the projection operator and explicitly construct a regularized projected value iteration (RP-VI), subsequently extending it to a sample-based RL algorithm. By appropriately regularizing the projection operator, the resulting projected value iteration becomes a contraction. By extending this regularized projection into the stochastic setting, we establish the PRQ algorithm and provide a rigorous theoretical analysis that proves finite-time convergence guarantees for PRQ under linear function approximation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。