解析目标更新如何让线性Q学习收敛，给出理论保障。
Target Updates May Stabilize Linear Q-Learning: Periodic and Soft Dynamics
- 用切换线性系统和联合谱半径分析目标更新机制
- 证明周期硬更新与软更新均能保证收敛到精确解
- 适合研究强化学习稳定性与理论分析的读者
Q-learning中的周期目标更新和演员-评论家方法中的软目标更新在实践中已被广泛证实为有效的稳定机制，但其精确的理论解释仍不完整。本文针对具有线性函数逼近的Q-learning（线性Q-learning）进行了严格而精确的分析，利用贝尔曼最大操作所引发的精确切换线性系统（SLS）动态，以及相应切换矩阵族的联合谱半径（JSR）。尽管线性Q-learning一般情况下可能不收敛，但本文在明确的谱条件和步长条件下，证明了周期硬目标更新和软目标更新均可保证收敛至精确的投影Q-贝尔曼解。主要分析针对确定性线性Q-learning展开，此时目标更新机制最为清晰。一旦建立均值递推对应的JSR证书，随机强化学习场景可通过将确定性模式替换为采样随机模式，并引入相应的随机噪声分析来处理。
原文摘要 · Abstract (English)
Periodic target updates in Q-learning and soft target updates in actor-critic methods are empirically well established stabilization mechanisms, but their precise theoretical explanation is still incomplete. This paper gives a rigorous and exact analysis of these mechanisms for Q-learning with linear function approximation (linear Q-learning) using the exact switched linear system (SLS) dynamics induced by the Bellman maximum and the joint spectral radius (JSR) of the resulting switching matrix families. Although linear Q-learning can fail to converge in general, we prove that, under explicit spectral and step-size conditions, periodic hard target updates and soft target updates can guarantee convergence to the exact projected Q-Bellman solution. The main analysis is carried out for deterministic linear Q-learning, where the target-update mechanism is most transparent. Once the corresponding JSR certificate is established for the mean recursion, the stochastic reinforcement-learning setting can be treated by replacing deterministic modes with sampled stochastic modes and adding the corresponding stochastic-noise analysis.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。