提出几何加权目标更新，提升线性Q学习稳定性。
Geometrically Averaged Hard Target Updates for Linear Q-Learning
- 用几何权重平均周期性目标更新，实现平滑过渡
- λ接近1时等价于投影Q值迭代，提升收敛性
- 适用于需要稳定训练的线性函数近似场景
周期性硬目标更新是现代深度Q学习中最常见的稳定机制之一。近期研究指出，目标更新可改善带函数逼近（包括线性逼近）的Q学习稳定性。本文引入并分析了所谓的λ-目标更新：通过λ-几何权重(1−λ)λ^{m−1}（λ∈[0,1]）对m周期目标更新映射进行加权平均。当λ=0时恢复单周期目标更新，而λ↑1时则趋近于投影Q值迭代。我们基于切换系统模型和相关工具，研究该机制在带线性函数逼近的Q学习（即线性Q学习）中的表现。为清晰起见，论文以确定性版本为例；该框架可扩展至随机强化学习设定。
原文摘要 · Abstract (English)
Periodic hard target updates are among the most common stabilization devices in modern deep Q-learning. Recent studies suggest that target updates can improve stability in Q-learning with function approximation, including linear function approximation. We introduce and analyze the so-called $λ$-target update, obtained by averaging the $m$-periodic target update maps with $λ$-geometric weights $(1-λ)λ^{m-1}$, $λ\in [0,1]$. The endpoint $λ=0$ recovers the one-period target update, while the continuous endpoint $λ\uparrow1$ recovers projected Q-value iteration. We study this mechanism for Q-learning with linear function approximation, namely linear Q-learning, using a switching-system model and related tools. For clarity, the paper treats a deterministic version; the formulation extends to stochastic reinforcement-learning settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。