证明了无需局部计时器的差分TD学习几乎必然收敛。
Almost Sure Convergence of Differential Temporal Difference Learning for Average Reward Markov Decision Processes
- 用标准递减学习率实现无局部计时器的差分TD学习。
- 提出三种条件，确保离策略n步差分TD也能收敛。
- 为实际应用提供更强理论支撑，适合研究强化学习理论者。
平均奖励是强化学习中衡量智能体长期性能的核心指标。差分时间差分（Differential TD）学习算法是平均奖励强化学习的重要进展，能在在线模式下高效学习与平均奖励相关的价值函数，适用于同策略和异策略设置。然而，现有收敛性保证依赖于与状态访问次数绑定的局部计时器，这在实践中并不使用，且无法推广到非表格化情形。本文通过证明，在任意n步下，使用标准递减学习率而无需局部计时器，同策略n步差分TD可实现几乎必然收敛。随后推导出三个充分条件，使异策略n步差分TD同样可在无局部计时器条件下收敛。这些结果强化了差分TD的理论基础，使其收敛分析更贴近实际应用。
原文摘要 · Abstract (English)
The average reward is a fundamental performance metric in reinforcement learning (RL) focusing on the long-run performance of an agent. Differential temporal difference (TD) learning algorithms are a major advance for average reward RL as they provide an efficient online method to learn the value functions associated with the average reward in both on-policy and off-policy settings. However, existing convergence guarantees require a local clock in learning rates tied to state visit counts, which practitioners do not use and does not extend beyond tabular settings. We address this limitation by proving the almost sure convergence of on-policy $n$-step differential TD for any $n$ using standard diminishing learning rates without a local clock. We then derive three sufficient conditions under which off-policy $n$-step differential TD also converges without a local clock. These results strengthen the theoretical foundations of differential TD and bring its convergence analysis closer to practical implementations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。